Fussen, Bavaria: Germany's most overlooked town that's next to Neuschwanstein Castle

Never heard of Fussen? Nor have most of the 1.5 million visitors who cram into Neuschwanstein Castle just four kilometres down the road. Certainly, the white-turreted folly of mad king Ludwig II of Bavaria is an eye-popping vision, but Fussen has a castle too – and churches, medieval alleyways and sunlit squares where locals quaff beer, happy in the knowledge they aren't about to be stampeded by herds of Asian tourists.

The tourists buzz right through Fussen in their coaches, and admittedly highway 16 doesn't showcase the town to best effect, unless you're an aficionado of hospitals and multi-storey carparks. Yet Fussen is 2000 years older than Neuschwanstein. It sits astride important historical trade routes and has a proud old town enclosed in ramparts. Instead of Neuschwanstein's kitschy swan decorations, it's adorned with dragons and a Dance of Death in which jaunty skeletons drag gentlemen in doublets to their doom.

Factor time in Fussen into your castle-hopping Bavarian itinerary, because it's a treat. The Romans founded it, and in the Middle Ages it was the first stop across the Alps on the road between Venice and Augsburg. (Today it's on the southern end of the Romantic Road driving route.) The former grain market is now Schrannenplatz, where the Markthalle dates from the 1480s. The indoor market hasn't yet surrendered to tourist sausages and postcards. Locals come here to eat Turkish salads and lake fish under the arches, or to have a drink after work.

Along Brunnengasse you can still see winch hooks atop tall gabled houses, once used to haul goods into the attic. Display boards in the courtyard at No. 3 show Fussen's trouble-plagued history of fires and wars. It somehow survived it all. The summer castle of the prince-bishops of Augsburg still stands. The fish market is gone but, rather appropriately, an Italian seafood restaurant called Il Pescatore graces the same cobbled square. In Fransiskanergasse you'll still see 15th-century doors, frescoes and a fountain depicting Fussen's coat of arms.

If you walk out this way past the inner ring of fortifications you'll come across a 14th-century church with an attached cemetery. The tombstones are adorned with old-fashioned German names and trades. Woodcutters and blacksmiths are buried under rambling roses and cedars. Birds twitter. It's a beautiful spot, and seems a very long way from Neuschwanstein.

The terrace nearby outside the Franciscan monastery has an outlook over the town's many layers of rooftops and gables that culminate at the castle. Head downwards first, through the old gate and onto the promenades along the Lech River, passing an old people's home founded in 1465 and still in use. Then start climbing up again through the old town, past the fabulous Benedictine monastery complex that houses the Dance of Death.

Next stop St Magnus Church. The baroque interior has sconces in the shape of dragon heads, a symbol of the devil being tamed by the light of Jesus. Even higher up is the late-Gothic castle with its frescoed Renaissance courtyard and views towards the Bavarian Alps and lush, rolling Allgau countryside.

Down below is Reichenstrasse, old Fussen's main shopping street. It follows the line of the Roman Via Claudia Augusta across the Alps, and finishes in a fine square surrounded by yellow, blue and orange houses. A stone saint gazes over locals soaking up the sun and Franconian wine at cafe tables. Sometimes a Korean or Spanish tourist wanders through, happy escapees from the big white castle down the road.