Chillon Castle, Switzerland: This is the most romantic castle on Earth

Is Chillon the world's most romantic castle? Literary heavy hitters thought so. Genevan philosopher Rousseau started the trend by setting a novel's pivotal scenes below its walls. Flaubert, Goethe, Dickens, Wordsworth and Hemingway enthused over its charms and alpine views. Now Chillon Castle appears on Swiss calendars, a turreted medieval stereotype on a blue lake backed by snowy mountains.

The dukes of Savoy built the first castle in the 12th century and grabbed taxes on the passing transalpine trade. You can see the calculating frames on which they counted the loot. Chillon transformed from grim fortress to elegant residence. Luxuries included latrines plunging into Lake Geneva, and tapestries, and ornate four-poster beds. Mantled chimneys were a double-edged innovation. Nobles who once huddled in the great hall with their household retreated into fire-warmed rooms, encouraging a widening social gap.

Chillon supplies a more interesting history than its saccharine appearance suggests. The plain, brooding interior doesn't present some hokey pseudo-medieval romance, but rather the uncomfortable, austere life of the Middle Ages. A dangerous life, too. Despite the embellishments, the castle was fundamentally a military base. You get a feel for the cruelty of the Middle Ages in Chillon's displays of war machines, armour, battle-axes and – another innovation of the time – the new-fangled crossbow famously associated with Swiss freedom fighter William Tell.

Technological advances would make castles obsolete. In 1536 a Bernese army used cannon to storm the castle. Chillon's days were done until Romantic writers revived its glory. In particular, Lord Byron made Chillon famous after holidaying on Lake Geneva in 1816 and penning the lengthy poem The Prisoner of Chillon, which became a rallying cry for liberty across Europe.

You can see Byron's name – along with the graffiti of other delinquent visitors – etched into a dungeon pillar where François Bonivard, the reforming prisoner of his poem, was purportedly chained by the evil Catholic duke of Savoy. The dungeon, neither dark nor damp, has pretty Romanesque arches and openings onto glittering lake water.

"Bonivard's dungeon was a nice, cool, roomy place, and I cannot see why he should have been so dissatisfied with it," noted Mark Twain in A Tramp Abroad. Somerset Maugham suggested it should be turned into a gymnasium.

The real Bonivard was a Genevan abbot who neglected his religious duties and collected mistresses and four wives. He was imprisoned in Chillon in a suite of rooms in deference to his rank, but four years of house arrest taught him no lessons. He soon found himself before the religious courts again for gambling, sleeping in church and wearing flowers behind his ear.

The original man sounds more interesting than the noble hero of the Byronic poem, but writers rarely let facts distract from purpose. You could say the same about tourism. Chillon Castle is a romantic symbol of a supposedly chivalrous age, but the best thing about it is the way it combines our expectations with the era's gritty reality. It's a beautiful place with a dark heart and lively tales to tell, and the more appealing for it.