The early history of the village is much more impressive than its current size would suggest. The first European party to investigate the district was that of ex-convict John Wilson in 1798.
John Price, from the party, recorded that it was 'most beautiful country, having nothing but fine large meadows with ponds of water in them, fine green hills but very thin of trees'. They climbed what is now called Gingenbullen Mountain to the north-west and Price wrote:'we had a most delightful prospect of the country and in my opinion one of the finest in the world...to the southward a most beautiful country, more particularly to the south-east'.
Over the next decade there were minor expeditions into the district. The Hume brothers, probably in the company of their uncle John Kennedy, investigated the area in 1814. With pastures around Sydney becoming scarce John Oxley drove some cattle into the area the following year.
In 1817 Hamilton Hume and Charles Throsby explored the country west of Sutton Forest. Throsby called the Sutton Forest area 'Cooloorigan'. Throsby returned in 1818 with surveyor-general James Meehan en route to Jervis Bay and again the following year en route to Bathurst. Governor Macquarie awarded Throsby 1000 acres for his efforts and made him superintendent over the construction of a road from Picton to the Goulburn Plains. Visiting the construction site in 1820 Macquarie named Throsby's Bong Bong grant 'Throsby Park'. With him was a working party and six other settlers. One of them, Charles Wright, had arrived in the colony in 1791. After working as a constable at Parramatta he took up the land at Sutton Forest (it now forms part of the Hillview estate) when he was 70 and died there at 93 years of age in 1842.
Governor Macquarie was much impressed with the beauty of the area and the quality of the soil. He named the settlement after Charles Manners Sutton (the speaker of the British House of Commons) on 2 November 1820.
Two emancipated convicts, Edward Shipley and George Sewell, were also given land grants in the area. Shipley, who was 62 years old at the time, established Stonehill (now known as Sutton Farm).in partnership with his wife Mary, another ex-convict, in 1822.
In 1828 James Atkinson of the Oldbury estate recommended the area opposite Charles Wright's Farm, on the Medway Rivulet, as an ideal townsite and the Surveyor-General agreed. Despite instructions to proceed with a town layout the village was not officially established until 1854. However, buildings began to slowly accumulate around an Anglican chapel which was erected in 1829. At the time it was the only church south of the Cow Pastures (the Camden area). A Church of England school was established as early as 1826. A small township began to grow up around three inns which lined the road - the Talbot (1833, renamed the Royal in 1866), the Red Cow (built on George Sewell's land grant) and the Hart (both 1834). The first store also appeared some time in the 1830s. Despite the laying of the foundation stone in 1837 it was 1861 before the All Saints Church of England, designed by Edmund Blacket, replaced the 1829 building.
A visitor in March of 1832 described Sutton Forest as 'a most luxurious spot...There is as much of the English village in miniature about this township as any I know of - a homeliness of scenery that strikes the attention, and induces a second pause to look again at the neat cottages, the snug little church, the light timber with its umbrageous foliage, and the refreshing lagoons on the roadside which incite the weary horse, or bullock to slake his thirst on a sultry summer's day. Nearly opposite the church is the comfortable dwelling of old 'Charley Wright', an old standard in the colony, and one of the first inhabitants in the district, whose hospitality is proverbial'.
In the 1840s it is recorded that a white woman living in Sutton Forest married a local Aborigine who had been baptised - a very unusual marriage in the early history of the colony.
Bushrangers frequented the district from the 1830s to the 1870s. The clergyman's house at Sutton Forest was raided in 1836. One of the most infamous bushrangers, Ben Hall, together with his gang, visited the district in 1865. They stuck up the people at Paddy's River. They robbed the toll collector at Hanging Rock, about 14 km south-west of Sutton Forest then struck again at Kelly's Hotel, about 5 km north of Hanging Rock. With the police in pursuit they withdrew to a vantage point at Hanging Rock and watched the police pass by, then returned to the Paddy's River settlement, gathered everybody there in two inns, forced some musicians to play music, danced till dawn, sent word they were going to Berrima to free everybody from the gaol - then disappeared.
When the railway arrived in 1867 Sydneysiders discovered the bracing climate and beautiful scenery of the Southern Highlands. Sutton Forest, like other towns of the area, became a holiday resort, particularly in the summer months when the humidity hit Sydney. Boarding houses began to appear and wealthy Sydneysiders began to build country retreats.
Among the new commuters were some of the colony's politicians and when eyes were cast about for a country retreat for the NSW governor, Sutton Forest was suggested. The government purchased 'Prospect' in 1882, the country homestead and lands of Robert Richardson, who had the house built after his wedding to a Sutton Forest woman in 1855. Extensive renovations were carried out, at considerable expense, the grounds and interior were greatly improved and Lord August Loftus became the first governor to occupy the house, which was renamed 'Hillview'.
Once Hillview had been established it was commonplace for the governors to travel to Moss Vale by rail. They travelled in a vice-regal carriage and alighted at Moss Vale where there was a special vice-regal waiting room. The retreat was used by all state governors until 1958. It became privately owned but the government re-purchased it in 1986.