The first known white residents at Twenty Mile Hollow were an ex-convict, named William James, and his wife. They lived in a crude stone and slab hut from which they illegally sold alcohol. An Irish ex-convict named Thomas Pembroke, together with his family of seven, was granted two acres at Twenty Mile Hollow in 1831. He established an inn shortly thereafter, which later became known as 'The King's Arms'.
In 1855 it was decided that the police presence at Weatherboard (now Wentworth Falls) be moved closer to Penrith and a police paddock and lock-up were established at Twenty Mile Hollow. They were located just up the hill from the inn, adjacent the present highway, between Arthur St and Woodbury St.
Another ex-convict, named William Buss, purchased the King's Arms in 1855 and it thence became known as 'Buss's Inn'. A popular watering hole, it served those traveling west to the goldfields.
The railway passed through in 1866-67 and the original railway station was known as Buss's Platform. However, Buss died in 1867 and the railway probably detracted from the road traffic and thus retarded custom at the inn. At any rate, in 1868 Buss's widow sold the building to Alfred Fairfax who renamed it Woodford House. Consequently the railway station was renamed Woodford which became the official name of the settlement.
Professor Edgeworth David, who taught geology and physical geography at the University of Sydney, and who accompanied Mawson and Shackleton on their trek to the Antarctic, built a house in Woodbury St in 1898. His book, The Geology of Australia, was regarded as the central geological text book of its era.