Prior to European settlement the local Aborigines called the area around Victor Harbor, 'wirramulla'. It was in this area, as early as 8 April, 1802, that Matthew Flinders commanding the 'Investigator' and Nicholas Baudin, the French explorer in 'Le Geographe', came across each other. It was on the basis of this unlikely event that Flinders named the stretch of coastline Encounter Bay. A plaque commemorating this event is located on 'The Bluff' at the southern end of Encounter Bay.
The area of South Australia which now stretches from Lake Alexandrina and the mouth of the Murray River around through Goolwa to Port Elliot and Victor Harbor developed as a unified whole. By 1829 Captain Charles Sturt had made his historic journey down the Murray River and there was a feeling that a settlement should be established near the mouth of the river so that the inland could be opened up.
In 1837 Colonel William Light, responding to this interest, inspected the area around the mouth of the Murray and concluded that the land was poor and the mouth of the river was probably not navigable. The following year Sturt endorsed Light's view that the mouth of the Murray could not be made safe for navigation. Light's assessment almost certainly little more than pure self-interest. There were suggestions at the time that Victor Harbour would make an ideal harbour for the whole South Australian colony. Colonel Light was so convinced that Adelaide was the ideal spot that he looked at Victor Harbor and dismissed it.
This inevitably led to the establishment of Adelaide on Gulf St Vincent but there was still a body of support for the utilisation of the Murray River and a number of proposals (most involving safer harbours and moving goods overland to points further up the river) were suggested.
Eventually a decision was made that Goolwa would become the last point for shipping on the Murray River (it was located on the last bend before the river entered the sea) and there was a debate as to whether Victor Harbor or Port Elliot would be the ocean port. It was eventually decided that Port Elliot was the best location but this was probably based on its proximity to Goolwa and the belief that a canal could be constructed between the two locations. In 1851 it was agreed to build a railway between Port Elliot and Goolwa at a cost of £20,000. It ended up costing £31,000 and wasn't completed until 1854. It was, by any conventional measure, a bit of a disaster. It rarely made a profit and the trains carrying the goods travelled at about 10 km/h and had to be unloaded before the goods could be moved to the ships because the waters at Port Elliot were too shallow and the jetty was not long enough. Add to this the problem of rocks off the shore and the constant battering the area receives from the Southern Ocean and it is easy to understand how, after a decade, the major port activities were moved to Victor Harbor.
Prior to its role as the premier port on the Fleurieu Peninsula, Victor Harbor had already established itself as a major location for the whalers and sealers who plied the waters of the Southern Ocean. By 1837 there was a whaling station on Granite Island and by 1838 Victor Harbour (it had been named by a Captain R. Crozier after the HMS Victor which surveyed the harbour at this time) was already recognised as a port.
The first European settlers moved into the area in 1839. Some lived on the mainland and worked at the local whaling stations. Others took up land and started grazing sheep and cattle. The town was already established, albeit as a rather unimportant little port, when in 1864, after seven ships had sunk off Port Elliot, it was decided to extend the horsedrawn railway from Goolwa to Victor Harbor and use the harbour as the main access point for goods travelling up and down the Murray River. By the 1880s some 25 000 bales of wool from all over western New South Wales and Queensland were being shipped down the Murray, travelling by train from Goolwa to Victor Harbor, and finally travelling to destinations all around the world.
This trade came to an abrupt halt in the 1890s when the railway lines were established and the river traffic died. Today the town is one of the most popular destinations on the Fleurieu Peninsula with families taking the horsedrawn carriage trips out to Granite Island, swimming on the beach, and enjoy the usual array of activities offered by a typical seaside resort town.