In Calhoun County, Port O'Connor is located southeast of Victoria along Hwy 185 on the Gulf of Mexico.

Hurricane Carla in 1961, which came ashore at Port O'Connor, leveled it, destroying the five-block downtown area. Growth resumed after the disaster, however, because of the community's proximity to Matagorda Air Force Base.

The town's population reached 1,010 in 1969. From 1975 to 1988 it was reported as 810. The number of businesses reached a high of twenty-seven in 1981.

In the late 1980s the town was a fishing, tourist, and retirement center, and in 1990 it reported a population of 1,184 and twenty-two businesses.