In 1858, Samuel Brannan — the man who shouted "Gold! Gold on the American River!" on the streets of San Francisco and became the first Gold Rush millionaire — paid $1,500 for lumber salvaged from a shipwreck off the coast at Ocean Beach. It's said that he used the wood to build the first Cliff House on the cliffs at Point Lobos in San Francisco. Other sources say the first Cliff House was constructed a few years later by a man named Captain Junius G. Foster.
Either way, in the 1860s, the restaurant became a Victorian tourist hot spot, a destination for Sunday horse and carriage riders day-tripping from the city, venturing west on a toll road that would later become Geary Boulevard.