The drought, a hard freeze and windy weather have all contributed to a 45% drop in California's 2014 table olive crop.
Farmers expect to harvest 50,000 tons, down from last year's crop of 91,000 tons, according to the National Agricultural Statistics Service.
Olive industry officials say the drop in production has been so severe that some farmers may not have enough olives to harvest.
"The crop is so light on some ranches that you may find fruit on one tree, but then you go past two or three trees and you find nothing," said Adin Hester, president of the Olive Growers Council in Visalia. "It is not good for some growers."
Hardest hit have been the growers in Tulare County, the leading olive oil grower. The region produces about 60% of the state's black olive crop.
"I have 30 acres and I am not going to put a crew in there," said Rod Burkett, who farms in the Porterville area. "I could send a crew in and have them pick all day, but they may end up with only half a bucket of fruit."
Burkett said growers were dealt several blows this growing seasons, including a hard freeze in December, wind and a lack of water. The triple whammy seriously stunted the fruit's development.
In some cases, farmers used what little surface water they had to try and keep their citrus trees alive.
"The olive trees won't die because of a lack of water this year," Burkett said. "But it may mean we won't have any fruit wood for next year."
This year's short crop could also benefit black olive importers that have been eating away at the U.S. market for years.
"It just means that the door is open even wider," Hester said. "This is going to be a tough year. It feels like Mother Nature hit us on both cheeks simultaneously."