In the ongoing quest to better anticipate earthquakes, scientists embarked on an ambitious experiment: Identify the likeliest places where magnitude-4.9 quakes or stronger would occur in seismically active California over a five-year period.
Half a dozen teams decided to give it a shot. They developed sophisticated computer models, submitted their best guesses and waited. As part of the ground rules, they could not change their forecasts, which were checked against actual quakes that hit during the study period.
The goal was to see whether there was a reliable way to flag a seismic hotspot before the ground shakes.
The exercise, which began in 2006 and wrapped up last December, was not aimed at predicting quakes, which remains elusive. Instead, researchers were asked to pinpoint regions where quakes were more likely to occur based on past seismic history, activity on fault lines or other factors. The state was divided into 7,700 sections and teams had to give probabilities of quakes of varying magnitudes occurring in each one.
During the test period, 31 quakes larger than magnitude-4.9 rattled the state, including the greater Los Angeles region, San Francisco Bay area and off the Northern California coast. The largest was the 2010 Easter earthquake centered in Baja California that produced sizable aftershocks along the U.S.-Mexico border.
So how did scientists do?
"No single model takes home all the gold," said seismologist John Vidale of the University of Washington who was not part of any team.
U.S. Geological Survey seismologist Sue Hough agreed. The experiment shows just "how difficult it is to even evaluate rigorously the success of prediction methods," she said.
By one measure, three groups that took into account all past quakes regardless if they were big or small fared better than the rest. One of them analyzed the performance and published results online Monday in the Proceedings of the National Academy of Sciences
Some scientists questioned whether a team that has a stake in the game should be a referee. One of the study's leaders, Donald Turcotte of the University of California, Davis, said it is unavoidable given the tight-knit earthquake science community.
"It is impossible to find people with expertise who are not participants in the experiment," he said.
Jeremy Zechar of the University of Southern California, who is heading his own review, takes issue with the yardstick used by Turcotte's team to gauge success. It was developed halfway into the experiment and was not part of the original criteria agreed to by the participants.
Seismologist David Jackson of the University of California, Los Angeles, likened the experiment to kiddie soccer.
There are "no official winners or losers, but plenty of scorekeeping from the sidelines" to learn the strengths and weaknesses of various quake theories, Jackson said.
While scientists cannot say with certainty exactly when and where the next quake will strike, they have an idea of how quakes behave. For example, a big quake will produce smaller aftershocks in the same area. There's still a lot that's unknown such as whether quakes too small to be felt play a role in increasing the chances of a bigger quake.
Geophysicist John Ebel of Boston College was not surprised that his work ranked low in the study because he made simple assumptions.
"There's still a lot we don't understand," Ebel said.
Since the California experiment ended, scientists are applying lessons learned to a similar international effort currently under way.