New York Gov. Andrew Cuomo on Thursday offered what appeared to be a surprising statistic on just how broadly the outbreak of the coronavirus has hit the state: Almost 14 percent of people tested in a new study were found to have antibodies to the disease it causes.
It might be good news. The data suggest that the state's mortality rate could be lower than previously thought, albeit much higher than for seasonal influenza. But it's still too early to reach that conclusion — or any other extrapolations people might want to make about easing lockdowns, experts say.
Beyond that data point, Cuomo's presentation offered little detail that would allow a full assessment of whether that many New Yorkers really were infected. The numbers were from a preliminary analysis of a study that wasn't released publicly, so most of its methodology remains unknown. The state Health Department didn't respond to a request for a copy of the study.
That doesn't mean the findings are wrong. But it does mean that extreme conclusions about the mortality rate or the state's ability to reopen shouldn't be drawn from the data.
"There's a risk of really serious misinterpretation here," said William Hanage, an associate professor of epidemiology at Harvard University's T.H. Chan School of Public Health in Boston. "But the most basic conclusion — that quite a large number of people may have been infected and are not turning up in the official case counts — that's extremely plausible and something we have been suspecting all along."
Full coverage of the coronavirus outbreak
The confusion highlights just how little is still known about the coronavirus, as well as the difficulties of doing difficult scientific work under the lens of a global microscope. Researchers are still piecing together important details about the virus, including just how fatal it is.
Understanding how many people have been infected is critical for public health officials to gauge when it might be safe for states to begin easing strict social distancing measures. Together with expanded diagnostic testing and contact tracing, the results of antibody tests can help states contain the spread of the virus and prevent local health care systems from being overwhelmed with subsequent waves of infections.
That has made stats like those offered up by Cuomo into crucial pieces of news — but also susceptible to confusion and miscalculation. Experts caution that preliminary findings from the study, as well as similar research that was released in California in the past week, could be skewed by statistical issues, unreliable test results and questionable ways the studies were designed.
The data are particularly crucial as a national debate has emerged over when and how to begin to ease lockdowns, with some conservative pundits pointing to prevalence tests like these as proof that the U.S. has overreacted to the outbreak. Public health professionals have been nearly unanimous in disagreeing.
The New York study found that out of 3,000 grocery store customers who were randomly tested across the state, 13.9 percent were positive for coronavirus antibodies, indicating that they had an active infection or had been infected and recovered. In New York City, 21.2 percent of those tested were positive for antibodies, according to Cuomo, who presented the preliminary results of the state's study Thursday in a news briefing.
If those infection rates are correct, it could mean that more than 2.7 million people in New York — and about 1.8 million people in New York City alone — have been infected, far more than the state's 257,216 confirmed cases.
Although the early results suggest that infections in New York have been much more widespread than the official counts indicate, experts say that getting a clear picture of the true number of coronavirus cases is extremely complex and that it's still not known whether, or how, antibodies translate into immunity from the virus.
"Outbreaks are like an iceberg, where the top part that we can see are the people who are hospitalized or die, but there's a big bottom part made up of people who were missed," said Natalie Dean, an assistant professor of biostatistics at the University of Florida in Gainesville. "Establishing how big that bottom part is helps us characterize the severity of the disease."
But some have called into question the accuracy of some antibody tests and how the early studies have accounted for the possibility of false-positive results. The New York City Department of Health put out an alert Wednesday warning about false negatives and false positives for antibody tests.
The tests used in New York were developed by the Wadsworth Center, a public health laboratory of the state Health Department. In a fact sheet provided by the agency, the test's specificity — as in its ability to identify antibodies specific to this coronavirus rather than coronaviruses in previous outbreaks — is 93 percent to 100 percent. The test's sensitivity, or its ability to deliver accurate positive results, wasn't listed.
Carl Bergstrom, a professor of biology at the University of Washington, said the early results of the New York study seem reasonable, particularly given the number of deaths recorded in New York City. He added that the findings' plausibility is bolstered because the testing occurred in a particularly hard-hit region of the country. As a result, the probability that a positive test truly reflects that someone has the disease rather than being a false positive is higher.
"If you're using an error-prone test in a low-incidence area, you're not getting much information," Bergstrom said. "But if you're testing a population with 20 percent infected, like in New York City, the chance that any particular positive is a true positive is much higher."
But Florian Krammer, a professor of vaccinology at the Icahn School of Medicine at Mount Sinai in Manhattan, questioned the infection rate for New York City, saying it seemed too high.
"I would think 6-8%, maybe 10% are closer to the truth. It would be nice to know more about the test, its sensitivity and specificity and the test population," he tweeted in response to Cuomo's presentation.
Accuracy concerns similarly plagued a pair of California-based antibody surveys, both of which have yet to undergo the standard peer-review process that is essential to any scientific research. One of the studies, led by researchers at Stanford University, suggested that coronavirus infections in Northern California could be up to 85 times higher than official counts. And another study, led by researchers at the University of Southern California, suggested that the true number of infections in Los Angeles County could be up to 55 times higher than what has been reported.
The California studies ignited fierce debates within the scientific community, with some scientists claiming that the authors didn't adequately account for the rate of false positives in their tests, which likely influenced the findings.
Some scientists also criticized the the Stanford survey's methodology, claiming that the way people were recruited for testing — through ads on Facebook — introduced biases likely to have skewed the results.
Although surveying shoppers at grocery stores may be more randomized than recruiting through social media ads, there are still questions about how much the New York study truly represents the population.
"These are people who were out and about shopping," Cuomo said Thursday. "They were not people who were in their homes. They were not people who were isolated. They were not people who were quarantined."
Download the NBC News app for full coverage and alerts about the coronavirus outbreak
Still, antibody surveys are a valuable way for public health officials to grasp the scale of an outbreak. If researchers know the true number of coronavirus cases, for instance, they can also more accurately pin down the fatality rate of COVID-19, the disease the virus causes.
In New York, the fatality rate based on the number of official cases was more than 7 percent, but that number was long thought to have been inflated because diagnostic testing shortages meant only the most serious cases were being confirmed.
If the infection rates based on the state's antibody testing hold true, New York's mortality rate would be closer to 0.5 percent to 1 percent — still higher than, for example, the estimated 0.1 percent mortality rate in the U.S. from the seasonal flu. However, Cuomo qualified that the calculations were based on the state's more than 15,500 reported deaths in hospitals and nursing homes and didn't include potential at-home deaths.
While a lower-than-predicted mortality rate is welcome news, there is a danger that the adjusted figures could cause people to underestimate the severity of the disease, according to Hanage.
In New York, for example, even if the mortality rate ends up being lower than 7 percent, the coronavirus has exacted a serious toll.
"The number of deaths in New York is already considerably more than you would expect if every single man, woman and child in the city had been infected with the flu," Hanage said. "If you then think about how, without any countermeasures, it would eventually infect 70 or 80 percent of the population, then hopefully people start to understand that this is still very dangerous."
