Oct. 28 — Sen. John Kerry leads President Bush by 49 percent to 47 percent. No, Bush leads Kerry, 49 percent to 43 percent.
Those surveys were reported on the same day, Monday. And they both could well be right.
Public opinion polling is one of the most misunderstood factors in elections. Pundits and the press breathlessly trumpet each new poll, looking for the definitive answer to who’s going to win. But unless one candidate is headed for a landslide, no poll can tell you. Not even after the fact — remember the 2000 Florida exit poll fiasco?
Pollsters and statisticians — even, in their off hours, political reporters will admit it — know this. A poll is a snapshot in time, taken of a sample of Americans who happen to be home by the telephone at the precise time a pollster calls. Almost always, the raw numbers have been massaged and manipulated with mind-bending mathematical formulas — and each pollster uses different formulas — in an attempt to shape a simulation of reality.
Take the first poll mentioned above, by ABC News and The Washington Post. They sampled 2,410 adults, about 86 percent of whom identified themselves as registered voters, of whom 1,631 were determined to be likely voters. From there, even ABC and The Post disagree on how to read the data.
Although they used the same raw data, ABC and The Post applied their own formulas to read the results. The result listed here, Kerry by two points, is the Post’s conclusion. ABC concluded that Kerry led by only one point.
Garbage in, garbage out
In the real world, the difference is meaningless. Both leads were within the poll’s margin of error. Make that margins of error. The Post said there was a 3-point margin of error, which means that if the same questions were asked over and over of the same number of people at the exact same time, either man’s number could be three points higher or lower 95 out of 100 times. (ABC pegged it at 2½ points.)
Conceivably — although, as Bill McInturff, who conducts polls with Peter Hart for NBC News and The Wall Street Journal said, “forget it” — The Post could well find that the same sample preferred Kerry by 52 percent to 44 percent, or Bush by 50 percent to 46 percent.
The average voter, bombarded by polls for several decades now, generally understands this, especially since mainstream news organizations almost always report the margin of error and a little bit of what it means. So that’s fine as far as it goes.
But that goes only as far as the reliability of the raw data. And most pollsters acknowledge that even as data-collection methods have improved over the years, the reliability of the data they collect has likely deteriorated.
“There are tons of things that can impact the accuracy of data that are complicated to control,” McInturff said in an interview. Unless they are rigorously monitored and corrected for, he said, you get “baloney.”
For example, different pollsters use different methods to determine who’s a “likely” voter. And the wording of questions is crucial. So is the order in which they’re asked.
McInturff cited two polls by a major newspaper in February and March. The newspaper — whose identity is being concealed to protect the guilty — reported that Bush was ahead in the swing states by 35 points. Just three weeks later, the same poll showed him leading by only 7 points.
“Someone did a garbage job,” he said.
Can you hear me now? Um, no
For one thing, you can blame modern technology. Polling is “an Industrial Age strategy that hasn’t woken up to Information Age reality,” said John P. Avlon, author of “Independent Nation: How the Vital Center is Changing American Politics.”
The big obstacle is cellular telephones, especially as they become so advanced that people abandon landline phones, Avlon said in an interview with MSNBC TV.
“Pollsters use landline call sheets, the same way they’ve done for the past 20 years,” he said. “Cell phone-only homes aren’t listed. They can’t be accessed.”
McInturff said Americans who relied solely on cell phones were “a significant problem,” representing as much as 20 percent of voters under age 29 in some parts of the country.
“You’re missing a chunk of people,” McInturff said, adding that the problem would only get worse in the next two to four years.
Equally at fault, McInturff said, is the reality that technology allows more people to work longer days or non-traditional shifts. “The number of people who are home is dwindling” when most pollsters call in the early evening, he said.
McInturff said the NBC/Journal poll corrected for these difficulties by making calls over a longer period of time, to catch more Americans and to catch them at home later in the evening. In the future, he predicted, pollsters would move toward using a combination of telephone calls and the Internet to find and screen respondents.
Good news for Kerry?
Avlon contended that those missed voters represented a potentially large “hidden vote” for Kerry.
“We’re seeing a drift in the demographic as we understand it,” he said.
People who use only cell phones and those who work evenings and weekends tend to be younger and to live in urban areas. “As a result, these folks are off the rolls, they’re off the charts, they’re not being called, and they’re not being counted.”
Avlon noted that “young voters give Kerry an edge ... and urban voters tend to vote Democratic traditionally. This is a group that could really swing the election and cause an election surprise.”
“I think with an election this close, every little bit counts, and this is more than a little bit,” he said. “This is millions of voters — it could easily cause an Election Day surprise.”