Consumer Reports recently conducted one of the most thorough tests ever of antivirus programs. But to really put these security programs through the paces, the magazine hired a firm to create 5,500 new viruses, using them to test the antivirus software products for their ability to detect unexpected threats.
Now antivirus companies are crying foul, saying the magazine ignored a long-standing principle not to invent new viruses.
"Creating new viruses for the purpose of testing and education is generally not considered a good idea,” wrote Igor Muttik of McAfee's antivirus lab on a public company blog this week. “Viruses can leak and cause real trouble." The entry helped touch off a firestorm.
Other antivirus commentators were far more inflammatory, accusing Consumer Reports of being irresponsible.
"The antivirus community has always been very strongly opposed to the creation of new malware for any purpose," wrote John Hawes, the technical consultant at antivirus Webzine Virus Bulletin. "There's just no need for it. Plenty of new viruses are being written all the time, why would anyone in a responsible position want to add to the glut?"
For a very good reason, said Consumer Union's Evan Beckford, who helped run the test. Nearly all antivirus programs do a good job of detecting known viruses. That's easy; and rarely are old viruses the cause of much trouble.
It's the new viruses that cause outbreaks like the LoveBug or Code Red. So antivirus software's ability to detect new, unexpected threats is paramount, he said.
"We need to anticipate how antivirus software will react to future threats. This is the only way we know to do it," Beckford said. "We think the benefits far outweigh the risks."
The viruses were created by paid outside consulting firm Independent Security Evaluators.
Better tests are essential
Malicious programs and recovery from virus attacks cost Americans about $5 billion last year, Beckford said, adding that more in-depth, objective testing of these packages is essential.
Widely respected computer security researcher Alan Paller agrees. As director of Research at independent security training firm SANS Institute, Paller helps thousands of technology professionals prepare for virus outbreaks. He thinks Consumer Reports' rigorous testing was fair and appropriate.
"I think it's extremely valuable because a great weakness of most leading antivirus tools is that they are slow in detecting new viruses," he said. Creating viruses in a lab environment isn't wrong, he added — only distributing them is wrong, he said.
But David Marcus, a security research manager at McAfee, said Consumer Reports was playing with fire by making the new malicious programs.
"I understand...if you want to test a car’s performance, you test the car put on road with lots of bumps on it,” Marcus said. “But when you are talking about malicious code, there's a threat to public. There are professionals who know how to handle viruses. It should be left to them."
Consumer Reports didn't create thousands of new viruses from scratch. Rather, it took a handful of existing viruses and created hundreds of slight variants, changing the malicious programs just enough to evade detection by an antivirus program with a list of known threats.
That's a common trick in the virus writing world; it's standard for a successful virus to inspire dozens of variants.
"In some cases (they were created) using freely available tool kits on the net put out there for virus writers to use," Beckford said. "We did exactly what script kiddies (young hackers) would do."
'Bad things can happen'
In the results, McAfee scored in the middle of the pack. BitDefender and Zone Labs scored at the top, in part for the two program's abilities to detect new viruses.
Marcus denied McAfee's lackluster result motivated the company's criticism of the study.
"The antivirus community is unified...that people should not write viruses," he said. "Bad things can happen. They get out."
The tight-knit antivirus community has spent years developing a set of ethics to deal with the many sticky situations that bubble up from computing’s underground. Universally, companies say they won't hire former virus writers, and they follow gentleman's agreements to share discovery of dangerous programs with each other for the common good.
Still, there are persistent accusations that security firms somehow fund or promote virus writing activity, which is clearly good for business. Such accusations have never been proven, but the emotions they dig up explain part of the antivirus community's knee-jerk reaction to any discussion involving creation of new viruses.
The issue of effective, independent testing continues to be a challenge for the industry, particularly as consumers are faced with an ever-widening array of threats online. In fact, Symantec Corp., which wouldn't comment on the controversy, did say it is holding a symposium on objective antivirus testing methods during the fall in New York.
Disagreement with Consumer Reports over testing methods is nothing new. But creation of potentially dangerous testing by-products is new for the magazine, and its parent, the non-profit entity Consumers Union. Now that the test is over, what will Consumer Reports do with its potentially destructive software?
"Those viruses exist right now only on a CD in a sealed container in a locked cabinet in our computer lab," Beckford said.
That's a CD the antivirus industry will no doubt want to get its hands on very soon.
"It's a good idea that if McAfee and rest of antivirus industry (gets a copy) to make sure consumers are protected," he said.
Of course, if antivirus software worked better, we'd all be protected from those variants already.
Consumer Reports’ September issue, including the review of computer security products, is on newsstands now. A detailed description of its testing methods is available at the magazine's Web site.