There was no denying the scope of it. When the latest online virus hit the University of Wisconsin, Madison, chief information officer Annie Stunden rallied her troops and decided to shut down the school’s external e-mail. But she knew things were truly mucked up as she waited for a press release about the mess that was e-mailed to her from across campus. After 20 minutes, it still hadn’t arrived. “And that,” she says, “was after we’d been dealing with it for hours.”
For folks who keep computers running, and for we who depend on them, this has been the summer of discontent. IT departments had finished dealing with the proliferative Sobig.E virus last month when August came along. Last week brought the Blaster worm, which hammered away at a recently discovered Windows security hole, and then the Welchia worm, which forced computers to download an anti-Blaster patch but caused its own sort of trouble. Though most workplaces were prepared, Blaster often wormed into individual PCs and made for extra work as tech departments chased it down.
Then came this week.
Sobig.F appeared on schedule just after Sobig.E expired, but this time the volume of mail was unprecedented. And while antivirus software could usually catch Sobig’s nefarious little gifts, the sheer volume of mail left servers wheezing and created long lag times for employees trying to check their accounts. IT departments had to reboot machines and clean out chock-full mail queues.
“It pretty much affected everybody,” says Joe Stewart of Internet security firm Lurhq. “Whether they got the infection or not, they had to spend some time dealing with it.”
Though few employees actually launched Sobig on their machines, it still did its damage in lost productivity. At thousands of companies, employees lost an hour or two as tech staff cleaned things up. Rarely catastrophic, but the net effect of a thousand tiny bites was often frustration.
Or exhaustion. Stunden’s staff had been working since last week to fix Blaster when they had to shift gears and shore up defenses against Sobig on the school’s 60,000 computers. “They were burning the midnight oil for two days,” says Stunden. “But, you know, two days isn’t bad.”
Part of the problem, she says, is that university systems often lack the controls of corporate systems. Professors and students often don’t want to be told how to manage their personal PCs.
Even with more stringent protections, many IT departments were taken aback by the sheer vigor with which Sobig.F proliferated. “This one snuck up on everybody,” says Timothy Ruland, who’s in charge of computer security at the U.S. Census Bureau. “I think everybody was still focused on Blaster. ... Nobody saw this coming at all.”
Gauging the impact
The overall impact of the two viruses was harder to assess, much like putting a price tag on, say, a massive blackout. Security firm MessageLabs found at times that Sobig was in one of every 17 e-mails. Stunden said only one or two of every 20 messages she got were legit, similar to users whose e-mail addresses were appropriated by the virus.
At the San Diego Data Processing Corporation, which handles the city’s computer systems, staff members spent an extra 50 hours scrubbing viruses from 30 infected computers, despite strong security. That’s in addition to the hour or two they spend every day weeding out viruses.
Blaster has affected an estimated 500,000 machines. Dollar figures were harder to figure out; one estimate suggested $50 million, others placed it in the billions for both viruses.
But the impact was clear, especially for smaller businesses, which frequently lack the resources to keep up with an endless string of antivirus updates and rely on just a few mail accounts. Not everyone was hit by both glitches, but even one could be enough to create frustration.
Though she updates her antivirus software every night, Annie Lambert, associate editor at Performance Horse magazine, still got hit by Blaster on both her laptop and desktop computers. She has spent 50 hours over the past two weeks, exasperated, restoring them to working shape. “Most of the day I’ve just been sitting on hold or talking on e-mail with some tech,” she says. “It’s just time consuming.”
Home users were equally frustrated. Even the most well-prepared had to take time to update their virus protection, and Sobig’s firehose of e-mail left many with a spam-filled inbox no matter what they did. And if they were still on dial-up? Forget it. Suddenly those dust-covered typewriters started to take on a certain appeal.
Part of the problem was that anyone who had their e-mail address posted publicly on the Web stood the risk of having it spoofed by Sobig as its “From:” line. Even if no one in an office opened offending e-mails, system administrators were still hit with waves of incoming replies from messages never sent by employees.
And while antivirus programs were diligently standing guard, their automated responses zinged back and forth between mail servers. They were dutifully reporting back to senders that their mail was infected and had been zapped — but the sender had never actually sent the message. The autoresponses, which often send back the original message as an attachment, added to the e-mail pile and could gum up already strained servers. “With Sobig it’s been a real problem,” says Stewart.
Providers keep it together
It wasn’t fatal, but Net access providers often witnesssed more bandwidth being chewed up by Sobig’s junk traffic. Though e-mail is not data-heavy, frustrated servers often started spewing out extra requests for Internet routing, and the virus attachment added kilobytes to each of millions of e-mails.
Major service providers like AOL and Yahoo insisted their servers were only slightly nudged by Sobig. (MSN refused to discuss the impact on its Hotmail servers.) As AOL described it, perhaps one-tenth of the overall spam they received was attributable to the virus. Yahoo spokeswoman Mary Osako said there was a “very minimal impact” on servers, in part because Web mail services can’t be used to send back virus mails even if they’re opened.
Some services, like access provider Covad, split off networks that provide online access from those that handle mail services. But when users wanted their e-mail, the struggling servers could hamper them. “Once that happens to those machines, it bogs down the system and real traffic can’t get through and real people doing real work cant get through,” says Anh Tran, Covad’s director of network services.
But if viruses and spam have become a way of life for larger providers — staffs like Tran’s already had abuse departments in place — it could create a more profound strain on smaller ones.
“It’s not necessarily Internet bandwith, it’s human bandwidth that’s been affected,” says Jim Comeaux, chief technical officer of California’s Redwire Broadband. Redwire called in extra help to supplement their 22-person staff manage straining servers, and spent long hours on the phone with frustrated customers. Though everyone was working extra hard to fix problems, Comeaux says, all that time spent bringing things back to normal ultimately means less time to spend on keeping their small business in the black: “Are we not getting things accomplished that are revenue-dollar generating? Sure.”
Companies quiet
Big companies were often hesitant to discuss any problems with systems, either out of concern for their reputations or concern they would be specifically targeted. Many insisted that it was business as usual, even for their IT departments.
“I can’t really say that it’s busier in terms of these worms and viruses,” says IBM Canada spokesperson Mike Quinn, whose company jumped into action when a major client, Air Canada, lost reservations and check-in systems this week to the Welchia worm. Things were back to normal Wednesday, but even Quinn acknowledged it was the latest in an almost surreal string of setbacks, that included recovery from last week’s blackout: “It’s not really a normal work week here to begin with.”
Some, though, quickly admitted setbacks. CSX informed reporters that a computer worm had crippled the railroad’s signaling systems Wednesday along the East Coast, and halted some commuter trains in Maryland. Online research firm Jupitermedia went a step further, sending a release to say that it was a victim, not an instigator, of the spam anyone might get from its addresses.
'Critical things can be affected'
Ruland, of the Census Bureau, found his agency mostly unscathed, in part because bureau employees must follow stringent security procedures to protect confidential data. His firewalls blocked most outgoing spam, but because many Census employees have public e-mails, they found full inboxes. It was a lot of work, he says, but never reached crisis mode.
Still, he’s waiting for Sobig.G to appear next month when Sobig.F expires. And he too sees the past couple weeks as a wake-up call — perhaps a sign that viruses have evolved past the point of frustrating nuisance. “It’s now gone beyond the stage where people can just say, ‘Well, we survived this one,’ ” he says. “They’re beginning to see that critical things can be affected.”
Worse still, the next big virus is likely to be completely new and devastating. As experts pointed out this week, Sobig’s current iteration seemed to be from spammers hunting vulnerable machines they could exploit. With viruses being used for commercial interests, network administrators worry that hackers who want to show off their skills — or online terrorists — may be hatching something truly devious. Each new development is a learning experience, but only after the damage has been done. And the stakes keep getting higher. Sunden recalls when the first Morris virus emerged in the late 1980s, and major universities watched as a tiny, rudimentary worm unraveled what was then a very nascent Internet.
“It was pretty exciting back then. It was the first time we’d seen anything like that happen,” she says. “It’s not fun anymore.”