Oldsmar, Florida, experienced one of the biggest fears in cybersecurity Friday — hackers looking to poison its water supply.
It's the kind of breach that has been warned about for years but is rarely seen. Experts say the hack, which was addressed quickly, is a prime example of why the cybersecurity of the U.S. water supply remains one of the greatest risks to the country's infrastructure.
And like the U.S. election system, it tends to be a sprawling and varied challenge.
"Water facilities are particularly problematic," said Suzanne Spaulding, who was the chief cybersecurity official at the Department of Homeland Security during the Obama administration. "When I first came into DHS and started getting the sector-specific briefings, my team said, 'Here's what you've got to know about water facilities: When you've seen one water facility, you've seen one water facility.'"
The U.S.'s 54,000 or so drinking water systems are run independently, by either local governments or small corporations. That means there are thousands of different security setups, often run by generalists who are responsible for the technology of their particular systems.
"I've been to numerous water treatment facilities where there is one IT person or two IT people," said Lesley Carhart, a principal threat analyst at the cybersecurity company Dragos. "And they have to handle everything from provisioning computers and devices that keep the infrastructure running to trying to do security.
"Most are very conscious of it, but they're just drowning," she said. "They don't know how to accomplish all the things they're required to do to both keep things running from an IT perspective and also fill compliance checkboxes."
All of Oldsmar's cybersecurity services, including the water treatment plant's, are managed by one man, City Manager Al Braithwaite, Assistant City Manager Felicia Donnelly said in an email.
In the case of the Oldsmar attack, all the hackers needed to gain access was to log in to a TeamViewer account, which lets remote users take full control of a computer, that was associated with the plant. That let them open and toy with a program that sets the chemical content for the underground water reservoir that provides the drinking water for nearly 15,000 people. The facility has backup alarms to measure unsafe chemical levels, but the hackers were at least briefly able to order the plant to poison the water.
With a few clicks, they told it to raise the levels of lye in the water from 100 to 11,100 parts per million. Anything more than 10,000 can lead to "difficulty swallowing, nausea/vomiting, abdominal pain, and potentially even damage to the gastrointestinal tract," Dr. Kelly Johnson-Arbor, a medical toxicology physician at the National Capital Poison Center, said in an email.
Bryson Bort, a cybersecurity consultant who helped start ICS Village, a nonprofit that raises awareness of cybersecurity for industrial systems, said such a practice — setting up a computer program to allow users to take control of sensitive industrial systems — is extremely common in industrial systems that don't have the means to employ staffs of experts to be on call at all hours.
"If you think about it, you have a challenge both technically and resourcewise with being able to manage things," he said in a phone interview. "So the ability to get an alert light at 3 a.m. and get that one expert has value. People are always mystified that this is the way it is, but this is the way it is. It's the convenience of these resource constraints. You don't have a choice."
Download the NBC News app for breaking news and politics
Foreign government-sponsored hackers regularly target U.S. industrial systems, which often are labyrinthine enough that a simple intrusion usually doesn't let them shut down infrastructure. It's unclear who or what was behind the Oldsmar hack.
Federal officials have long fretted about a potential "cyber Pearl Harbor" incident, in which hackers could physically damage American infrastructure. While that hasn't happened, the U.S. is eager to push back when an adversary country gets too close.
In 2013, a hacker broke into computers that controlled Bowman Dam in Rye, New York, and could have gotten access to its controls if it hadn't been offline for maintenance. Three years later, the Justice Department charged an Iranian national with the hack, saying he worked for a company tied to the Iranian Revolutionary Guard Corps.
And last year, the Treasury Department sanctioned a Russian government institution suspected of having created a powerful, destructive program called Triton, which targets industrial systems.
There's no public evidence that an American company had been seriously harmed through Triton. But that doesn't mean those countries' hackers don't try to exploit the open holes in American infrastructure, Carhart said. It means they know better than to cause cavalier damage.
"The foreign state hackers are there. They are in the water utilities, I promise you. But they know better than to poke buttons today," she said.
"They're going to wait until they've got a really good reason to poke buttons. They're there. We find them all the time."