Equifax announced late Wednesday that the source of the hole in its defenses that enabled hackers to plunder its databases was a massive server bug first revealed in March.
For the rest of the IT world, fixing that flaw was a "hair on fire moment," a security expert said, as companies raced to install patches and secure their servers. But at Equifax, criminals were able to pilfer data from mid-May to July, when the credit bureau says it finally stopped the intrusion.
"We know that criminals exploited a U.S. website application vulnerability," Equifax said in an update on its website Wednesday night. "The vulnerability was Apache Struts CVE-2017-5638." Equifax said it was working with a leading cybersecurity firm, reported to be Mandiant, to investigate the breach. Mandiant declined an NBC News request for comment.
Related: The One Move to Make After Equifax Breach
The Apache Software Foundation, which oversees the Apache Struts project, said in a press release Thursday that a software update to patch the flaw was issued in March, one day after it was first discovered.
"The Equifax data compromise was due to their failure to install the security updates provided in a timely manner," the Foundation said in the statement.
Equifax said in a statement to NBC News Thursday that the investigation continues, and "we identified the vulnerability on July 29 and took immediate action to stop the intrusion."
In an era where we have iPhones that can scan our face onto a dancing poop emoji, cars that can drive themselves, and robots that are poised to take over most of our jobs, how is it that the largest data holding firm in the world couldn't do its job? How does something like this even happen?
Timeline of a Hack
True to its name, "Apache Struts" is like the struts in your car. It's a version of a popular server software that medium to large companies use as a framework to make it easier for developers to build top-to-bottom custom websites. It can handle everything from interactive screens and logins, to web apps and database management.
Get breaking news and insider analysis on the rapidly changing world of media and technology right to your inbox.
"Apache is one of the most widely used open-source web servers that run on the internet," said Ron Winward, a security researcher for Radware, which provides cybersecurity services for data centers.
But its all-in-one approach and popularity can also be a weakness. Because the software touches all aspects of a company's website, once you break in through one window you can get in to the rest of the house — and the rest of the houses in the neighborhood.
Back in March when the vulnerability was first discovered, it was a virtual feeding frenzy for 24 hours as hackers raced to exploit the flaw to try to take over web servers used by banks, governments, and high-profile internet companies; probing and downloading data; and installing malware and botnets.
Related: FTC Launches Equifax Probe, Websites and Phones Jammed With Angry Consumers
With unfettered access, hackers would have been able to execute commands just like they were the administrators. "They basically control the system," said Brian Markus of Aries Security. That includes sucking off any kind of data they wanted directly from the database.
"If you're someone unpatched and your business relies on Apache Struts," said Winward, "it was a bad day."
The fixes weren't simple either. Because of the nature of the flaw, the repair job might also mean rewiring entire applications instead of just installing a simple software update. Despite the difficulties, within a day, Cisco Talos IT threat analysis team wrote "many of the sites have already been taken down and the payloads are no longer available."
And at Equifax?
"Even if it wasn't patched, you could rely on other devices," said Winward. Under a robust security system, this would include monitoring various logs to detect anomalous behavior in files, database, or suspicious commands being executed, security sources told NBC News.
It's possible that the attacker installed a tool during the frenzy, the hole was patched up, the sources said, but the database download was configured to go so slowly and quietly that it remained undetected in the background.
Related: Massive Equifax Data Breach Could Affect Half of the U.S. Population
But experts say that even under this scenario, the drain of several gigabytes of data on nearly half of all Americans should have been detected by other means. Tools and procedures are readily available off the shelf.
Equifax's apparent ignorance of standard security protocol doesn't even seem to be limited to its U.S.-based operations: Cybersecurity researcher Brian Krebs reported that an online tool used by Argentinian Equifax employees could be accessed just by typing "admin" as the login and password.
"We will make changes," Equifax CEO Richard F. Smith said in a USA Today op-ed. The company has offered affected consumers a year of free credit monitoring and waived credit freeze fees for the next 30 days.
"It's very clear that the credit bureaus, because of the lack of strong oversight authority, have paid some lawyers and their consumer victims go-away money, rather than upgrade their systems and make investments because they don't have to," Ed Mierzwinski, federal Consumer Program Director and Senior Fellow for U.S. PIRG, told NBC News.
"The credit bureaus have never been forced to clean up their mistakes."
Given that the FTC, FBI, and several state attorneys general have launched investigations into Equifax, on top of 30-plus consumer class action lawsuits, that calculus may change.