FAA outage: Damaged database file took down safety system, grounding flights

The U.S. aviation warning system that crashed for more than an hour Wednesday traces its origins to ocean-faring ships and has been under continuous reforms for years, experts say.

At least one aviation industry group has called for it to be replaced altogether.

The Federal Aviation Administration grounded all flights blaming an unspecified failure in the Notice to Air Missions system. NOTAM issues a near-constant stream of acronyms and abbreviations to alert pilots to a host of potential dangers, from parachuters and bad weather to legal airspace restrictions and flocks of birds.

By Wednesday evening, the agency had pinpointed the problem as a damaged database file, and there is no evidence of a cyberattack, it said.

Regardless of the cause, the NOTAM system has long been a source of frustration for pilots and others in the aviation industry, who say it overloads them with information that’s irrelevant to their flights and makes it difficult to identify actually useful information.

Read the full story here.

More than 1,300 flights were delayed and nearly 100 were canceled Wednesday afternoon after a corrupted file knocked out a government system that provides pilots with critical information, halting flights across the country overnight.

Flights resumed Wednesday morning. According to the tracking site FlightAware, Denver International Airport led the country with delays, at 111.

Charlotte Douglas International Airport in North Carolina was second, with 95.

The average departure and arrival delays were under an hour at both airports, according to the site.

Nearly 10,000 flights had been delayed across the country Wednesday, according to the site. More than 1,300 had been canceled.

A corrupted file affected both the primary and backup systems of the Federal Aviation Administration’s Notice to Air Missions system, a senior government official said. 

The failure of the critical system prompted a ground halt at airports across the country early Wednesday before passengers slowly began boarding flights again.

It isn’t clear how the file was corrupted. An investigation continues.

Transportation Secretary Pete Buttigieg told NBC News' Andrea Mitchell that "there is no direct indication of any kind of external or nefarious activity, but we are not yet prepared to rule that out."

He added that officials are working "to see exactly what was going on inside the files that were in the system, leading to this irregularity."

"This is an incredibly complex system," Buttigieg said later in the interview. "So glitches or complications happen all the time, but we can’t allow them to ever lead to this level of disruption, and we won’t ever allow them to lead to a safety problem."

The Transportation Department forced airlines to pay out millions of dollars in refunds last year for canceled and delayed flights, but Buttigieg stopped short of agreeing that it should refund travelers after the FAA outage when Mitchell posed the question.

"We’re not for-profit companies selling tickets that the way an airline is," he said. "Our responsibility is to make sure that everybody is safe, and we’re always going to err on the side of safety. ... When there’s an issue on the government side of the house, when there’s an issue in FAA, we’re going to own it, we’re going to understand it, and we’re going to make very clear what’s needed in order to fix it and go after that plan."

The largest pilots union in North America encouraged travelers Wednesday to be patient after the nationwide ground halt on flights as data from the tracking site FlightAware.com showed ongoing delays.

“We are in regular contact with the Federal Aviation Administration and will continue to work with them and airline managements to ensure our aviation system continues to be as safe as possible," the Air Line Pilots Association International said in a statement.

The group represents 67,000 pilots at 40 U.S. and Canadian airlines.

FlightAware showed delays of as many as 48% of flights for Southwest Airlines, 44% for American Airlines and 38% for United Airlines and Delta Air Lines.

Meanwhile, a spokesman for the American Airlines pilots union said he respected the decision to issue the nationwide stop, as it was most likely necessary to ensure safe travel.

Still, "this shouldn't be happening," said Dennis Tajer, a spokesman for the Allied Pilots Association. "This is a learning moment — but we have to get those moments down to a minimum."

Performing plastic surgery was on the line if Dr. Robert Feczko didn’t make it home to North Carolina. 

His ski trip to Colorado ended as his Delta fight from Denver International Airport to Atlanta was delayed two hours. Complicating matters, Feczko, 37, would most likely miss his connecting flight to Raleigh, North Carolina, and push back the final stretch of his trip — a 90-minute drive home to Greenville, North Carolina.

Feczko said that he wanted to feel fresh to perform surgery Thursday morning but that before he left Colorado, he wasn’t sure whether he’d even make it home.

“I’m more concerned about my flight routing through Atlanta,” Feczko said, waiting in line to check his ski equipment. “I’m worried about tomorrow. I have a surgery at 7:30 a.m.”

Canada's air navigation service provider, Nav Canada, was experiencing the same computer system outage the FAA did, it tweeted shortly after 12:30 p.m. ET.

"NAV CANADA's Canadian NOTAM entry system is currently experiencing an outage affecting newly issued NOTAMs, and we are working to restore function," the statement read. "We are not currently experiencing any delays related to this outage. We are assessing impacts to our operations and will provide updates as soon as they are available."

Vanessa Adams, a spokesperson for NAV Canada, said in a statement that the outage began at about 10:20 a.m. ET and that power was restored at about 1:15 p.m.

"We are still investigating the root cause of the failure," Adam said. "At this time, we do not believe the cause is related to the FAA outage experienced earlier today."

As of about 3 p.m. ET, both Toronto Pearson International Airport and Montréal-Pierre Elliott Trudeau International Airport had average departure delays of 28 minutes (and increasing), while Ottawa International Airport had average arrival delays of 32 minutes.

Earlier Wednesday, Air Canada tweeted that all customers traveling to and from the U.S. should check the status of their flights on its website after the FAA outage.

More than a quarter of Air Canada flights — 123 — were delayed as of 3 p.m. ET, FlightAware showed, while 13 were canceled.

At Denver International Airport, the computer outage delayed Mine Mizrak’s Southwest flight to Los Angeles and forced her to miss her connecting Turkish Airlines flight to her native Istanbul, where she planned to reunite with family.

Mizrak, a mechanical engineer, moved to Denver last year, leaving behind her mother and other relatives, whom she hasn't seen since.

“I’ve been looking forward to this for the past month,” she said, sitting down in a chair counting the time until her next flight departs. Once she does, her trip will become pricier.

While Mizrak, 25, paid $1,000 in airfare, she said, she’ll have to pay an additional $500 once she lands in California to ensure she gets home because of her missed flight at LAX.

“I’ve been crying, because I could’ve spent that money on something else in Istanbul,” Mizrak said. She said Turkish Airlines won’t reimburse her for the money because the outages didn’t affect international flights.

A Turkish Airlines spokesperson didn’t immediately respond to an inquiry.

American Airlines and Delta Air Lines resumed operations late Wednesday morning after the FAA outage, although with significant delays.

At 10:45 a.m. ET, more than 950 American Airlines mainline and regional flights had departed, according to information provided by the airline. 

As of 12:30 p.m. ET, just under 1,200 American Airlines flights were delayed, amounting to 40% of its flights, and more than 180 had been canceled, amounting to 6% of its flights, FlightAware reported.

In a statement, American Airlines said it was continuing to “closely monitor” the FAA outage and was “working to minimize further disruption to our customers and operation.” Customers whose flights were affected by the outage could rebook their travel for Wednesday and Thursday “without any additional fees,” the airline said, directing travelers to its website or its app for the latest flight information.

Delta was slightly less affected than American, with just over 1,000 flights, or 35%, delayed and 58, or 2%, canceled as of 12:30 p.m. ET, according to FlightAware.

Nearly half of Southwest Airlines flights were delayed as of about 11:15 a.m. ET, according to FlightAware, making the already beleaguered carrier one of the most affected by the morning's outage.

More than 1,700 Southwest flights were delayed, amounting to 45% of its overall flights.

The latest delays come just weeks after the airline canceled thousands of flights in the travel-heavy days after Christmas, which it blamed on "operational challenges” following days of severe winter weather.

In a statement provided to NBC News, Southwest Airlines spokesperson Dan Landson said the airline anticipates "some schedule adjustments will be made throughout the day."

"As always, we encourage Southwest Customers to check their flight status at Southwest.com or via our mobile app," Landson continued. "We’ve also posted a Travel Advisory on our website to highlight the flexible rebooking options being offered to Customers."

Ground stops have been lifted at Chicago's O'Hare and Midway international airports, the Chicago Department of Aviation said in a statement shortly after 10 a.m.

"Residual delays or cancellations will likely continue throughout the day as a result of this morning’s outage," it said, adding that travelers should continue to check their flight status before heading to the airports.

As of about 10:45 a.m. ET, O'Hare was reporting average arrival delays of an hour and 16 minutes (and decreasing), and average departure delays of an hour and 39 minutes (and increasing), according to FlightAware. More than 200 of its flights — 23% — were delayed, and 39 flights were canceled.

Midway was experiencing average arrival delays of 48 minutes (and decreasing) and departure delays of an hour and 51 minutes (and decreasing), according to the tracking website. More than 40% of its flights — 116 — were delayed, and 22 were canceled.

Wednesday's FAA computer outage was a "catastrophic system failure" and "a clear sign that America’s transportation network desperately needs significant upgrades," according to the head of the U.S. Travel Association.

"Americans deserve an end-to-end travel experience that is seamless and secure. And our nation’s economy depends on a best-in-class air travel system," said Geoff Freeman, the president and CEO of the nonprofit and advocacy group representing more than 1,100 member organizations in the travel industry.

"We call on federal policymakers to modernize our vital air travel infrastructure to ensure our systems are able to meet demand safely and efficiently,” he added in a statement.

A senior law enforcement official told NBC News that the FBI has seen no evidence that a cyberattack caused the computer outage that grounded thousands of flights.

Cyber security experts say the most common cause of problems like the one Wednesday is a bad software update.

Most people will never have heard of "NOTAM," but it is the reason thousands of travelers were stuck in airports or stewing over delayed or canceled flights Wednesday.

The acronym stands for "Notice to Air Missions," and refers to the computer system that distributes "information essential to personnel concerned with flight operations but not known far enough in advance to be publicized by other means," according to a webpage on the Federal Aviation Administration website. (The page was no longer accessible shortly after 9:30 a.m. ET.)

The notices flag abnormalities such as "runways being closed for maintenance, ground stations being out, construction cranes that may be in the proximity of a runway," NBC aviation analyst Capt. John Cox said on MSNBC’s “Morning Joe."

The notices are written in a specific format with "a unique language characterized by the use of specialized contractions" standardized by the International Civil Aviation Organization, according to the FAA. To the untrained eye, the notices look like a random series of letters and numbers.

“It’s a pretty extensive list that the crews get just before departure,” Cox said. “For this NOTAM system to be out — I don’t ever remember it failing before, and I’ve been flying 53 years, so it really is unusual.”

Transportation Secretary Pete Buttigieg vowed that his agency will get to the "root causes" of the massive system meltdown that grounded flights across America.

Wednesday's FAA debacle is the latest airline headache that's impacted mass numbers of U.S. travelers. Buttigieg on Tuesday said his department would be holding Southwest Airlines accountable for its holiday season cancellations.

More than 540 Delta flights were delayed as of 9:13 a.m. ET Wednesday, amounting to 19% of the overall, while 14 had been canceled, according to FlightAware.

The airline said in a tweet it was "safely focused on managing our operation during this morning’s FAA ground stop for all carriers."

After six weeks apart, Wyatt Cosich was about to board the plane taking him home to see his fiancé, Samantha Martinez, when she called and told him about thousands of flights being grounded.

Cosich, 22, told NBC News by telephone Wednesday that he was lining up to get on the plane at San Antonio International Airport, set to take off at 6:15 a.m. local time (7:15 a.m. ET), when Martinez, 23, said they would have to be separated for a little while longer.

“I was really looking forward to seeing her,” Cosich said of Martinez, adding that he had traveled to Texas from their home in Newton, North Carolina, for a job opportunity. He added the pair were getting married Sept. 9.

“There about five gates and around 200 to 300 people here,” he said, adding that airport staff had been providing regular updates to passengers, although most of the time they had been saying, “no more news.”

Unlike some passengers who he said had been getting frustrated with the delays, Cosich said he was calm and while it was unclear when and how he would get home, he praised the airport staff for their professionalism: “They’ve been great.”

American Airlines — which had just over 100 of its flights canceled and nearly 300 delayed as of 8:20 a.m. ET, according to the flight tracker FlightAware — said in a tweet just before 8 a.m. ET that it is "closely monitoring the situation and working with the FAA to minimize customer disruptions."

NBC aviation analyst Capt. John Cox said on MSNBC's "Morning Joe" that domestic flights won't return to their normal schedules until Thursday or Friday given the "domino effect" of the delays and cancellations caused by the nationwide computer outage.

"There's a domino effect to this — airplanes move around the world, and consequently, as an example, an airplane that's trapped in New York, in four hours, is expected to be in Los Angeles, in five hours. And so the people in Los Angeles that are depending on that airplane, their flight will be either delayed or canceled, and the airplane would then say, 'Go to Hawaii, and then come back,'" he said. "So you've got all of these airplanes moving around throughout the day."

"I think it'll certainly be tomorrow at the earliest, and potentially the day after, before the system's back to 100%," Cox added.

The tracking website FlightAware reported that 2,512 flights had been delayed leaving or entering the United States by 7:56 a.m. Wednesday.

Total cancellations stood at 254.

All American Airlines flights from Paris have been delayed until further notice, Groupe ADP, an international airport operator based in the French capital, told NBC News on Wednesday.

Air France, meanwhile, said its flights from the U.S. were going ahead as planned and were not affected by the FAA notice, and it understood that the problems with the system would be fixed soon.

United Airlines said in a statement Wednesday it had temporarily delayed all domestic flights until at least 9 a.m. ET.

"The FAA system that sends out important real-time flight hazards and restrictions to all commercial airline pilots — Notice to Air Missions (NOTAM) — is currently suffering a nationwide outage. United has temporarily delayed all domestic flights and will issue an update when we learn more from the FAA."

In a tweet, Southwest Airlines urged customers to check the status of their flights in the Southwest app or on its website.

About 760 flights within, into and out of the U.S. were delayed as of around 6:30 a.m. Wednesday morning, according to online flight tracker FlightAware. Around 90 flights were listed as canceled.

“Operations across the National Airspace System are affected,” the FAA said in a statement.

“The FAA is working to restore its Notice to Air Missions System. We are performing final validation checks and repopulating the system now,” it said.

Click here to read the full article.