NASA shut down a massive air-safety survey project without ever properly evaluating, explaining or publicizing its purpose and results, and thus lost a chance for valuable insight into safety issues, the space agency's inspector general said Monday.
The watchdog office said NASA should interpret and analyze the results of its interviews with some 30,000 pilots, but NASA in a written response continued to reject that idea.
NASA will evaluate the methodology that its staff used in the $11.3 million project, but going further to actually report on the findings isn't worthwhile because the interviews, which were stopped at the end of 2004, are less relevant with the passage of time, wrote the agency's associate administrator, Jaiwon Shin.
The intent of the National Aviation Operations Monitoring Service was to help prevent aviation accidents by viewing air safety through the eyes of pilots, flight attendants, mechanics and air traffic controllers. Their insights would add to the picture provided by other aviation monitoring programs, the audit said.
The interviews ran from 2001 to 2004. The program came to light when NASA last fall rejected an Associated Press request for the results, stating that disclosing the information could have a negative effect on airline profits and public confidence in the air lines. Congress launched an investigation, and NASA's own inspector general began its audit.
Asked for comment, NASA spokesman David Mould said Monday night that he was unaware that the audit had been released and that he would look at it.
The audit criticized the way the project was run from the beginning. It also said the partial results that NASA Administrator Michael Griffin posted on the Internet under pressure from Congress did not help the public understand the project or its value.
Griffin has said he saw little value in the program, but the audit disagreed.
"The government may have missed an opportunity to foster a deeper understanding of the aviation safety environment from 2001 to 2004 because its working groups were unable to reach a consensus on the validity or value of the NAOMS data," the audit found. "As a result NASA was reluctant to publish a report detailing research and conclusions garnered from the collected NAOMs survey data."
The Federal Aviation Administration had objected to the survey's results that showed a greater rate of safety incidents than were found in other government monitoring systems, and it questioned the methodology, the audit said. FAA and other government groups joined working sessions with NASA but did not try to interpret or validate the results.
The audit criticized NASA staff for failing to provide sufficient direction for the contract company that did the survey, Battelle Memorial Institute, and said Battelle underestimated the amount of work required for the project.
As a result, the survey never interviewed air traffic controllers or flight attendants as intended, and fell short of its intended validation and release of the data. Instead NASA turned the program over to the Air Line Pilots Association to conduct its own Web-based version, which by definition would be different than the telephone survey that Battelle had conducted.
The survey was part of a larger NASA effort to contribute to proactive air safety management, and that larger program also has ended.
The NAOMS trends and findings were not meant to stand alone, the audit said, but "the true value of its data cannot be accurately assessed without comparison and reconciliation with other data sets" for an overall picture.