If you fly, you may have been disappointed on 31 December 2007 when NASA released 16,000 pages’ worth of survey data on flight safety. Despite spending $11 million on gathering the data, which involved questioning 500 US commercial pilots every month for four years, NASA administrator Michael Griffin says the results are unusable because the survey’s methodology was flawed.
Conducted by the Battelle Memorial Institute in Santa Clara, California, the interviewed pilots about safety issues such as the frequency of near misses, diversions to other airports due to technical failures, and breaks in communication with air traffic control. While existing accident reporting systems allow ground staff to report safety breaches and analyse data from flight recorders, NAOMS was supposed to flag up problems that only pilots see and that increase the probability of accidents.
That didn’t happen. Battelle’s findings ended up contradicting accepted statistics, such as those for engine failure: the survey’s rate for this was four times that recorded by the Federal Aviation Administration (91av, 10 November 2007, p 5). NASA attributes this problem to Battelle counting some incidents more than once and has deemed the survey , although it has been obliged to release the raw data under freedom- of-information legislation.
Advertisement
Despite these problems, the raw data may still have some value, according to Jon Krosnick, an expert in survey methodology at Stanford University in California who acted as a consultant on the survey’s design. Krosnick accepts that Battelle’s analysis is incorrect, but argues that if properly analysed, the data could still provide unique and important information about air safety.
One way to correct the analysis, he says, is to calculate the probability that an event such as a near miss has been reported by more than one person, and take that into account when reporting the total number of such events. “Every event with two pilots witnessing it has twice the probability of being reported. If two aircraft almost hit each other, and they have three pilots in each plane, that’s six times the probability.” Although correcting the analysis in this way involves a margin for error, Krosnick says the result would still be a useful measure of aviation safety.
In addition, Battelle may not have taken account of the fact that some pilots did not fly every week, says Krosnick, causing the incident rate to be underestimated. The analysis may also have wrongly assumed that incidents occurred in the same calendar year as the interviews in which they were reported. Both these errors could be corrected by NASA, says Krosnick, or an external group if NASA releases key background data.
Revising the results in these ways could both help nip problems in the bud and boost passengers’ trust. “We think the data should be re-analysed to instill confidence in the flying public,” says Peter Janhunen of the Airline Pilots Association in Washington DC.
Aviation – Learn more in our comprehensive .
