Challenges of Measuring Performance for FAA's Safety Oversight System

FAA regulates the safety of the aviation industry through the safety oversight system, which is a system of rulemaking, standard-setting, certification, accident investigation, rule enforcement, and surveillance activities. Federal programs, including those of FAA, use performance indicators to measure the achievement of program goals. As part of a broader program of developing risk management methodologies, FAA is researching performance indicators that can be used to measure the performance of the safety oversight system. One of its goals is to create performance indicators that can describe the safety oversight system's influence on safety outcomes such as fatalities. Creating performance measures that link activities to safety outcomes is challenging because it is difficult to establish the causation between oversight activities and these safety outcomes. This challenge is not unique to FAA, and external reviewers such as the Government Accountability Office have recommended that other high-reliability sectors, including rail, develop such indicators. In addition to safety outcomes, other aspects of safety oversight system performance can be described with meaningful metrics. The background and motivation for oversight evaluation in the aviation industry and in general are discussed, as well as the challenges, some generic and some unique, of evaluating aviation safety oversight activities. Research is also presented on how safety oversight evaluation is conducted outside aviation.