Given the same set of training, qualifications, and information, experts in a reliable and trustworthy system should theoretically make similar decisions. However, undesired decision variability (“noise”) has been observed in many high-stakes domains, which is both concerning and unsurprising given the complexity of decisions and known influence of several factors. Moreover, not all variability is undesirable, and the study of intuitive decision-making in naturalistic settings demonstrates the skill of the human expert when evaluating complex cases. The current paper describes a “noise audit” where we evaluated individual risk-informed decision making for regulatory action at U.S. commercial nuclear power plants. In a scenario study, individuals exhibited high consistency in inspection type decisions, although they varied substantially when assigning a significance code, resulting in many decisions that were different than what would be expected by the agency. Together, the results indicate the presence of decision “noise” in one of the two nuclear regulatory processes examined, though some of that variability is desirable given plant and case-specific differences.
Barhorst-Cates et al. (Thu,) studied this question.