Estimating diagnostic accuracy under uncertainty about disease status: a sepsis case study

OBJECTIVES: Expert panels in diagnostic accuracy research typically classify the target condition dichotomously (ie, present or absent) for each study participant. This may however lead to biased accuracy estimates when experts are uncertain about this classification. Eliciting probabilistic estimates may provide a solution. METHODS: We compared three approaches for estimating index test diagnostic accuracy using probabilistic estimates on target condition presence from an expert panel: (i) dichotomous approach: forcing dichotomous target condition classification based on expert panel probability; (ii) direct weighting approach: weighting index test results directly with the expert panel probability; (iii) Bayesian approach: formal likelihood model of observing index test results given expert panel probability. The SPACE study (SePsis in ACutely ill patients in the Emergency room), investigating diagnostic performance of various sepsis prediction models (the systemic inflammatory response syndrome SIRS, quick sequential organ failure assessment qSOFA, and modified early warning score MEWS) and clinical bedside judgment (CBJ) was used as a case study. RESULTS: The analysis included 390 study participants, of which 79 (20.3%) had sepsis according to dichotomous classification by the expert panel. However, the expert panel, even after reviewing all information including follow-up, expressed considerable uncertainty about the final sepsis diagnosis in 65% of all patients, as the mean expert panel probability was between 0.2 and 0.8 (so not close to 0 or 1). The dichotomous approach yielded different diagnostic accuracy estimates compared to the Bayesian approach. For example, estimated sensitivity and specificity of SIRS were 95% (95% confidence interval CI, 88%-98%) and 46% (95% CI, 41%-52%) using the dichotomous approach, and 99% (95% CI, 96%-100%) and 60% (95% CI, 53%-68%) using the Bayesian approach. Diagnostic accuracy estimates differed between approaches and varied in direction and size across prediction models. CONCLUSION: Probabilistic estimates of target condition presence elicited from expert panels provide valuable insight into remaining uncertainty that is ignored in dichotomous target condition classification. The Bayesian approach yields valid estimates of diagnostic accuracy incorporating any uncertainty about target condition status expressed by the expert panel, assuming that expert probabilities accurately reflect the true probability given the pattern of observed test results.