A variance decomposition approach to inconclusives in forensic black box studies

Abstract In the USA, ‘black box’ studies are increasingly being used to estimate the error rate of forensic disciplines. A sample of forensic examiner participants is asked to evaluate a set of items whose source is known to the researchers but not to the participants. Participants are asked to make a source determination (typically an identification, exclusion, or some kind of inconclusive). We study inconclusives in two black box studies, one on fingerprints and one on bullets. Rather than treating all inconclusive responses as functionally correct (as is the practice in reported error rates in the two studies we address), irrelevant to reported error rates (as some would do), or treating them all as potential errors (as others would do), we propose that the overall pattern of inconclusives in a particular black box study can shed light on the proportion of inconclusives that are due to examiner variability. Raw item and examiner variances are computed and compared with the results of a logistic regression model that takes account of which items were addressed by which examiner. The error rates reported in black box studies are substantially smaller than ‘failure rate’ analyses that take inconclusives into account. The magnitude of this difference is highly dependent on the particular study at hand.