Bayesian combination of correlated subjective probability estimates

Abstract The combination of human forecasters’ subjective probability estimates usually improves upon the estimates provided by individual forecasters. In order to combine the probability estimates in a Bayes optimal way, prior work proposed a normative Bayesian fusion model that models the estimates with a beta distribution conditioned on their truth value. However, this model assumes conditionally independent probability estimates, although estimates provided by different forecasters are usually correlated. Here, we introduce a Bayesian model for combining subjective probability estimates that explicitly considers their correlation. We assume that an estimate provided by a forecaster for a given query depends on both the forecaster’s skill and the query’s difficulty. The correlation between probability estimates provided by different forecasters is assumed to be caused by the queries that make the forecasters provide similar estimates, for example, correct and highly confident estimates for very easy queries. Our model represents the probability estimates with a beta distribution conditioned on their truth value. It explicitly models the forecasters’ skills and the queries’ difficulties with skill parameters specific for each forecaster and difficulty parameters specific for each query. In this way, it can model the correlations between probability estimates and consider it when combining the estimates. Evaluations on a data set consisting of the subjective probability estimates of 85 human forecasters for 180 queries show improved fusion performance in terms of Brier score compared to related Bayesian fusion models. In particular, it outperforms independent fusion models that suffer from overconfidence.