Predicting text recognition word error rate of image documents without ground truth transcripts

Abstract Unsupervised error estimation is examined under the framework of the statistical decision theory. The error probability of a classifier trained with a training set of a finite, fixed size is considered, along with an unsupervised naive estimate of this error. In general, the naive estimates tend to be significantly smaller than the empirical errors measured using ground truth class labels, but the estimated values can be easily calibrated with a function whose parameters can be trained using moderate amounts of class-labelled data. This way, true error rates can be accurately predicted for new unlabelled data. These ideas are applied to the essential classification problem which underlies the automatic transcription of text images. As a result, various methods are developed to predict the Word Error Rate (WER) of a text image recognizer, on unseen sets of images for which no ground truth transcripts are available. Experiments on three large handwritten text datasets show that the error rates predicted by some of these methods are sufficiently accurate for practical applications. More specifically, absolute deviations of the predicted word error percentages from the corresponding real WER are lower than 1.7% for the three datasets, and the relative values of these deviations with respect to the WER of each dataset, are 5.6%, 4.2% and 6.1%.