Deep self-supervised learning algorithm applied to tone-mapped image quality assessment

Abstract We propose modifying the Barlow twins (BT) algorithm, to train convolutional neural networks (CNNs) which extract features that are specifically tailored for tone-mapped image quality assessment (TMIQA). In BT, a CNN is trained with pairs of images that are similar in terms of visual content. In our modified approach, pairs of images sampled from the dataset are similar in terms of image quality rather than visual content. This modification makes the feature vectors more suitable for TMIQA. Using quality features extracted from such CNNs, we train support vector regression (SVR) models that map such features into a score that summarizes the overall quality impression of an image. We denote the composition of feature extractor with SVR model as a metric. We use three datasets in our experiments: patch-based tone-mapping database (PBTDB), embedded signal processing laboratory (ESPL), and tone-mapped image database (TMID). We compare our metrics with three TMIQA metrics that are based on hand-crafted features: blind tone-mapped quality index (BTMQI), high dynamic range image gradient based evaluator-1 (HIGRADE-1), and HIGRADE-2. We perform nine experiments: three intra-dataset experiments, that involve training and testing on the same dataset; and six cross-dataset experiments, that involve training on one dataset and testing on a different dataset. The proposed metrics are advantageous in six experiments. In terms of the Pearson correlation coefficient between predicted and ground-truth mean opinion score values, our best results in the intra-dataset experiments show an improvement over the state-of-the-art by 1.2%. In the cross-dataset experiments, we observe improvements up to 55.2%.