The task of obtaining informative object representations involves training a model, called an encoder, which constructs informative, compressed representations of signals it receives as input. One approach to solving this problem is through the use of self-supervised learning (SSL) methods. An advantage of these methods lies in utilizing only unlabeled data, which is significantly more abundant than labeled data. Among SSL methods, contrastive approaches are particularly prominent; these are based on bringing representations of semantically similar objects (positive pairs) closer together and pushing representations of different signals (negative pairs) apart. Many modern contrastive SSL methods used for obtaining representations of dependent data—where elements within a sample are semantically related—employ a loss function originally designed for independent data. In this work, we propose a theoretically justified approach for selecting a loss function suitable for continuous dependent data, i.e., data in which neighboring elements within the sample can be considered a positive pair. The analysis presented introduces various ways to model similarity between objects and corresponding loss functions, explicitly accounting for correlations between objects. To empirically assess the effectiveness of the proposed loss functions, we focused on temperature and drought forecasting tasks, which can be classified as continuous dependent data. The results demonstrate that our model, combined with the proposed loss functions, outperforms approaches based on the assumption of semantic independence between data, i.e., when all elements of the sample are semantically unrelated. These findings highlight the importance of considering such dependencies for developing high-quality encoders.
Marusov et al. (Fri,) studied this question.