Deep transfer learning for image classification: a survey

Abstract Deep neural networks such as convolutional neural networks (CNNs) and transformers have achieved many successes in image classification in recent years. It has been consistently demonstrated that best practice for image classification is when large deep models can be trained on abundant labeled data. However, there are many real world scenarios where the requirement for large amounts of training data to get the best performance cannot be met. In these scenarios, transfer learning can help improve performance. To date, there have been no surveys that comprehensively review deep transfer learning as it relates to image classification overall. We believe it is important for future progress in the field that current knowledge is collated and the overarching patterns analyzed and discussed. In this survey we formally define deep transfer learning and the problem it attempts to solve in relation to image classification. We survey the current state of the field and identify where recent progress has been made. We show where the gaps in current knowledge are and make suggestions for how to progress the field to fill in these knowledge gaps. We present new taxonomies of the solution and applications of transfer learning for image classification. These taxonomies make it easier to see overarching patterns of where transfer learning has been effective and, where it has failed to fulfil its potential. This also allows us to suggest where the problems lie and how it could be used more effectively. We demonstrate that under this new taxonomy, many of the applications where transfer learning has been shown to be ineffective or even hinder performance are to be expected when taking into account the source and target datasets and the techniques used. In many of these cases, the key problem is that methods and hyperparameter settings designed for large and very similar target datasets are used for smaller and much less similar target datasets. We identify alternative choices that could lead to better outcomes.