面向自监督学习的课程式动态数据剪枝方法

Key Points

• The research aims to enhance the efficiency of self-supervised pretraining through dynamic data pruning.
• Designed an exponential loss importance representation.
• Utilized momentum coefficients for exponential moving average of historical losses.
• Implemented a curriculum-based dynamic pruning strategy using one-dimensional clustering.
• Applied a sliding window mechanism to control training progress of different curricula.
• Conducted experiments using several mainstream pretraining frameworks like MAE and MoCo.
• Achieved superior performance with higher data pruning rates compared to existing methods.
• Demonstrated nearly lossless performance in downstream tasks compared to full data training strategies.
• Proved the effectiveness of core components through ablation experiments.

Abstract