Analysis reveals how performance distributions differ across architectures and tasks in deep learning, indicating robustness and tail risk implications.
Key Points
This research aims to examine the impact of non-determinism on deep learning performance distributions across various architectures and tasks.
Conducted 186 experiments on different deep learning architectures for image classification and time series forecasting.
Executed each experiment 100 times with varying random seeds to create performance distributions.
Quantified robustness using metrics for spread, symmetry, and tail risk.
Performance distributions are often non-Gaussian, especially in time series forecasting.
Time series models exhibit significantly higher tail risk, with nearly three times more underperforming outliers compared to image classification models.
Mean performance does not reliably predict robustness, indicating the need for distributional analysis for model selection.