We study collapse in softmax-based exponential normalization and show that it can be described at leading order by a single scalar quantity Sₜ = βₜ Δₜ, where Δₜ is the gap between the two largest scores. We find that the bulk probability mass follows the scaling relation log Bₜ ≈ −Sₜ in the regime Sₜ ≫ log K, and that the collapse threshold scales as S* (K) ~ log K. These results are derived from a decomposition of the log-partition function and supported by numerical experiments across score distributions and perturbations within the softmax framework. The findings provide a minimal and predictive description of collapse as a structural consequence of exponential normalization. Keywords: softmax collapse, scaling laws, exponential normalization, beta delta invariant, bulk probability mass, partition function, log B approx minus S, log B ≈ −S Code & ReproducibilityAll numerical experiments and figures in this paper can be reproduced using the code available at: shipworm-o/softmax-collapse-figs: Appendix B figures code for softmax collapse phase transition
Kim et al. (Wed,) studied this question.