Estimating (Non)Linear Selection on Reaction Norms: A General Framework for Labile Traits

Individual reaction norms describe how labile phenotypes vary as a function of organisms' expected trait values (intercepts) and plasticity across environments (slopes), as well as their degree of stochastic phenotypic variability or predictability (residuals). These reaction norms can be estimated empirically using multilevel, mixed‐effects models and play a key role in ecological research on a variety of behavioral, physiological, and morphological traits. Many evolutionary models have also emphasized the importance of understanding reaction norms as a target of selection in heterogeneous and dynamic environments. However, it remains difficult to empirically estimate nonlinear selection on reaction norms, inhibiting robust tests of adaptive theory and accurate predictions of phenotypic evolution. To address this challenge, we propose generalized multilevel models for estimating stabilizing, disruptive, and correlational selection on the reaction norms of labile traits, which can be applied to any repeatedly measured phenotype using a flexible Bayesian framework. Our modeling approach avoids inferential bias by simultaneously accounting for uncertainty in reaction norm parameters and their potentially nonlinear fitness effects. We formally introduce these nonlinear selection models and provide detailed discussion on their interpretation and potential extensions. We then validate their application in a Bayesian framework using simulations. We find that our models facilitate unbiased Bayesian inference across a broad range of effect sizes and desirable power for hypothesis tests with large sample sizes. Coding tutorials are further provided to aid empiricists in applying these models to any phenotype of interest using the Stan probabilistic programming language in R. The proposed modeling framework should, therefore, readily enhance tests of adaptive theory for a variety of labile traits in the wild.