Machine learning models are increasingly deployed in critical energy infrastructure, where domain experts require transparent explanations for decision-making. SHAP (SHapley Additive exPlanations) has become a popular method for energy systems applications. However, energy data exhibit inherent correlations due to physical constraints, operational relationships, and market dynamics, posing challenges for interpreting SHAP-based explanations. This work investigates how feature correlations influence SHAP-based explanations using controlled synthetic experiments and real-world power grid data. Our analysis shows that only correlation-aware methods can attribute importance to economically linked features, such as solar generation in predicting fossil fuels, which may reflect genuine systemic interdependencies that are valuable for prediction and scientific understanding. Our findings highlight the tradeoff between true to the model explanations that reflect model behavior and true to the data approaches that consider real-world dependencies. In complex energy systems with circular dependencies, temporal dynamics, and hidden constraints, explanation validity cannot be universally defined. We emphasize the need for practitioners’ awareness of the trade-offs between model analysis, scientific discovery, and operational understanding.
Nikoltchovska et al. (Mon,) studied this question.