XRL-LLM: Explainable Reinforcement Learning Framework for Voltage Control

Reinforcement learning (RL) agents are increasingly deployed for voltage control in power distribution networks. However, their opaque decision-making creates a significant trust barrier, limiting their adoption in safety-sensitive operational settings. This paper presents XRL-LLM, a novel framework that generates natural language explanations for RL control decisions by combining game-theoretic feature attribution (KernelSHAP) with large language model (LLM) reasoning grounded in power systems domain knowledge. We deployed a Proximal Policy Optimization (PPO) agent on an IEEE 33-bus network to coordinate capacitor banks and on-load tap changers, successfully reducing voltage violations by 90.5% across diverse loading conditions. To make these decisions interpretable, KernelSHAP identifies the most influential state features. These features are then processed by a domain-context-engineered LLM prompt that explicitly encodes network topology, device specifications, and ANSI C84.1 voltage limits.Evaluated via G-Eval across 30 scenarios, XRL-LLM achieves an explanation quality score of 4.13/5. This represents a 33.7% improvement over template-based generation and a 67.9% improvement over raw SHAP outputs, delivering statistically significant gains in accuracy, actionability, and completeness (p<0.001, Cohen’s d values up to 4.07). Additionally, a physics-grounded counterfactual verification procedure, which perturbs the underlying power flow model, confirms a causal faithfulness of 0.81 under critical loading. Finally, five ablation studies yield three broader insights. First, structured domain context engineering produces synergistic quality gains that exceed any single knowledge component, demonstrating that prompt composition matters more than the choice of foundational model. Second, even an open source 8B-parameter model outperforms templates given the same prompt, confirming the framework’s backbone-agnostic value. Most importantly, counterfactual faithfulness increases alongside load severity, indicating that post hoc attributions are most reliable in the high-stakes regimes where trustworthy explanations matter most.