Field knowledge-informed reinforcement learning for synergistic control of active and passive thermal storages in buildings

As critical demand flexibility resources in decarbonized power systems, heating, ventilation, and air-conditioning (HVAC) systems require adaptive mechanisms to navigate various dynamic conditions, like real-time electricity pricing and complex thermal dynamics. Reinforcement learning (RL) applied for HVAC control, while promising, struggles with the control demands of large systems with active and passive thermal storages, i.e., thermal storage facilities (like water tanks) and building thermal masses. Hereby, this paper develops a field knowledge-informed reinforcement learning (FRL) method. First, a historical data-driven model in a physics-restricted state-space form is proposed as a high-fidelity environment simulator to accurately capture thermal dynamics and accelerate RL training. Second, a dynamic potential-based reward shaping technique integrates expert knowledge to significantly enhance convergence stability and speed. Performance evaluations confirmed the FRL method's superior synergistic control across challenging boundary conditions, e.g., various configurations of active–passive storages and volatile electricity tariffs. The FRL method's performance closely approaches the global optimum, consistently converging to within 6% of the theoretical best result and outperforming baseline strategies, including rule-based control, real-time model predictive control, and vanilla RL control. Particularly, in cases with commensurate capacities of active and passive storages (where the capacity ratio E a / E p , ranges from 0.25 to 1.25) and highly volatile pricing signals—conditions representative of evolving decarbonized power systems—FRL achieved at least 26% greater optimization in the combined operation score (considering operational cost and discomfort) compared to benchmarks. This work contributes a novel method to address the complexity of large-scale HVAC systems by integrating field knowledge into RL control, enabling RL for real-world control tasks. • Novel field knowledge-based reinforcement learning control method • Historical data-driven model to capture complex building thermal dynamics • Tailored Inverse Reinforcement Learning and Reward Shaping techniques • Combined artificial & human intelligence outperforming individual methods