• Propose an RL-based energy management scheme to minimize distribution network operating costs with limited prior knowledge. • Propose a two-step knowledge guidance strategy based on 3D interval classification to improve the RL action quality. • Propose a knowledge-guided SAC algorithm to solve the energy management problem with improved efficiency and optimality. This paper proposes a limited prior knowledge-aided reinforcement learning (LKRL) method for optimal energy management in active distribution networks. Different from existing RL-based energy management schemes that fully rely on automated learning models, limited prior knowledge will be utilized to rectify bad actions and improve the learning performance. Firstly, an optimal energy management model is formulated for three-phase unbalanced distribution networks to describe the mathematical problem and provide the basis for constructing the Markov Decision Process for reinforcement learning. The objective is to minimize the total operation cost, subject to power flow balance, voltage/transmission capacity limits, as well as operating constraints of distributed generators (DGs) and energy storage systems (ESSs). Secondly, a two-step knowledge guidance strategy is developed based on historical superior samples and a 3D interval classification method. The strategy can generate coarse yet effective action intervals under different system states, which are implemented in both action exploration and exploitation processes for improved learning performance. Thirdly, a knowledge-guided soft actor-critic (KGSAC) algorithm is proposed to solve the energy management problem. Compared to standard RL algorithms, knowledge guidance terms are designed and incorporated into the loss function, maintaining consistent updates of actor and critic networks. An adaptive adjustment mechanism of knowledge weighting factors is also proposed, enabling efficient learning towards satisfactory rewards with limited training steps. Case studies are conducted on modified IEEE 123-bus unbalanced distribution networks and comparative results showcase the effectiveness of the proposed method in achieving near-optimal and efficient energy management.
Ren et al. (Wed,) studied this question.