Graph-based Safe Reinforcement Learning for Dynamic Optimal Power Flow with Hybrid Action Space Considering Time-varying Network Topologies

The proliferation of distributed energy resources and time-varying network topologies in active distribution networks presents unprecedented challenges for network operators. While reinforcement learning (RL) has shown promise in addressing network-constrained energy scheduling, it faces difficulties in managing the complexities of dynamic topologies and discrete-continuous hybrid action spaces. To address these challenges, a graph-based safe RL approach is proposed to learn dynamic optimal power flow under time-varying network topologies. This proposed approach leverages graph convolution operators to handle network topology changes, while safe RL with parameterized action ensures policy development. Specifically, the graph convolution operator abstracts key characteristics of the network topology, enabling effective power flow management in non-stationary environments. Besides that, a parameterized action constrained Markov decision process is employed to handle the hybrid action space and ensure compliance with physical network constraints, thereby accelerating the deployment of safe policy for hybrid action spaces. Numerical results demonstrate that the proposed approach efficiently navigates the discrete-continuous decision space while accounting for the constraints imposed by the dynamic nature of power flow in time-varying network topologies.