Safe Deep Reinforcement Learning for Electric Power Distribution Grid Operations

Daniel Glover

Incredible progress has been made over the last decade utilizing deep reinforcement learning (DRL) algorithms by the power systems research community to address a variety of grid challenges in modern electric power distribution networks. The aggressive integration of distributed energy resources and deployment of advanced inverter-based technologies at the grid-edge have introduced various complexities and uncertainties into daily operational decision-making and system planning. Particularly at the distribution system level, traditional optimization approaches which have historically relied on deterministic modeling and completely known system state information, are now suffering due to limited observability, stochastic DER penetrations, and modeling inaccuracies, exhibiting the need for adoption of alternative approaches which can learn to optimize grid assets while remaining robust to unforeseen challenges over time. Although the grid has undergone massive communication infrastructure and measurement monitoring upgrades to provide big data for utilities in order to enhance situational awareness, the use of artificial intelligence in utility control centers remains limited. Despite DRL algorithms outperforming many conventional optimization solvers in the research community, utilities remain hesitant to fully embrace DRL as a tool for assisting grid operators due to a lack of safety guarantees, algorithmic transparency, and user technical knowledge. Along with many other learning-based paradigms, DRL is not well interpreted by power system practitioners. Moreover, a majority of research in the power systems community exploring DRL as a candidate solution to overcome operational hurdles has focused mainly on outperforming benchmark optimization methods, generally at the cost of sacrificing safety protocol for performance or not providing any method(s) of safety-related analysis.The goal of this work is to explore various notions of safety in the context of DRL for centralized distribution grid operational tasks, and develop improved techniques for addressing algorithmic, operational, and physical safety concerns in an effort to help realize DRL at the utility scale. First, we discuss the role of DRL in active distribution network grid operations, formalizing the process of translating common optimization models into probabilistic frameworks which can be solved by DRL, and present an open-source environment for DRL algorithm development and benchmarking. Second, we propose a safety-based learning technique for model-free based Volt-VAR control to coordinate multiple solar photovoltaic (PV) systems to achieve local and global objectives. The methodology embeds physical constraint information into the learning process objective to adhere to physical safety and regulatory standards. Third, we discuss the utilization of potential DRL counterpart models using Bayesian modeling with Gaussian Process (GP) surrogates to help circumvent DRL weaknesses by providing inherent uncertainty quantification and acting as non-parametric function approximators. This includes demonstrating GP capabilities for learning a multiphase unbalanced power flow approximation and managing battery energy storage systems with significant computational improvements. Finally, we discuss ongoing and future work to address the simulation-to-reality gap, proposing strategies to detect and overcome distributional shift in distribution grid operations with DRL, providing robust candidate hybrid DRL-GP modeling solutions to improve policy transfer from simulations to the real world power systems.

Safe Deep Reinforcement Learning for Electric Power Distribution Grid Operations

Files and links (1)

Abstract

Metrics

Details