Multi-Agent Deep Reinforcement Learning for Voltage Control With Coordinated Active and Reactive Power Optimization

The increasing penetration of distributed renewable energy resources causes voltage fluctuations in distribution networks. The controllable active and reactive power resources such as energy storage (ES) systems and electric vehicles (EVs) in active distribution networks play an important role in mitigating the voltage excursions. This paper proposes a two-timescale hybrid voltage control strategy based on a mixed-integer optimization method and multi-agent reinforcement learning (MARL) to reduce power loss and mitigate voltage violations. In the slow-timescale, the active and reactive power optimization problem involving capacitor banks (CBs), on-load tap changers (OLTC), and ES systems is formulated as a mixed-integer second-order cone programming problem. In the fast-timescale, the reactive power of smart inverters connected to solar photovoltaic systems and active power of EVs are adjusted to mitigate short-term voltage fluctuations with a MARL algorithm. Specifically, we propose an experience augmented multi-agent actor-critic (EA-MAAC) algorithm with an attention mechanism to learn high-quality control policies. The control policies are executed online in a decentralized manner. The proposed hybrid voltage control strategy is validated on an IEEE testing distribution feeder. The numerical results show that our proposed control strategy is not only sample-efficient and robust but also effective in mitigating voltage fluctuations.

• Publication year

  2022

• Venue

  IEEE Transactions on Smart Grid

• Publication date

  2022-11-01

• Fields of study

  Computer Science, Engineering, Environmental Science

• Identifiers
  DOI 10.1109/TSG.2022.3185975
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Model-augmented safe reinforcement learning for Volt-VAR control in power distribution networks

  2022cited by this paper
• A Customized Voltage Control Strategy for Electric Vehicles in Distribution Networks With Reinforcement Learning Method

  2021cited by this paper
• Deep Reinforcement Learning in Power Distribution Systems: Overview, Challenges, and Opportunities

  2021influential reference
• Reinforcement Learning-based Smart Inverter Control with Polar Action Space in Power Distribution Systems

  2021cited by this paper
• A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning

  2020cited by this paper
• Online Multi-Agent Reinforcement Learning for Decentralized Inverter-Based Volt-VAR Control

  2020cited by this paper
• Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks

  2020cited by this paper
• Model-Free Voltage Regulation of Unbalanced Distribution Network Based on Surrogate Model and Deep Reinforcement Learning

  2020cited by this paper
• Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient

  2020cited by this paper
• Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems

  2020cited by this paper
• Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems

  2020cited by this paper
• Reinforcement Learning with Augmented Data

  2020cited by this paper
• Volt-VAR Control in Power Distribution Systems with Deep Reinforcement Learning

  2019cited by this paper
• Three-Stage Robust Inverter-Based Voltage/Var Control for Distribution Networks With High-Level PV

  2019cited by this paper
• Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning

  2019cited by this paper
• Dynamic Distribution Network Reconfiguration Using Reinforcement Learning

  2019cited by this paper
• Addressing Function Approximation Error in Actor-Critic Methods

  2018cited by this paper
• A centralized-based method to determine the local voltage control strategies of distributed generator operation in active distribution networks

  2018influential reference
• Network Partition and Voltage Coordination Control for Distribution Networks With High Penetration of Distributed PV Units

  2018influential reference
• Optimal Scheduling for Electric Vehicle Charging With Discrete Charging Levels in Distribution Grid

  2018cited by this paper
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

  2018cited by this paper
• Voltage and Reactive Power Control to Maximize the Energy Savings in Power Distribution System With Wind Energy

  2018cited by this paper
• Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning

  2018cited by this paper
• Volt-VAR Control Through Joint Optimization of Capacitor Bank Switching, Renewable Energy, and Home Appliances

  2018cited by this paper
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning

  2018cited by this paper
• Emergence of Grounded Compositional Language in Multi-Agent Populations

  2017cited by this paper
• Cooperative Multi-agent Control Using Deep Reinforcement Learning

  2017cited by this paper
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

  2017influential reference
• Continuous control with deep reinforcement learning

  2015influential reference
• A Model Predictive Control Approach for Low-Complexity Electric Vehicle Charging Scheduling: Optimality and Scalability

  2015cited by this paper
• Optimal Operation of Distribution Networks Considering Energy Storage Devices

  2015cited by this paper
• A Framework for Volt-VAR Optimization in Distribution Systems

  2015cited by this paper
• MPC-Based Voltage/Var Optimization for Distribution Circuits With Distributed Generators and Exponential Load Models

  2014cited by this paper
• Exact Convex Relaxation of Optimal Power Flow in Radial Networks

  2013cited by this paper
• Distributed Volt/VAr Control by PV Inverters

  2013cited by this paper
• Coordinated Active Power Curtailment of Grid Connected PV Inverters for Overvoltage Prevention

  2011cited by this paper
• Radial distribution load flow using conic programming

  2006cited by this paper
• In that case.

  2005cited by this paper
• YALMIP : a toolbox for modeling and optimization in MATLAB

  2004cited by this paper
• Probabilistic pursuit-evasion games: theory, implementation, and experimental evaluation

  2002cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper
• Radial distribution test feeders

  1991cited by this paper
• Optimal sizing of capacitors placed on a radial distribution system

  1989cited by this paper
• Network reconfiguration in distribution systems for loss reduction and load balancing

  1989cited by this paper

• Human-in-the-Loop Reinforcement Learning Method for Volt/Var Control in Active Distribution Network With Safe Operation Mechanism

  2026cites this paper
• Quantum-Enhanced PPO for Demand Follower Model in Stochastic Industrial Smart Grid Environment

  2026cites this paper
• Prototype Federated Reinforcement Learning for Voltage Regulation in Distribution Systems With Physics-Aware Spatial-Temporal Graph Perception

  2026cites this paper
• Uncertainty-Aware Model-Based Multi-Agent Deep Reinforcement Learning for Robust Active Voltage Control

  2026cites this paper
• A comprehensive review of reinforcement learning-based voltage control in smart grids

  2026cites this paper
• Assessing the maximum carbon reduction potential of regional power grids from grid operators' perspective

  2026cites this paper
• Distribution system Volt-VAR control via transformer-enhanced reinforcement learning with diffusion-driven data augmentation

  2026cites this paper
• Bi-Level Collaborative Voltage Regulation for Distribution Networks with High-Penetration Renewables and Multi-Microgrids Considering Operational Economy

  2026cites this paper
• Adaptive Multi‐Objective Coordinated Scheduling of Distribution Areas Based on Deep Reinforcement Learning

  2026cites this paper
• Collaborative Stability and Benefit Optimization of V2G Based on Multiagent Graph Reinforcement Learning With Privacy Protection

  2026cites this paper
• Reinforcement-Learning Control for Coordinated Voltage Regulation in Distribution Systems Using a Power Hardware-in-the-Loop Platform

  2026cites this paper
• Planning and Integration of a Multi-Robot Adaptive Charging Network for Voltage Regulation via Unscented Kalman Filter-Based Deep Reinforcement Learning

  2026cites this paper
• Reinforcement Learning-Based Adaptive Wire Gauge Selection for Zonal Automotive Harness Design Under Dynamic Driving Scenarios

  2026cites this paper
• Multi-objective partitioning and cooperative peak shaving for distributed photovoltaic clusters via multi-agent reinforcement learning

  2026cites this paper
• Optimized Adaptive Voltage Control for Distribution Network Using Utility-Scale BESS and PV Systems

  2026cites this paper
• Energy storage-based voltage optimization and coordinated control for large-scale DPV AC/DC hybrid distribution networks

  2026cites this paper
• Two-Timescale Coordination of Discretely and Continuously Adjustable Devices in ADNs With DRL and Physical Convex Optimization

  2025influential citation
• Optimality Analysis of Reinforcement Learning-based Reactive Power Control

  2025cites this paper
• Machine learning-driven intelligent voltage control in renewable energy grids

  2025cites this paper
• Decentralized Voltage and Var Control of Active Distribution Network Based on Parameter-Sharing Deep Reinforcement Learning

  2025cites this paper
• Low-Carbon Energy Management for Networked Multi-Energy Microgrids Using Multi-Agent Soft Actor-Critic Algorithm

  2025cites this paper
• Deep reinforcement learning-based controller for DC-link voltage regulation and voltage sag compensation in a solar PV-integrated UPQC system

  2025cites this paper
• SCD-RL: Safe Control Decision Reinforcement Learning for Uncertain Line Faults in Smart Grids

  2025cites this paper
• Coordinated electric vehicles dispatch for multi-service provisions: A comprehensive review of modelling and coordination approaches

  2025cites this paper
• Multiagent reinforcement learning in enhancing resilience of microgrids under extreme weather events

  2025cites this paper
• Optimization of Cooperative Operation of Multiple Microgrids Considering Green Certificates and Carbon Trading

  2025cites this paper
• Explainable Artificial Intelligence for Reactive Power Optimization of Distribution Networks

  2025cites this paper
• Optimizing Advertising Billboard Coverage in Urban Networks: A Population-Weighted Greedy Algorithm with Spatial Efficiency Enhancements

  2025cites this paper
• Hierarchical Reinforcement Learning for Volt/Var and Wireless Communication Co-Scheduling in Active Distribution Network

  2025cites this paper
• Deep Reinforcement Learning Applied to Electric Distribution Network Applications Using Real Time Simulation

  2025cites this paper
• Fully Distributed Dynamic Event-Triggered Consensus of Heterogeneous Multiagent Systems With Applications to Voltage Control

  2025cites this paper
• An attention-based joint value estimation strategy for multi-agent coordination optimization

  2025cites this paper
• Smart Scheduling of Microgrids: An Integrated Approach for Power Management, Voltage Control, and Distributed Solutions

  2025cites this paper
• Deep Reinforcement Learning for Voltage Stability: DDQN-Based Control with Real-Time Hardware Validation

  2025cites this paper
• Machine Learning for Photovoltaic Power Forecasting Integrated with Energy Storage Systems: A Scientometric Analysis, Systematic Review, and Meta-Analysis

  2025cites this paper
• Data-Driven Voltage Regulation of Distribution Grid Using Nonlinear Autoregressive Model with Exogenous Inputs (NARX)

  2025cites this paper
• Bi-Level Reinforcement Learning Based Volt/Var Control with Hybrid Action Space for Distribution Networks

  2025cites this paper
• Optimal reactive power and voltage control in regional power grids using temporal segmentation of wind and PV generation

  2025cites this paper
• Enhanced Multi-Agent Reinforcement Learning for Power Quality Enhancement and False Data Injection Defense in Multi-Microgrid Systems

  2025cites this paper
• Reactive Power Optimization Strategy for Power Grid with High Proportion of Renewable Energy Based on Reinforcement Learning Algorithm

  2025cites this paper
• Coordinated voltage control for distribution network and multi-microgrids based on improved EM-RACE multi-agent reinforcement learning

  2025cites this paper
• Coordinated Optimization of Distributed Energy Resources Based on Spatio-Temporal Transformer and Multi-Agent Reinforcement Learning

  2025cites this paper
• Human-Machine Bidding Strategy for Distributed Energy Resources Based on Multiagent Inverse Reinforcement Learning

  2025cites this paper
• Physics-Informed Reinforcement Learning for Large-Scale EV Smart Charging Considering Distribution Network Voltage Constraints

  2025cites this paper
• An Optimization Strategy for Power Grid Maintenance Scheduling Based on Multi-Agent Deep Reinforcement Learning

  2025cites this paper
• Reactive power optimization via deep transfer reinforcement learning for efficient adaptation to multiple scenarios

  2025cites this paper
• Two-Stage TSO-DSO Services Provision Framework for Electric Vehicle Coordination

  2025cites this paper
• Eco-friendly strategic decision-making towards grid modernization with fully renewable water and power community hubs

  2025cites this paper
• Hydrogen Energy Storage System Participated Decentralized Voltage Control With Multi-Agent Deep Reinforcement Learning Method

  2025cites this paper
• Coordinated Reactive Power Optimization for Transmission and Distribution System With Gross Prediction Errors: A Modified Belief Markov Decision Process-Based Reinforcement Learning Methodology

  2025cites this paper
• Dendritic Net Driven Quadratic Day-Ahead Voltage Control for Power System With Distributed Generation

  2025cites this paper
• Local Distribution Voltage Control Using Large-Scale Coordinated PV Inverters: A Novel Multi-Agent Deep Reinforcement Learning-Based Approach

  2025cites this paper
• FGSM-CW Hardened Static Android-Malware Detection with Incremental PCA on the CCCS–CIC AndMal-2020 Corpus

  2025cites this paper
• Adaptive Deep Reinforcement Learning for VVC in High DER-Integrated Distribution Grids

  2025cites this paper
• An Adaptive MARL Large Model for Dispatch Strategy Generation in Logistics-Energy Spatiotemporal Coordination of Container Seaports

  2025cites this paper
• Two timescales coordination of reconfiguration switches and inverters using deep reinforcement learning

  2025cites this paper
• Optimal Power Flow for High Spatial and Temporal Resolution Power Systems with High Renewable Energy Penetration Using Multi-Agent Deep Reinforcement Learning

  2025cites this paper
• A Review of Smart Grid Evolution and Reinforcement Learning: Applications, Challenges and Future Directions

  2025cites this paper
• Spatio-Temporal Adaptive Voltage Coordination Control Strategy for Distribution Networks with High Photovoltaic Penetration

  2025cites this paper
• GARL: A Generative Adversarial Reinforcement Learning for Transferable Power Grid Dispatch

  2025cites this paper
• Autonomous Voltage Regulation for Smart Distribution Network With High-Proportion PVs:A Graph Meta-Reinforcement Learning Approach

  2025cites this paper
• A Survey of Machine Learning Techniques for Optimal Capacitor Placement and Sizing in Smart Distribution Networks

  2025cites this paper
• Fault Detection and Isolation for Multiagent Systems Under Event-Triggered Communication: A Zonotopic Joint Estimation Method

  2025cites this paper
• A scalable demand-side energy management control strategy for large residential districts based on an attention-driven multi-agent DRL approach

  2025cites this paper
• A Fault-Tolerant Distributed Agent-Based Framework for Active/Reactive Power Optimization Using Sensitivity Gradient Methods in Smart Grids

  2025cites this paper
• Coordination optimization of multi-micro-network and distribution network based on multi-agent reinforcement learning

  2024cites this paper
• Graph Learning-Based Voltage Regulation in Distribution Networks With Multi-Microgrids

  2024influential citation
• Multi-agent graph reinforcement learning for decentralized Volt-VAR control in power distribution systems

  2024cites this paper
• Event-Triggered Voltage Regulation for High-PV-Penetration Networks With Time Delays

  2024cites this paper
• Physics-Guided Multi-Agent Deep Reinforcement Learning for Robust Active Voltage Control in Electrical Distribution Systems

  2024cites this paper
• A temporal and spatial electric vehicle charging optimization scheme with DSO-EVA coordination framework

  2024cites this paper
• Application and progress of artificial intelligence technology in the field of distribution network voltage Control：A review

  2024cites this paper
• Simultaneous Voltage Regulation and Unbalance Compensation in Distribution Systems With an Information-Driven Learning Approach

  2024cites this paper
• Decentralized control in active distribution grids via supervised and reinforcement learning

  2024cites this paper
• Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

  2024cites this paper
• Softly Collaborated Voltage Control in PV Rich Distribution Systems With Heterogeneous Devices

  2024cites this paper
• Artificial intelligence-based methods for renewable power system operation

  2024influential citation
• Multi-Agent Reinforcement Learning for Energy Networks: Computational Challenges, Progress and Open Problems

  2024cites this paper
• Multiagent Soft Actor–Critic Learning for Distributed ESS Enabled Robust Voltage Regulation of Active Distribution Grids

  2024cites this paper
• Volt-PF Control Mode for Distribution Feeder Voltage Management Under High Penetration of Distributed Energy Resources

  2024cites this paper
• Optimizing Microgrid Energy Management: Intelligent Techniques

  2024cites this paper
• Attention-Enhanced Multi-Agent Reinforcement Learning Against Observation Perturbations for Distributed Volt-VAR Control

  2024cites this paper
• Spatiotemporal Deep Learning for Power System Applications: A Survey

  2024cites this paper
• A coordinated active and reactive power optimization approach for multi-microgrids connected to distribution networks with multi-actor-attention-critic deep reinforcement learning

  2024cites this paper
• Safety Deep Reinforcement Learning Approach to Voltage Control in Flexible Network Topologies

  2024cites this paper
• Research on Cluster Control of Distribution Network with High-Penetration Distributed Photovoltaic

  2024cites this paper
• Utilizing Deep Reinforcement Learning for High-Voltage Distribution Network Expansion Planning

  2024cites this paper
• Real-time power system dispatch scheme using grid expert strategy-based imitation learning

  2024cites this paper
• An Intelligent Coordinated Control System for Power Transformers Using Deep Q-Network

  2024cites this paper
• On the Accuarcy of Linear DistFlow Method: A Comparison Survey

  2024cites this paper
• Advancements in data-driven voltage control in active distribution networks: A Comprehensive review

  2024cites this paper
• Multi-Agent Safe Reinforcement Learning Based Real-Time Volt/Var Optimization for Modern Distribution Networks

  2024cites this paper
• Voltage regulation and energy loss minimization for distribution networks with high photovoltaic penetration and EV charging stations using dual-stage model predictive control

  2024cites this paper
• Robust Deep Reinforcement Learning for Volt-VAR Optimization in Active Distribution System Under Uncertainty

  2024cites this paper
• Bayesian Optimization for Deep Reinforcement Learning for Robust Volt-Var Control

  2024cites this paper
• Review of the opportunities and challenges to accelerate mass‐scale application of smart grids with large‐language models

  2024cites this paper
• Optimization Model for Reactive Voltage Control in Active Distribution Networks Based on Transformer Based Deep Reinforcement Learning

  2024cites this paper
• Energy Intelligence: A Systematic Review of Artificial Intelligence for Energy Management

  2024cites this paper
• Reactive Power and Voltage Optimization for Regional Power Grids with New Energy Stations under the Goal of “Double Carbon”

  2024cites this paper
• Reactive Power Optimization Method of Power Network Based on Deep Reinforcement Learning Considering Topology Characteristics

  2024cites this paper