Energy Storage Management via Deep Q-Networks

Energy storage devices represent environmentally friendly candidates to cope with volatile renewable energy generation. Motivated by the increase in privately owned storage systems, this paper studies the problem of real-time control of a storage unit co-located with a renewable energy generator and an inelastic load. Unlike many approaches in the literature, no distributional assumptions are being made on the renewable energy generation or the real-time prices. Building on the deep Q-networks algorithm, a reinforcement learning approach utilizing a neural network is devised where the storage unit operational constraints are respected. The neural network approximates the action-value function which dictates what action (charging, discharging, etc.) to take. Simulations indicate that near-optimal performance can be attained with the proposed learning-based control policy for the storage units.

• Publication year

  2019

• Venue

  IEEE Power & Energy Society General Meeting

• Publication date

  2019-03-26

• Fields of study

  Mathematics, Computer Science, Engineering, Environmental Science

• Identifiers
  DOI 10.1109/PESGM40551.2019.8973808arXiv 1903.11107
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Optimal Distributed Energy Storage Management Using Relaxed Dantzig-Wolfe Decomposition

  2018cited by this paper
• Optimal Operation and Economic Value of Energy Storage at Consumer Locations

  2017cited by this paper
• Value of storage technologies for wind and solar energy

  2016cited by this paper
• Real-Time Energy Storage Management for Renewable Integration in Microgrid: An Off-Line Optimization Approach

  2015cited by this paper
• Wind power curtailment and energy storage in transmission congestion management considering power plants ramp rates

  2015cited by this paper
• Near-Optimal Method for Siting and Sizing of Distributed Storage in a Transmission Network

  2015cited by this paper
• Human-level control through deep reinforcement learning

  2015influential reference
• Assessing the economic value and optimal structure of large-scale electricity storage

  2013cited by this paper
• Playing Atari with Deep Reinforcement Learning

  2013cited by this paper
• Optimal power flow with large-scale storage integration

  2013cited by this paper
• Development of a High Resolution, Real Time, Distribution-Level Metering System and Associated Visualization, Modeling, and Data Analysis Functions

  2013cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Optimal Management Strategy of a Battery-Based Storage System to Improve Renewable Energy Integration in Distribution Networks

  2012cited by this paper
• Sizing Energy Storage to Accommodate High Penetration of Variable Energy Resources

  2012cited by this paper
• Cutting Down Electricity Cost in Internet Data Centers by Using Energy Storage

  2011cited by this paper
• Optimal Allocation of ESS in Distribution Systems With a High Penetration of Wind Energy

  2010cited by this paper
• The economics of using plug-in hybrid electric vehicle battery packs for grid storage

  2010cited by this paper
• Welfare Impacts of Electricity Storage and the Implications of Ownership Structure

  2010cited by this paper
• Modeling Real-Time Balancing Power Market Prices Using Combined SARIMA and Markov Processes

  2008cited by this paper
• Reinforcement Learning: An Introduction

  1998influential reference
• Reinforcement Learning for Robots Using Neural Networks

  1992cited by this paper

• Imbalance-Aware Scheduling for PV-Battery Storage Systems Using Deep Reinforcement Learning

  2025cites this paper
• Reinforcement Learning for Energy Storage Management in Renewable Systems

  2025cites this paper
• Should we use model-free or model-based control? A case study of battery management systems

  2024cites this paper
• Safe Deep Reinforcement Learning-Based Real-Time Multi-Energy Management in Combined Heat and Power Microgrids

  2024cites this paper
• Should We Use Model-Free or Model-Based Control? A Case Study of Battery Control

  2024cites this paper
• Routing and Scheduling of Mobile Energy Storage System for Electricity Arbitrage Based on Two-Layer Deep Reinforcement Learning

  2023cites this paper
• Learning-Accelerated ADMM for Distributed DC Optimal Power Flow

  2022cites this paper
• Battery control with lookahead constraints in distribution grids using reinforcement learning

  2022cites this paper
• A Multi-Use Framework of Energy Storage Systems Using Reinforcement Learning for Both Price-Based and Incentive-Based Demand Response Programs

  2022cites this paper
• A reinforcement learning-based demand response strategy designed from the Aggregator’s perspective

  2022cites this paper
• Reinforcement Learning-Based Optimal Battery Control Under Cycle-Based Degradation Cost

  2021cites this paper
• Real-time Autonomous Line Flow Control Using Proximal Policy optimization

  2020cites this paper
• Review of Control Strategies for Lithium-ion Battery Energy Storage Systems in Distribution Networks

  2020cites this paper
• Real-time Voltage Control Using Deep Reinforcement Learning

  2019cites this paper
• Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning

  2019cites this paper
• Learning-Accelerated ADMM for Distributed Optimal Power Flow

  2019cites this paper