Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control

Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleatoric uncertainty. Disentangling and evaluating these uncertainties simultaneously stands a chance of improving the agent's final performance, accelerating training, and facilitating quality assurance after deployment. In this work, we propose an uncertainty-aware reinforcement learning algorithm for continuous control tasks that extends the Deep Deterministic Policy Gradient algorithm (DDPG). It exploits epistemic uncertainty to accelerate exploration and aleatoric uncertainty to learn a risk-sensitive policy. We conduct numerical experiments showing that our variant of DDPG outperforms vanilla DDPG without uncertainty estimation in benchmark tasks on robotic control and power-grid optimization.

• Publication year

  2022

• Venue

  IEEE International Joint Conference on Neural Network

• Publication date

  2022-07-18

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/IJCNN55064.2022.9892771arXiv 2207.13730
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Risk-Averse Offline Reinforcement Learning

  2021cited by this paper
• Continuous Control With Ensemble Deep Deterministic Policy Gradients

  2021influential reference
• Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning

  2021influential reference
• Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

  2021cited by this paper
• Soft Actor-Critic With Integer Actions

  2021cited by this paper
• Exploration in Deep Reinforcement Learning: A Comprehensive Survey

  2021cited by this paper
• PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems

  2021cited by this paper
• ADER: Adapting between Exploration and Robustness for Actor-Critic Methods

  2021influential reference
• A Survey of Exploration Methods in Reinforcement Learning

  2021cited by this paper
• Deep Ensembles from a Bayesian Perspective

  2021cited by this paper
• GMAC: A Distributional Perspective on Actor-Critic Framework

  2021cited by this paper
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning

  2021influential reference
• Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation

  2021influential reference
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning

  2021cited by this paper
• AUTO-ENCODING VARIATIONAL BAYES

  2020cited by this paper
• Deep reinforcement learning for energy management in a microgrid with flexible demand

  2020cited by this paper
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges

  2020cited by this paper
• DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning

  2020influential reference
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning

  2020influential reference
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning

  2020cited by this paper
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

  2020influential reference
• Improving Robustness via Risk Averse Distributional Reinforcement Learning

  2020cited by this paper
• Reinforcement learning in sustainable energy and electric systems: a survey

  2020cited by this paper
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

  2020cited by this paper
• Sample-based Distributional Policy Gradient

  2020cited by this paper
• Worst Cases Policy Gradients

  2019cited by this paper
• Statistics and Samples in Distributional Reinforcement Learning

  2019influential reference
• Distributional Reinforcement Learning for Efficient Exploration

  2019influential reference
• Estimating Risk and Uncertainty in Deep Reinforcement Learning

  2019influential reference
• Distributional Deep Reinforcement Learning with a Mixture of Gaussians

  2019cited by this paper
• Better Exploration with Optimistic Actor-Critic

  2019cited by this paper
• Fully Parameterized Quantile Function for Distributional Reinforcement Learning

  2019cited by this paper
• Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy

  2019cited by this paper
• Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

  2019cited by this paper
• Addressing Function Approximation Error in Actor-Critic Methods

  2018cited by this paper
• UCB EXPLORATION VIA Q-ENSEMBLES

  2018cited by this paper
• Information-Directed Exploration for Deep Reinforcement Learning

  2018influential reference
• Uncertainty in Neural Networks: Approximately Bayesian Ensembling

  2018cited by this paper
• Implicit Quantile Networks for Distributional Reinforcement Learning

  2018influential reference
• Randomized Prior Functions for Deep Reinforcement Learning

  2018cited by this paper
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

  2018cited by this paper
• Distributional Advantage Actor-Critic

  2018cited by this paper
• A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

  2018cited by this paper
• ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search

  2018cited by this paper
• Reinforcement Learning Approach for Optimal Distributed Energy Management in a Microgrid

  2018cited by this paper
• Distributed Distributional Deterministic Policy Gradients

  2018influential reference
• Smart Grid Optimization by Deep Reinforcement Learning over Discrete and Continuous Action Space

  2018cited by this paper
• Deep Reinforcement Learning with Risk-Seeking Exploration

  2018cited by this paper
• Self-Adaptive Double Bootstrapped DDPG

  2018influential reference
• Uncertainty-driven Imagination for Continuous Deep Reinforcement Learning

  2017cited by this paper
• Distributional Reinforcement Learning with Quantile Regression

  2017influential reference
• A Distributional Perspective on Reinforcement Learning

  2017cited by this paper
• Learning to Run with Actor-Critic Ensemble

  2017cited by this paper
• Deep Exploration via Bootstrapped DQN

  2016influential reference
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

  2016cited by this paper
• Learning the Variance of the Reward-To-Go

  2016cited by this paper
• Continuous control with deep reinforcement learning

  2015cited by this paper
• A comprehensive survey on safe reinforcement learning

  2015cited by this paper
• Human-level control through deep reinforcement learning

  2015cited by this paper
• Prioritized Experience Replay

  2015cited by this paper
• Deep Reinforcement Learning with Double Q-Learning

  2015cited by this paper
• A practical guide to robust optimization

  2015cited by this paper
• Algorithms for CVaR Optimization in MDPs

  2014cited by this paper
• Temporal Difference Methods for the Variance of the Reward To Go

  2013cited by this paper
• Risk-Sensitive Reinforcement Learning

  2013cited by this paper
• Parametric Return Density Estimation for Reinforcement Learning

  2010cited by this paper
• Double Q-learning

  2010cited by this paper
• Nonparametric Return Distribution Approximation for Reinforcement Learning

  2010cited by this paper
• Aleatory or epistemic? Does it matter?

  2009cited by this paper
• MUTUAL FUND PERFORMANCE*

  2007cited by this paper
• Optimization of conditional value-at risk

  2000cited by this paper
• Value at Risk: The New Benchmark for Managing Financial Risk

  2000cited by this paper
• Bayesian Q-Learning

  1998cited by this paper
• Advances in prospect theory: Cumulative representation of uncertainty

  1992cited by this paper
• The variance of discounted Markov decision processes

  1982cited by this paper

• A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective

  2025cites this paper
• Reinforcement Learning to Achieve Real-time Control of a Quadruple Inverted Pendulum

  2025cites this paper
• UACER: An Uncertainty-Adaptive Critic Ensemble Framework for Robust Adversarial Reinforcement Learning

  2025cites this paper
• A Hybrid Decision-Making Framework for UAV-Assisted MEC Systems: Integrating a Dynamic Adaptive Genetic Optimization Algorithm and Soft Actor–Critic Algorithm with Hierarchical Action Decomposition and Uncertainty-Quantified Critic Ensemble

  2025cites this paper
• Ensemble Distribution Distillation for Self-Supervised Human Activity Recognition

  2025cites this paper
• Uncertainty Quantification for Efficient and Risk-Sensitive Reinforcement Learning

  2023cites this paper
• Navigating autonomous vehicles in uncertain environments with distributional reinforcement learning

  2023cites this paper