Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-value-at-risk (CVaR) objective, as opposed to a standard risk-neutral expectation. We refer to such problem as CVaR MDP. Our first contribution is to show that a CVaR objective, besides capturing risk sensitivity, has an alternative interpretation as expected cost under worst-case modeling errors, for a given error budget. This result, which is of independent interest, motivates CVaR MDPs as a unifying framework for risk-sensitive and robust decision making. Our second contribution is to present an approximate value-iteration algorithm for CVaR MDPs and analyze its convergence rate. To our knowledge, this is the first solution algorithm for CVaR MDPs that enjoys error guarantees. Finally, we present results from numerical experiments that corroborate our theoretical findings and show the practicality of our approach.

• Publication year

  2015

• Venue

  Neural Information Processing Systems

• Publication date

  2015-06-06

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1506.02188
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Time-Consistent Decisions and Temporal Decomposition of Coherent Risk Functionals

  2016cited by this paper
• A Convex Analytic Approach to Risk-Aware Markov Decision Processes

  2015influential reference
• Algorithms for CVaR Optimization in MDPs

  2014cited by this paper
• Policy Gradients for CVaR-Constrained MDPs

  2014cited by this paper
• Optimizing the CVaR via Sampling

  2014cited by this paper
• Policy Gradients for CVaR-Constrained MDPs

  2014cited by this paper
• Fast gradient descent method for Mean-CVaR optimization

  2013cited by this paper
• Lightning Does Not Strike Twice: Robust MDPs with Coupled Uncertainty

  2012cited by this paper
• Robustness and risk-sensitivity in Markov decision processes

  2012cited by this paper
• Markov Decision Processes with Average-Value-at-Risk criteria

  2011cited by this paper
• Dynamic Programming and Optimal Control

  2010influential reference
• Risk-constrained Markov decision processes

  2010influential reference
• Conditional Value at Risk (CoVAR)

  2010cited by this paper
• Bias and Variance Approximation in Value Function Estimates

  2007cited by this paper
• Master Funds in Portfolio Analysis with General Deviation Measures

  2006cited by this paper
• The Robustness-Performance Tradeoff in Markov Decision Processes

  2006cited by this paper
• Robust Dynamic Programming

  2005cited by this paper
• Robust Control of Markov Decision Processes with Uncertain Transition Matrices

  2005cited by this paper
• Interpolation and Approximation by Polynomials

  2003cited by this paper
• Envelope Theorems for Arbitrary Choice Sets

  2002cited by this paper
• Measuring Market Risk

  2002cited by this paper
• Optimization of conditional value-at risk

  2000influential reference
• Coherent Measures of Risk

  1999influential reference
• Risk Sensitive Markov Decision Processes

  1997cited by this paper
• Percentile performance criteria for limiting average Markov decision processes

  1995cited by this paper
• The variance of discounted Markov decision processes

  1982cited by this paper
• Dynamic Programming and Optimal Control, Vol. II

  1976influential reference

• Exploring the construction and optimization of collaborative forest fire rescue systems in China

  2026cites this paper
• From Risk-Neutral to Risk-Sensitive Reinforcement Learning: Actor–Critic vs REINFORCE with Tail-Based Risk Measures

  2026cites this paper
• Robust Policy Optimization to Prevent Catastrophic Forgetting

  2026cites this paper
• Dynamic Programming for Epistemic Uncertainty in Markov Decision Processes

  2026cites this paper
• New Bayesian optimisation framework for robust multi–objective design: decoupling performance and uncertainty

  2026cites this paper
• Online Risk-Averse Planning in POMDPs Using Iterated CVaR Value Function

  2026influential citation
• Cross-Domain Deep Reinforcement Learning for Real-Time Resource Allocation in Transportation Hubs: From Airport Gates to Seaport Berths

  2026cites this paper
• Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity

  2026cites this paper
• Tackling value estimation bias in successor features by distributional reinforcement learning

  2026cites this paper
• Efficient Risk-sensitive Planning via Entropic Risk Measures

  2025cites this paper
• Dynamic Reinsurance Treaty Bidding via Multi-Agent Reinforcement Learning

  2025cites this paper
• Renewable Energy Following Strategy via Quantified Aggregated Heterogeneous Electric Vehicle Flexibility

  2025cites this paper
• Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions

  2025cites this paper
• Data Center Job Scheduling and Energy Management Under Uncertain Environments

  2025cites this paper
• Risk-Aware Decision Making for Human–Robot Collaboration With Effective Communication

  2025cites this paper
• State Entropy Regularization for Robust Reinforcement Learning

  2025cites this paper
• Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning

  2025cites this paper
• Tail-Optimized Caching for LLM Inference

  2025cites this paper
• FR-LUX: Friction-Aware, Regime-Conditioned Policy Optimization for Implementable Portfolio Management

  2025cites this paper
• The Evolution of IBM's Quantum Information Software Kit (Qiskit): A Review of its Applications

  2025cites this paper
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training

  2025cites this paper
• Risk-Aware Trajectory Optimization and Control for an Underwater Suspended Robotic System

  2025cites this paper
• Quantum Decision Transformers (QDT): Synergistic Entanglement and Interference for Offline Reinforcement Learning

  2025cites this paper
• On the Fundamental Limitations of Dual Static CVaR Decompositions in Markov Decision Processes

  2025influential citation
• Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment

  2025cites this paper
• On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration

  2025cites this paper
• Extreme Scenario Characterization for High Renewable Energy Penetrated Power Systems over Long Time Scales

  2025cites this paper
• Risk-Averse Fine-tuning of Large Language Models

  2025influential citation
• Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments

  2025cites this paper
• Measures of Variability for Risk-averse Policy Gradient

  2025cites this paper
• ROSCOM: Robust Safe Reinforcement Learning on Stochastic Constraint Manifolds

  2025cites this paper
• ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning

  2025cites this paper
• Robust & Low-Complexity Task Scheduling Algorithms for a Mobile Edge Computing System

  2025cites this paper
• Risking your Tail: Modeling Individual Differences in Risk-sensitive Exploration using Bayes Adaptive Markov Decision Processes

  2025influential citation
• Socially Aware Robot Crowd Navigation via Online Uncertainty-Driven Risk Adaptation

  2025cites this paper
• Individual differences in tail risk sensitive exploration using Bayes-adaptive Markov decision processes

  2025cites this paper
• Markov Decision Processes with Value-at-Risk Criterion

  2025cites this paper
• Tail Distribution of Regret in Optimistic Reinforcement Learning

  2025cites this paper
• Risk-Sensitive Microgrid Management Using Proximal Policy Optimization

  2025cites this paper
• Multi-Stage Adaptive Robust Scheduling Framework for Nonlinear Solar-Integrated Transportation Networks

  2025cites this paper
• Optimism as Risk-Seeking in Multi-Agent Reinforcement Learning

  2025cites this paper
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty

  2025cites this paper
• Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations

  2025cites this paper
• Adversarial Diffusion for Robust Reinforcement Learning

  2025cites this paper
• Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation

  2025cites this paper
• Time-aware Motion Planning in Dynamic Environments with Conformal Prediction

  2025cites this paper
• Robustness to Modeling Errors in Risk-Sensitive Markov Decision Problems With Markov Risk Measures

  2025cites this paper
• Risk-Sensitive Reinforcement Learning With ϕ-Divergence-Risk

  2025cites this paper
• Performance Enhanced Risk-Aware MPPI Using Gaussian Process for Robust Mobile Robot Control

  2025cites this paper
• Posted Pricing for Online Selection: Limited Price Changes and Risk Sensitivity

  2025cites this paper
• A primal-dual approach to double-risk-constrained LQR for practical control under non-Gaussian noise.

  2025cites this paper
• Semismooth Newton Methods for Risk-Averse Markov Decision Processes

  2025cites this paper
• Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning

  2025cites this paper
• Risk-averse policies for natural gas futures trading using distributional reinforcement learning

  2025cites this paper
• Yet Another Distributional Bellman Equation

  2025cites this paper
• GPU-Accelerated SPOCK for Scenario-Based Risk-Averse Optimal Control Problems

  2025cites this paper
• Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation

  2025cites this paper
• Decentralized Reinforcement Learning with Risk Aversion in Multi-Agent Systems

  2025cites this paper
• Integrating reinforcement learning with visual generative models: foundations and advances

  2025cites this paper
• RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization

  2025cites this paper
• Multi-trajectory optimization for train using distributional reinforcement learning with conditional value-at-risk

  2025cites this paper
• Adaptive Insurance Reserving with CVaR-Constrained Reinforcement Learning under Macroeconomic Regimes

  2025cites this paper
• Bounding Conditional Value-at-Risk via Auxiliary Distributions with Bounded Discrepancies

  2025cites this paper
• Risk-aware Direct Preference Optimization under Nested Risk Measure

  2025cites this paper
• Risk-Averse Traversal of Graphs with Stochastic and Correlated Edge Costs for Safe Global Planetary Mobility

  2025cites this paper
• A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety

  2025cites this paper
• Uncertainty-Aware Strategies: A Model-Agnostic Framework for Robust Financial Optimization through Subsampling

  2025cites this paper
• Threshold UCT: Cost-Constrained Monte Carlo Tree Search with Pareto Curves

  2024cites this paper
• Enhancing Risk Assessment in Transformers with Loss-at-Risk Functions

  2024cites this paper
• Risk-Aware Decision Making in Restless Bandits: Theory and Algorithms for Planning and Learning

  2024cites this paper
• Moor: Model-based offline policy optimization with a risk dynamics model

  2024influential citation
• Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis

  2024influential citation
• Revisiting Safe Exploration in Safe Reinforcement learning

  2024cites this paper
• A Review of Safe Reinforcement Learning: Methods, Theories, and Applications

  2024cites this paper
• POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning

  2024cites this paper
• Bootstrapping Expectiles in Reinforcement Learning

  2024cites this paper
• Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning

  2024cites this paper
• Risk Sensitive Distributional Soft Actor Critic for Portfolio Management

  2024cites this paper
• Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory

  2024cites this paper
• Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

  2024cites this paper
• Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk

  2024influential citation
• Offline Bayesian Aleatoric and Epistemic Uncertainty Quantification and Posterior Value Optimisation in Finite-State MDPs

  2024cites this paper
• Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

  2024cites this paper
• Sequential decision-making under uncertainty: a robust MDPs review

  2024cites this paper
• Simplification of Risk Averse POMDPs with Performance Guarantees

  2024cites this paper
• Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence

  2024cites this paper
• Peak Time-Windowed Risk Estimation of Stochastic Processes

  2024cites this paper
• Constrained Reinforcement Learning for Safe Heat Pump Control

  2024cites this paper
• Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation

  2024cites this paper
• Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning

  2024cites this paper
• Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts

  2024cites this paper
• Dynamic Capital Requirements for Markov Decision Processes

  2024influential citation
• CVaR-Constrained Policy Optimization for Safe Reinforcement Learning

  2024cites this paper
• A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization

  2024influential citation
• Networks of robust wireless charging sensor grids

  2024cites this paper
• Constrained Reinforcement Learning with Smoothed Log Barrier Function

  2024cites this paper
• Optimal control barrier functions for RL based safe powertrain control

  2024cites this paper
• A Reinforcement Learning Benchmark for Autonomous Driving in General Urban Scenarios

  2024cites this paper
• Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes

  2024cites this paper
• Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively

  2024cites this paper