A Survey of Multi-Objective Sequential Decision-Making

Sequential decision-making problems with multiple objectives arise naturally in practice and pose unique challenges for research in decision-theoretic planning and learning, which has largely focused on single-objective settings. This article surveys algorithms designed for sequential decision-making problems with multiple objectives. Though there is a growing body of literature on this subject, little of it makes explicit under what circumstances special methods are needed to solve multi-objective problems. Therefore, we identify three distinct scenarios in which converting such a problem to a single-objective one is impossible, infeasible, or undesirable. Furthermore, we propose a taxonomy that classifies multi-objective methods according to the applicable scenario, the nature of the scalarization function (which projects multi-objective values to scalar ones), and the type of policies considered. We show how these factors determine the nature of an optimal solution, which can be a single policy, a convex hull, or a Pareto front. Using this taxonomy, we survey the literature on multi-objective methods for planning and learning. Finally, we discuss key applications of such methods and outline opportunities for future work.

• Publication year

  2013

• Venue

  Journal of Artificial Intelligence Research

• Publication date

  2013-10-01

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1613/jair.3987arXiv 1402.0590
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Multiobjective Reinforcement Learning: A Comprehensive Overview

  2015cited by this paper
• A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run

  2013cited by this paper
• Hypervolume indicator and dominance reward based multi-objective Monte-Carlo Tree Search

  2013influential reference
• Reinforcement learning in robotics: A survey

  2013cited by this paper
• Computing Convex Coverage Sets for Multi-objective Coordination Graphs

  2013cited by this paper
• Linear fitted-Q iteration with multiple reward functions

  2013influential reference
• Tree-based fitted Q-iteration for multi-objective Markov decision processes in water resource management

  2013cited by this paper
• Approximation of Lorenz-Optimal Solutions in Multiobjective Markov Decision Processes

  2013cited by this paper
• Reinforcement Learning in Games

  2012influential reference
• Pareto Curves for Probabilistic Model Checking

  2012cited by this paper
• Preference-based reinforcement learning: a formal framework and a policy iteration algorithm

  2012cited by this paper
• A Theory of Goal-Oriented MDPs with Dead Ends

  2012cited by this paper
• Multi-objective reinforcement learning method for acquiring all pareto optimal policies simultaneously

  2012cited by this paper
• Stochastic Pareto local search: Pareto neighbourhood exploration and perturbation strategies

  2012cited by this paper
• SAVES: a sustainable multiagent application to conserve building energy considering occupants

  2012cited by this paper
• Evolutionary Computation for Reinforcement Learning

  2012cited by this paper
• Path-Constrained Markov Decision Processes: bridging the gap between probabilistic model-checking and decision-theoretic planning

  2012cited by this paper
• An Empirical Comparison of Two Common Multiobjective Reinforcement Learning Algorithms

  2012cited by this paper
• Stochastic Safest and Shortest Path Problems

  2012cited by this paper
• Reinforcement Learning and Markov Decision Processes

  2012cited by this paper
• Improving wet clutch engagement with reinforcement learning

  2012cited by this paper
• Multi-objective reinforcement learning based routing in cognitive radio networks: Walking in a random maze

  2012cited by this paper
• Multi Objective Resource Scheduling in LTE Networks Using Reinforcement Learning

  2012cited by this paper
• Multi-agent Multi-objective Learning Using Heuristically Accelerated Reinforcement Learning

  2012cited by this paper
• Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes

  2011cited by this paper
• An Evolutionary Algorithm with Advanced Goal and Priority Specification for Multi-objective Optimization

  2011cited by this paper
• Multi-objective fitted Q-iteration: Pareto frontier approximation in one single run

  2011influential reference
• On Minimizing Ordered Weighted Regrets in Multiobjective Markov Decision Processes

  2011cited by this paper
• Quantitative Multi-objective Verification for Probabilistic Systems

  2011cited by this paper
• Empirical evaluation methods for multiobjective reinforcement learning algorithms

  2011cited by this paper
• Evolving policies for multi-reward partially observable markov decision processes (MR-POMDPs)

  2011cited by this paper
• Monte-Carlo tree search and rapid action value estimation in computer Go

  2011cited by this paper
• Markov Decision Processes with Multiple Long-Run AverageObjectives

  2011cited by this paper
• Planning under Uncertainty for Robotic Tasks with Mixed Observability

  2010cited by this paper
• Multiobjective Reinforcement Learning for Traffic Signal Control Using Vehicular Ad Hoc Network

  2010cited by this paper
• Efficient Reinforcement Learning with Multiple Reward Functions for Randomized Controlled Trial Analysis

  2010influential reference
• Multi-objective optimization of freeway traffic flow via a fuzzy reinforcement learning method

  2010cited by this paper
• Multi-objective optimization by reinforcement learning for power system dispatch and voltage stability

  2010cited by this paper
• MULTI-OBJECTIVE REINFORCEMENT LEARNING FOR TRAFFIC SIGNAL COORDINATE CONTROL

  2010cited by this paper
• Multi-objective reinforcement learning algorithm for MOSDMP in unknown environment

  2010cited by this paper
• Tree‐based reinforcement learning for optimal water reservoir operation

  2010cited by this paper
• Enhanced Q-learning algorithm for dynamic power management with performance constraint

  2010cited by this paper
• Informing sequential clinical decision-making through reinforcement learning: an empirical study

  2010cited by this paper
• On Finding Compromise Solutions in Multiobjective Markov Decision Processes

  2010influential reference
• Solving Multi-objective Reinforcement Learning Problems by EDA-RL - Acquisition of Various Strategies

  2009cited by this paper
• Multi-policy Optimization in Self-organizing Systems

  2009cited by this paper
• GIS and Intelligent Agents for Multiobjective Natural Resource Allocation: A Reinforcement Learning Approach

  2009cited by this paper
• Learning controllers for human-robot interaction

  2009cited by this paper
• EDA-RL: estimation of distribution algorithms for reinforcement learning problems

  2009cited by this paper
• Dynamic scheduling of maintenance tasks in the petroleum industry: A reinforcement approach

  2009cited by this paper
• Constructing Stochastic Mixture Policies for Episodic Multiobjective Reinforcement Learning Tasks

  2009influential reference
• Responsive elastic computing

  2009cited by this paper
• Incorporating flood control rule curves of the Columbia River hydroelectric system in a multireservoir reinforcement learning optimization model

  2009cited by this paper
• A Reinforcement Learning Approach to Setting Multi-Objective Goals for Energy Demand Management

  2009cited by this paper
• Distributed W-Learning: Multi-Policy Optimization in Self-Organizing Systems

  2009cited by this paper
• Water reservoir control under economic, social and environmental constraints

  2008cited by this paper
• Learning all optimal policies with multiple criteria

  2008influential reference
• A Comprehensive Survey of Multiagent Reinforcement Learning

  2008cited by this paper
• Risk-aware decision making and dynamic programming

  2008cited by this paper
• Online Planning Algorithms for POMDPs

  2008cited by this paper
• Evolutionary many-objective optimization: A short review

  2008cited by this paper
• On the Limitations of Scalarisation for Multi-objective Reinforcement Learning of Pareto Fronts

  2008cited by this paper
• Safe physical human robot interaction-past, present and future

  2008cited by this paper
• Constrained reinforcement learning from intrinsic and extrinsic rewards

  2007cited by this paper
• Parallel reinforcement learning for weighted multi-criteria model with adaptive margin

  2007influential reference
• Computing Optimal Stationary Policies for Multi-Objective Markov Decision Processes

  2007cited by this paper
• Probabilistic Planning is Multi-objective!

  2007cited by this paper
• Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning

  2007cited by this paper
• Bandit Based Monte-Carlo Planning

  2006cited by this paper
• Anytime Point-Based Approximations for Large POMDPs

  2006cited by this paper
• Collective Multi-Objective Planning

  2006cited by this paper
• Reinforcement Learning for MDPs with Constraints

  2006influential reference
• Markov Decision Processes with Multiple Objectives

  2006cited by this paper
• Dynamic preferences in multi-criteria reinforcement learning

  2005cited by this paper
• Many-Objective Optimization: An Engineering Design Perspective

  2005cited by this paper
• Multiobjective decision processes under uncertainty : Applications, problem formulations, and solution strategies

  2005cited by this paper
• Perseus: Randomized Point-based Value Iteration for POMDPs

  2005cited by this paper
• APPLYING MACHINE LEARNING TO LOW‐KNOWLEDGE CONTROL OF OPTIMIZATION ALGORITHMS

  2005cited by this paper
• Tree-Based Batch Mode Reinforcement Learning

  2005influential reference
• Risk-Sensitive Reinforcement Learning Applied to Control under Constraints

  2005cited by this paper
• Influence diagrams with multiple objectives and tradeoff analysis

  2004cited by this paper
• A Geometric Approach to Multi-Criterion Reinforcement Learning

  2004cited by this paper
• Power systems stability control: reinforcement learning framework

  2004cited by this paper
• Decision-Theoretic Military Operations Planning

  2004cited by this paper
• Policy gradient reinforcement learning for fast quadrupedal locomotion

  2004cited by this paper
• Distributed Reinforcement Learning using Bi-directional Decision Making for Multi- criteria Control of Multi-Stage Flow Systems

  2003cited by this paper
• Transition-independent decentralized markov decision processes

  2003cited by this paper
• Q-Decomposition for Reinforcement Learning Agents

  2003cited by this paper
• Local episode-based learning of multi-objective behavior coordination for a mobile robot in dynamic environments

  2003cited by this paper
• Neural networks and reinforcement learning in control of water systems

  2003cited by this paper
• Performance assessment of multiobjective optimizers: an analysis and review

  2003cited by this paper
• Evolutionary Algorithms for Solving Multi-Objective Problems

  2002cited by this paper
• Reinforcement learning in the operational management of a water system

  2002cited by this paper
• A REINFORCEMENT LEARNING METHOD FOR DYNAMIC OBSTACLE AVOIDANCE IN ROBOTIC MECHANISMS

  2002cited by this paper
• Reinforcement Learning for Call Admission Control and Routing under Quality of Service Constraints in Multimedia Networks

  2002cited by this paper
• Reinforcement Learning with Bounded Risk

  2001cited by this paper
• Off-Policy Temporal Difference Learning with Function Approximation

  2001cited by this paper
• The Steering Approach for Multi-Criteria Reinforcement Learning

  2001influential reference
• Importance sampling for reinforcement learning with multiple objectives

  2001cited by this paper
• A New Approach for the Solution of Multiple Objective Optimization Problems Based on Reinforcement Learning

  2000cited by this paper

• Multi-objective reinforcement learning for provably incentivising alignment with value systems

  2026cites this paper
• A multi-objective goal-oriented reinforcement learning algorithm for dynamic multi-objective sequential decision making

  2026cites this paper
• Preference adaptive multi-objective HVAC control using a vectorized deep transformer Q-network

  2026cites this paper
• Learning Fair Policies in Multi-objective Preference-Based Reinforcement Learning

  2026cites this paper
• NeuroAction: a neuroevolutionary approach to reinforcement learning for autonomous vehicles

  2026cites this paper
• ALMo: Interactive Aim-Limit-Defined, Multi-Objective System for Personalized High-Dose-Rate Brachytherapy Treatment Planning and Visualization for Cervical Cancer

  2026cites this paper
• Stochastic Decision Horizons for Constrained Reinforcement Learning

  2026cites this paper
• Generalized policy improvement for efficient and robust multi-objective reinforcement learning

  2026cites this paper
• Reinforcement Learning for Multi-Objective Multi-Echelon Supply Chain Optimisation

  2026cites this paper
• Task Offloading Algorithm Based on Preference Weight Adaptive Multi-objective Reinforcement Learning for Cloud-Assisted Mobile Edge Computing in Internet of Vehicles

  2026cites this paper
• Adaptive scalarization in multi-objective reinforcement learning for enhanced robotic arm control

  2026cites this paper
• Interpretable group gated priority traces for scalarized multi-objective reinforcement learning in autonomous drone operations

  2026cites this paper
• Preference-based probabilistic planning with partially-ordered temporal goals

  2026cites this paper
• Online Navigation Planning for Long-term Autonomous Operation of Underwater Gliders

  2026cites this paper
• Optimizing Energy and Data Collection in UAV-aided IoT Networks using Attention-based Multi-Objective Reinforcement Learning

  2026cites this paper
• Probe-then-Commit Multi-Objective Bandits: Theoretical Benefits of Limited Multi-Arm Feedback

  2026cites this paper
• Multi-Attribute Utility Deep Reinforcement Learning method for Sequential Multi-Criteria Decision problems: Application to human resource planning

  2026cites this paper
• Post Hoc Extraction of Pareto Fronts for Continuous Control

  2026cites this paper
• Position: General Alignment Has Hit a Ceiling; Edge Alignment Must Be Taken Seriously

  2026cites this paper
• Tackling value estimation bias in successor features by distributional reinforcement learning

  2026cites this paper
• Reward-free Alignment for Conflicting Objectives

  2026cites this paper
• Hindsight Preference Replay Improves Preference-Conditioned Multi-Objective Reinforcement Learning

  2026cites this paper
• Service Migration Strategies Based on Partially Observable and Multi-Objective Optimization

  2026cites this paper
• An analysis of distributional reinforcement learning with Gaussian mixtures

  2026cites this paper
• Communication-Corruption Coupling and Verification in Cooperative Multi-Objective Bandits

  2026cites this paper
• Learning Policy Representations for Steerable Behavior Synthesis

  2026influential citation
• Prompt-to-policy: Leveraging large language models to guide deep reinforcement learning in public health emergencies.

  2026cites this paper
• LAD: Learning Advantage Distribution for Reasoning

  2026cites this paper
• Multiobjective Hierarchical Reinforcement Learning for Plant-Wide Optimization of Bayer Process

  2026cites this paper
• A multi-objective graph reinforcement learning framework for urban public facility location problem

  2026cites this paper
• Algorithm and Strategy Construction for Sure-Almost-Sure Stochastic Parity Games

  2026cites this paper
• MORL-IC: Multi-objective reinforcement learning approaches for analog integrated circuit optimization

  2026cites this paper
• Reward Engineering for Reinforcement Learning in Software Tasks

  2026cites this paper
• Evolving Interpretable Constitutions for Multi-Agent Coordination

  2026cites this paper
• Multi-Objective Site Selection of Underground Smart Parking Facilities Using NSGA-III: An Ecological-Priority Perspective

  2025cites this paper
• Iterative Environment Design for Deep Reinforcement Learning Based on Goal-Oriented Specification

  2025cites this paper
• Product Recommendation with Price Personalization According to Customer's Willingness to Pay Using Deep Reinforcement Learning

  2025cites this paper
• Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach

  2025influential citation
• Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learning

  2025influential citation
• AI Agents in Finance and Fintech: A Scientific Review of Agent-Based Systems, Applications, and Future Horizons

  2025cites this paper
• Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning

  2025cites this paper
• From fair solutions to compromise solutions in multi-objective deep reinforcement learning

  2025cites this paper
• Studying the Korean Word-Chain Game with RLVR: Mitigating Reward Conflicts via Curriculum Learning

  2025cites this paper
• An Approximate Embedding for Designing Ethical Reinforcement Learning Environments

  2025cites this paper
• Multi-objective optimization for dynamic logistics scheduling based on hierarchical deep reinforcement learning

  2025cites this paper
• Game-Theoretic Understandings of Multi-Agent Systems with Multiple Objectives

  2025cites this paper
• On Generalization Across Environments In Multi-Objective Reinforcement Learning

  2025cites this paper
• Coordination Requires Simplification: Thermodynamic Bounds on Multi-Objective Compromise in Natural and Artificial Intelligence

  2025cites this paper
• Markov Decision Processing Networks

  2025cites this paper
• Causally-Enhanced Reinforcement Policy Optimization

  2025cites this paper
• Pareto Set Learning for Multi-Objective Reinforcement Learning

  2025cites this paper
• Runtime Analysis of Evolutionary Algorithms for Multiparty Multiobjective Optimization

  2025cites this paper
• Anticipatory Networks: Concepts and Selected Applications

  2025cites this paper
• Fairness-constrained multigroup influence maximization

  2025cites this paper
• A Conversational Agent That Learns to be Aligned With the Moral Value of Respect

  2025cites this paper
• Towards a taxonomy of tasks for human sequential decision-making

  2025cites this paper
• HPRS: hierarchical potential-based reward shaping from task specifications

  2025cites this paper
• An Empirical Study of Lagrangian Methods in Safe Reinforcement Learning

  2025cites this paper
• Linear programming for finite-horizon vector-valued Markov decision processes

  2025cites this paper
• Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning

  2025cites this paper
• Multi-Objective Reinforcement Learning for Critical Scenario Generation of Autonomous Vehicles

  2025cites this paper
• An efficient all-pairs approach for multi-objective dynamic shortest path problems

  2025cites this paper
• Phenomenal interface theory: a model for basal consciousness

  2025cites this paper
• CHiPS: Composing Hierarchical Pareto Solutions for Scalable Planning in Multi-Objective MDPs

  2025cites this paper
• Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization

  2025cites this paper
• VORTEX: Aligning Task Utility and Human Preferences through LLM-Guided Reward Shaping

  2025cites this paper
• A Decomposition Optimization-Based Multiobjective Reinforcement Learning Algorithm for Obtaining Nonconvex Pareto Fronts

  2025cites this paper
• CoEvolution: A Comprehensive Trustworthy Framework For Connected Machine Learning And Secure Interconnected AI Solutions

  2025cites this paper
• Combining MORL with Restraining Bolts to Learn Normative Behaviour

  2025cites this paper
• LLM-Driven Pareto-Optimal Multi-Mode Reinforcement Learning for Adaptive UAV Navigation in Urban Wind Environments

  2025cites this paper
• Reinforcement Learning for Multi-Objective Multi-Echelon Supply Chain Optimisation

  2025cites this paper
• Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control

  2025cites this paper
• Fairness in Cooperative Multi-agent Multi-objective Reinforcement Learning using the Expected Scalarized Return

  2025cites this paper
• Hybrid reward-driven reinforcement learning for efficient quantum circuit synthesis

  2025cites this paper
• Multi objectives reinforcement learning for smart buildings: A systematic review of algorithms, applications and future perspectives

  2025cites this paper
• Optimizing market-making strategies: A multi-objective reinforcement learning approach with pareto fronts

  2025cites this paper
• Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light

  2025cites this paper
• CAPO: Towards Enhancing LLM Reasoning through Verifiable Generative Credit Assignment

  2025cites this paper
• Multi Objective Quantile Based Reinforcement Learning for Modern Urban Planning

  2025cites this paper
• Reinforcement Learning Agent for Multi-Objective Online Process Parameter Optimization of Manufacturing Processes

  2025cites this paper
• Proactive Data Placement in Heterogeneous Storage Systems via Predictive Multi-Objective Reinforcement Learning

  2025cites this paper
• Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds

  2025influential citation
• Mitigating Tail Latency for On-Device Inference With Load-Balanced Heterogeneous Models

  2025cites this paper
• Explainability for Unmanned Aerial Vehicle Control via Multi-Objective Reinforcement Learning

  2025cites this paper
• Hierarchical Sensing Framework for Polymer Degradation Monitoring: A Physics-Constrained Reinforcement Learning Framework for Programmable Material Discovery

  2025cites this paper
• The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models

  2025cites this paper
• Learning in Repeated Multi-Objective Stackelberg Games with Payoff Manipulation

  2025cites this paper
• AI-Powered Building Ecosystems: A Narrative Mapping Review on the Integration of Digital Twins and LLMs for Proactive Comfort, IEQ, and Energy Management

  2025cites this paper
• Generalizable Pareto-Optimal Offloading With Reinforcement Learning in Mobile Edge Computing

  2025influential citation
• Multi-Objective Reinforcement Learning for Data Collection-Energy Consumption Trade-off in UAV-Aided IoT Networks

  2025cites this paper
• Generative and Adaptive AI for Sustainable Supply Chain Design

  2025cites this paper
• Multi-objective reinforcement learning framework for beneficent artificial intelligence

  2025influential citation
• Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting

  2025cites this paper
• Scalable Multi-Objective Robot Reinforcement Learning through Gradient Conflict Resolution

  2025cites this paper
• Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning

  2025cites this paper
• Superconducting quantum computing optimization based on multi-objective deep reinforcement learning

  2025cites this paper
• Benchmarking and Improving LLM Robustness for Personalized Generation

  2025cites this paper
• Philosophy-informed Machine Learning

  2025cites this paper
• Grouped Satisficing Paths in Pure Strategy Games: a Topological Perspective

  2025cites this paper
• Adaptive fertilizer management for optimizing nitrogen use efficiency with constrained reinforcement learning

  2025cites this paper