Constrained discounted Markov decision processes with Borel state spaces

We study discrete-time discounted constrained Markov decision processes (CMDPs) on Borel spaces with unbounded reward functions. In our approach the transition probability functions are weakly or set-wise continuous. The reward functions are upper semicontinuous in state-action pairs or semicontinuous in actions. Our aim is to study models with unbounded reward functions, which are often encountered in applications, e.g., in consumption/investment problems. We provide some general assumptions under which the optimization problems in CMDPs are solvable in the class of stationary randomized policies. Then, we indicate that if the initial distribution and transition probabilities are non-atomic, then using a general purification result of Feinberg and Piunovskiy, stationary optimal policies can be deterministic. Our main results are illustrated by five examples.

• Publication year

  2018

• Venue

  at - Automatisierungstechnik

• Publication date

  2018-06-01

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1016/j.automatica.2019.108582arXiv 1806.00190
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Topological Spaces

  2019influential reference
• Fatou's Lemma for Weakly Converging Measures under the Uniform Integrability Condition

  2018influential reference
• Sufficiency of Deterministic Policies for Atomless Discounted and Uniformly Absorbing MDPs with Multiple Criteria

  2018cited by this paper
• Finite Approximations in Discrete-Time Stochastic Control: Quantized Models and Asymptotic Optimality

  2018cited by this paper
• On the Expected Total Reward with Unbounded Returns for Markov Decision Processes

  2017influential reference
• A stochastic search algorithm for voltage and reactive power control with switching costs and ZIP load model

  2016cited by this paper
• Conditions for the Solvability of the Linear Programming Formulation for Constrained Discounted Markov Decision Processes

  2016influential reference
• Stochastic Dynamic Programming

  2016cited by this paper
• Stochastic Dynamic Programming

  2016influential reference
• Stationary Markov perfect equilibria in risk sensitive stochastic overlapping generations models

  2014cited by this paper
• Stationary Markov perfect equilibria in discounted stochastic games

  2013cited by this paper
• Constrained Markov control processes with randomized discounted cost criteria: Occupation measures and extremal points

  2013cited by this paper
• Splitting Randomized Stationary Policies in Total-Reward Markov Decision Processes

  2012influential reference
• Stochastic Games with Unbounded Payoffs: Applications to Robust Control in Economics

  2011cited by this paper
• Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors

  2011influential reference
• Discounted dynamic programming with unbounded returns: Application to economic models

  2011influential reference
• An Approximate Stochastic Annealing algorithm for finite horizon Markov decision processes

  2010cited by this paper
• Economic Dynamics: Theory and Computation

  2009cited by this paper
• Unbounded mappings and weak convergence of measures

  2008cited by this paper
• Non-randomized policies for constrained Markov decision processes

  2007cited by this paper
• Stochastic optimal control : the discrete time case

  2007cited by this paper
• On the Dvoretzky–Wald–Wolfowitz Theorem on Nonrandomized Statistical Decisions

  2006cited by this paper
• Extreme Points of Sets of Randomized Strategies in Constrained Optimization and Control Problems

  2005cited by this paper
• Dynamic programming equations for discounted constrained stochastic control

  2004influential reference
• Strategic measures in optimal control problems for stochastic sequences

  2000influential reference
• Nonatomic total rewards Markov decision processes with multiple criteria

  2000cited by this paper
• Constrained Markovian decision processes: the dynamic programming approach

  2000cited by this paper
• Multiple Objective Nonatomic Markov Decision Processes with Total Reward Criteria

  2000influential reference
• Constrained Markov control processes in Borel spaces: the discounted case

  2000cited by this paper
• Discrete-time Markov control processes

  1999cited by this paper
• Constrained dynamic programming with two discount factors: applications and an algorithm

  1999cited by this paper
• Constrained Markov Decision Processes

  1999influential reference
• Controlled Markov set-chains with discounting

  1998cited by this paper
• Constrained Markov decision processes with total cost criteria: Lagrangian approach and dual linear program

  1998influential reference
• Book Review: Discrete-time Markov control processes: Basic optimality criteria

  1997influential reference
• Relaxation in Optimization Theory and Variational Calculus

  1997cited by this paper
• Optimal Control of Random Sequences in Problems with Constraints

  1997influential reference
• Notes on equivalent stationary policies in Markov decision processes with total rewards

  1996cited by this paper
• Constrained Discounted Dynamic Programming

  1996cited by this paper
• Constrained Markov Decision Models with Weighted Discounted Rewards

  1995cited by this paper
• Ergodic Control of Markov Chains with Constraints---The General Case

  1994influential reference
• Constrained Discounted Markov Decision Chains

  1991cited by this paper
• Scorza Dragoni type theorems

  1991cited by this paper
• On compactness of the space of policies in stochastic dynamic programming

  1989influential reference
• Constrained admission control to a queueing system

  1989cited by this paper
• A convex analytic approach to Markov decision processes

  1988influential reference
• On the weak topology on a space of probability measures induced by policies

  1988influential reference
• Flow control protocols for integrated networks with partially observed voice traffic

  1987cited by this paper
• Classical descriptive set theory

  1987influential reference
• Optimal flow control of a class of queueing networks in equilibrium

  1983cited by this paper
• On Dynamic Programming and Statistical Decision Theory

  1979influential reference
• Measure and integral

  1978cited by this paper
• Regular Conditional Expectations of Correspondences

  1977cited by this paper
• Markov programming by successive approximations with respect to weighted supremum norms

  1977cited by this paper
• On dynamic programming: Compactness of the space of policies

  1975influential reference
• Measurable Selections of Extrema

  1973cited by this paper
• On Optimal Strategies in Control Problems with Constraints

  1972cited by this paper
• Constrained Markov Decision Chains

  1972cited by this paper
• Real Analysis

  1972cited by this paper
• Atomes conditionnels d'un espace de probabilité

  1966cited by this paper
• Negative Dynamic Programming

  1966cited by this paper
• A Note on Memoryless Rules for Controlling Sequential Control Processes

  1966cited by this paper
• Mathematical foundations of the calculus of probability

  1965influential reference
• Markov Processes and Their Applications

  1962cited by this paper
• Elimination of Randomization in Certain Statistical Decision Procedures and Zero-Sum Two-Person Games

  1951cited by this paper
• Control and Cybernetics Dynamic Programming in Constrained Markov Decision Processes

  year unknowninfluential reference

• Discrete-time minimax hybrid control with cost constraints

  2026cites this paper
• Absorbing Markov Decision Processes: Geometric Properties and Sufficiency of Finite Mixtures of Deterministic Policies

  2025cites this paper
• On the Feinberg-Piunovskiy Theorem and its extension to chattering policies

  2025influential citation
• Mean-Variance optimization in discrete-time decision processes with general utility function

  2025cites this paper
• Extreme Occupation Measures in Markov Decision Processes with an Absorbing State

  2024cites this paper
• Constrained Markov Decision Processes with Non-constant Discount Factor

  2024cites this paper
• Multistep Look-Ahead Policy Iteration for Optimal Control of Discrete-Time Nonlinear Systems With Isoperimetric Constraints

  2024cites this paper
• Extreme occupation measures in Markov decision processes with a cemetery

  2023cites this paper
• Constrained-Cost Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

  2023cites this paper
• A Strong Duality Result for Cooperative Decentralized Constrained POMDPs

  2023cites this paper
• Cooperative Multi-Agent Constrained POMDPs: Strong Duality and Primal-Dual Reinforcement Learning with Approximate Information States

  2023cites this paper
• Approximate Constrained Discounted Dynamic Programming With Uniform Feasibility and Optimality

  2023cites this paper
• On Approximate and Weak Correlated Equilibria in Constrained Discounted Stochastic Games

  2022cites this paper
• Zero-sum risk-sensitive stochastic games with unbounded payoff functions and varying discount factors

  2022cites this paper
• Constrained Discounted Stochastic Games

  2021influential citation
• Necessity of the terminal condition in the infinite horizon dynamic optimization problems with unbounded payoff

  2021cites this paper
• Unbounded dynamic programming via the Q-transform

  2020influential citation
• A note on the existence of optimal stationary policies for average Markov decision processes with countable states

  2020cites this paper
• Fatou's Lemma for Weakly Converging Measures under the Uniform Integrability Condition

  2018cites this paper