Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems

This paper presents a novel method of global adaptive dynamic programming (ADP) for the adaptive optimal control of nonlinear polynomial systems. The strategy consists of relaxing the problem of solving the Hamilton-Jacobi-Bellman (HJB) equation to an optimization problem, which is solved via a new policy iteration method. The proposed method distinguishes from previously known nonlinear ADP methods in that the neural network approximation is avoided, giving rise to significant computational improvement. Instead of semiglobally or locally stabilizing, the resultant control policy is globally stabilizing for a general class of nonlinear polynomial systems. Furthermore, in the absence of the a priori knowledge of the system dynamics, an online learning method is devised to implement the proposed policy iteration technique by generalizing the current ADP theory. Finally, three numerical examples are provided to validate the effectiveness of the proposed method.

• Publication year

  2013

• Venue

  IEEE Transactions on Automatic Control

• Publication date

  2013-12-30

• Fields of study

  Mathematics, Computer Science, Engineering

• Identifiers
  DOI 10.1109/TAC.2015.2414811arXiv 1401.0020
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Decentralized Adaptive Optimal Control of Large-Scale Systems With Application to Power Systems

  2015cited by this paper
• Approximate dynamic programming via iterated Bellman inequalities

  2015cited by this paper
• Adaptive Control Design and Analysis (

  2014cited by this paper
• Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles

  2014cited by this paper
• Optimal Controller Synthesis for Nonlinear Dynamical Systems

  2014cited by this paper
• Semidefinite relaxations for stochastic optimal control policies

  2014cited by this paper
• Adaptive dynamic programming and optimal control of nonlinear nonaffine systems

  2014cited by this paper
• Robust Adaptive Dynamic Programming and Feedback Stabilization of Nonlinear Systems

  2014cited by this paper
• Linearly Solvable Stochastic Control Lyapunov Functions

  2014cited by this paper
• Reinforcement Learning and Approximate Dynamic Programming (RLADP)Â -Â Foundations, Common Misconceptions, and the Challenges Ahead

  2013cited by this paper
• Robust adaptive dynamic programming for linear and nonlinear systems: An overview

  2013cited by this paper
• Robust control synthesis of polynomial nonlinear systems using sum of squares technique

  2013cited by this paper
• I and J

  2012cited by this paper
• Reinforcement Learning and Approximate Dynamic Programming for Feedback Control

  2012cited by this paper
• Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics

  2012cited by this paper
• Approximate dynamic programming via sum of squares programming

  2012cited by this paper
• Dynamic Approximate Solutions of the HJ Inequality and of the HJB Equation for Input-Affine Nonlinear Systems

  2012cited by this paper
• Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles

  2012cited by this paper
• Optimal Control: Lewis/Optimal Control 3e

  2012cited by this paper
• Semidefinite Optimization and Convex Algebraic Geometry

  2012cited by this paper
• Constrained Nonlinear Polynomial Time-Delay Systems: A Sum-of-Squares Approach to Estimate the Domain of Attraction

  2012cited by this paper
• Constructive Nonlinear Control

  2011cited by this paper
• Performance bounds and suboptimal policies for linear stochastic control via LMIs

  2011cited by this paper
• Discrete-time stochastic optimal control via occupation measures and moment relaxations

  2009cited by this paper
• Simultaneous Stabilization and Robust Control of Polynomial Nonlinear Systems Using SOS Techniques

  2009cited by this paper
• Neuro-Dynamic Programming

  2009cited by this paper
• Reinforcement learning and adaptive dynamic programming for feedback control

  2009cited by this paper
• Approximate dynamic programming: solving the curses of dimensionality

  2009cited by this paper
• Reinforcement Learning and Adaptive Dynamic Programming for Feedback Control

  2009cited by this paper
• 2009 Special Issue: Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems

  2009cited by this paper
• Adaptive Dynamic Programming: An Introduction

  2009cited by this paper
• Optimal control

  2008cited by this paper
• Dynamic Programming and Optimal Control Problem Set: Deterministic Systems and the Shortest Path Problem

  2008cited by this paper
• Simultaneous Stabilization and Robust Control for Polynomial Nonlinear Systems Using SOS

  2007cited by this paper
• Approximate Dynamic Programming

  2007cited by this paper
• Influence of cultivation temperature on the ligninolytic activity of selected fungal strains

  2006cited by this paper
• Relaxing dynamic programming

  2006cited by this paper
• SOSTOOLS and its control applications

  2005influential reference
• Stabilization of nonaffine systems: a constructive method for polynomial systems

  2005cited by this paper
• Nonlinear control synthesis by sum of squares optimization: a Lyapunov-based approach

  2004cited by this paper
• Handbook of Learning and Approximate Dynamic Programming (IEEE Press Series on Computational Intelligence)

  2004cited by this paper
• YALMIP : a toolbox for modeling and optimization in MATLAB

  2004cited by this paper
• Nonlinear Systems

  2004influential reference
• The Linear Programming Approach to Approximate Dynamic Programming

  2003cited by this paper
• Control of sandwich nonlinear systems

  2003cited by this paper
• Active suspension design using linear parameter varying control

  2003cited by this paper
• GloptiPoly: global optimization over polynomials with Matlab and SeDuMi

  2002cited by this paper
• Introducing SOSTOOLS: a general purpose sum of squares programming solver

  2002cited by this paper
• Semidefinite programming

  2002cited by this paper
• Global robust stabilization of cascaded polynomial systems

  2002cited by this paper
• SOSTOOLS: Sum of squares optimization toolbox for MATLAB

  2002cited by this paper
• Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization

  2000influential reference
• L2-Gain and Passivity in Nonlinear Control

  1999cited by this paper
• Forward Completeness, Unboundedness Observability, and their Lyapunov Characterizations

  1999cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Useful nonlinearities and global stabilization of bifurcations in a model of jet engine surge and stall

  1998cited by this paper
• Design of Robust Adaptive Controllers for Nonlinear Systems with Dynamic Uncertainties

  1998cited by this paper
• Inverse optimal design of input-to-state stabilizing nonlinear controllers

  1997cited by this paper
• Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation

  1997cited by this paper
• Ideals, varieties, and algorithms - an introduction to computational algebraic geometry and commutative algebra (2. ed.)

  1997cited by this paper
• Nonlinear and adaptive control de-sign

  1995cited by this paper
• The LMI control toolbox

  1994cited by this paper
• Dynamic Programming

  1993cited by this paper
• Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra

  1992cited by this paper
• Universal Approximation Using Radial-Basis-Function Networks

  1991cited by this paper
• Multilayer feedforward networks are universal approximators

  1989cited by this paper
• Extending H-infinity Control to Nonlinear Systems: Control of Nonlinear Systems to Achieve Performance Objectives

  1987cited by this paper
• Generalized polynomial approximations in Markovian decision processes

  1985cited by this paper
• Robust Adaptive Control

  1984cited by this paper
• An Approximation Theory of Optimal Control for Trainable Manipulators

  1979cited by this paper
• On the Observability of Polynomial Systems, I: Finite-Time Problems

  1979cited by this paper
• Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences

  1974cited by this paper
• FUNCTIONAL APPROXIMATIONS AND DYNAMIC PROGRAMMING

  1959influential reference
• IDEALS

  1932cited by this paper
• and as an in

  year unknowncited by this paper

• Reinforcement Learning for Control with Probabilistic Stability Guarantee: A Finite-Sample Approach

  2026cites this paper
• Optimal Output–Feedback Tracker Design of Linear Quadratic Tracking Problem Using High-Order Filter and Incremental Data Adaptive Dynamic Programming

  2026cites this paper
• Adaptive optimal control for nonlinear networked systems under encoding-decoding mechanism: A generalized policy iteration approach

  2026cites this paper
• Data-based constrained H∞ optimal control for unknown nonlinear systems via adaptive dynamic programming

  2025cites this paper
• Learning optimal safety certificates for unknown nonlinear control systems

  2025cites this paper
• Traffic prediction-based long-term energy management approach incorporating engine transient control for HEVs

  2025cites this paper
• Adaptive Critic Control With Knowledge Transfer for Uncertain Nonlinear Dynamical Systems: A Reinforcement Learning Approach

  2025cites this paper
• Primal-dual adaptive dynamic programming for finite-horizon optimal control of nonlinear systems with isoperimetric constraints

  2025cites this paper
• Nonlinear Model Predictive Control With Value Function Regression: A Tessellation‐Based Sampling Approach

  2025cites this paper
• Learning finite-horizon optimal control with unknown control-affine dynamics

  2025cites this paper
• Model-Free Inverse Optimal Control for Completely Unknown Nonlinear Systems by Adaptive Dynamic Programming

  2025influential citation
• Output Feedback-Based Adaptive Optimal Output Regulation for Continuous-Time Strict-Feedback Nonlinear Systems

  2025cites this paper
• Bounding the Error of Value Functions in Sobolev Norm Yields Bounds on Suboptimality of Controller Performance

  2025cites this paper
• Fault-Tolerant Optimized Control of Switched Complex Networks via an Adaptive Dynamic Programming Approach

  2025cites this paper
• Policy Iteration of Nonlinear Discrete-time Systems with Input Time-Delay

  2025cites this paper
• Sum of Squares Polynomial Approximate Dynamic Programming for the Ergodic Problem

  2025cites this paper
• Guaranteed Time Control using Linear Matrix Inequalities

  2025influential citation
• Prescribed Performance Optimal Consensus of MASs With Connectivity Preservation

  2025cites this paper
• Feedback Stabilization of Polynomial Systems: From Model-based to Data-driven Methods

  2025cites this paper
• Self-triggered adaptive dynamic programming based on experience-replay and spectral adaptive law.

  2025cites this paper
• Online Adaptive Optimal Control Algorithm Based on Weighted Policy Iteration

  2025cites this paper
• A New Semidefinite Relaxation for Linear and Piecewise-Affine Optimal Control with Time Scaling

  2025cites this paper
• Data‐Based H∞${H_\infty }$ Optimal Tracking Control of Completely Unknown Linear Systems Under Input Constraints

  2025cites this paper
• Optimal Control of Flotation Reagents Via Sos and Reinforcement Learning

  2025cites this paper
• Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

  2024cites this paper
• Data-Driven Actuator Allocation for Actuator Redundant Systems

  2024cites this paper
• An Effective PQ-Decoupling Control Scheme Using Adaptive Dynamic Programming Approach to Reducing Oscillations of Virtual Synchronous Generators for Grid Connection With Different Impedance Types

  2024cites this paper
• Adaptive Optimal Output-Feedback Control of Discrete-Time Systems based on Hybrid Iteration

  2024cites this paper
• A model-free learning approach for active damping of a rotating cylindrical shell using ADP control with rotating-frame state reconstruction technique

  2024cites this paper
• A Novel Framework for Game-Based Optimal Event-Triggered Control of Multi-Input Nonlinear Systems

  2024cites this paper
• Bridging the Gap Between Reinforcement Learning and Nonlinear Output-Feedback Control

  2024cites this paper
• Experience replay based online adaptive robust tracking control for partially unknown nonlinear systems with asymmetric constrained‐input

  2024influential citation
• Online Q-learning for stochastic linear systems with state and control dependent noise

  2024cites this paper
• Reinforcement learning-based optimal control of uncertain nonlinear systems

  2024cites this paper
• Probabilistic Adaptive Dynamic Programming for Optimal Reliability-Critical Control With Fault Interruption Estimation

  2024influential citation
• Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis

  2024cites this paper
• Synergetic Learning Neuro-Control for Unknown Affine Nonlinear Systems With Asymptotic Stability Guarantees

  2024cites this paper
• Optimal Adaptive Control of Linear Stochastic Systems With Quadratic Cost Function

  2024cites this paper
• Dynamic Programming Algorithm using Furniture Industry 0/1 Knapsack Problem

  2023cites this paper
• Observer-based event-triggered recursive optimal tracking control for a class of strict-feedback nonlinear systems.

  2023cites this paper
• A Lyapunov characterization of robust policy optimization

  2023cites this paper
• Indirect Adaptive Optimal Control in the Presence of Input Saturation*

  2023cites this paper
• Modeling and Control of a New Spherical Robot with Cable Transmission

  2023cites this paper
• An Actor-Critic Framework for Online Control With Environment Stability Guarantee

  2023cites this paper
• Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

  2023cites this paper
• Suboptimal Controller Synthesis for Cart-Poles and Quadrotors via Sums-of-Squares

  2023cites this paper
• Optimal consensus control for double‐integrator multiagent systems with unknown dynamics using adaptive dynamic programming

  2023influential citation
• Adaptive control for stochastic nonlinear systems with time‐varying delays via multidimensional Taylor network

  2023cites this paper
• Stochastic Optimal Linear Control for Generalized Cost Functions With Time-Invariant Stochastic Parameters

  2023cites this paper
• Event-Triggered Output Feedback Control for a Class of Nonlinear Systems via Disturbance Observer and Adaptive Dynamic Programming

  2023cites this paper
• Dynamic Event-Triggered Reinforcement Learning Control of Stochastic Nonlinear Systems

  2023cites this paper
• Approximate Optimal Controller Synthesis for Cart-Poles and Quadrotors via Sums-of-Squares

  2023cites this paper
• Data-Driven LQR using Reinforcement Learning and Quadratic Neural Networks

  2023cites this paper
• Adaptive Optimal Stabilization Regulation of Continuous-Time Systems via Virtual System and Measurement Feedback

  2023cites this paper
• Derivative Feedback Control Using Reinforcement Learning

  2023cites this paper
• Distributed Lebesgue Approximation Model for Distributed Continuous-Time Nonlinear Systems

  2023cites this paper
• Online solving Nash equilibrium solution of N-player nonzero-sum differential games via recursive least squares

  2023cites this paper
• Reinforcement Learning for Mean-Field Linear Quadratic Control with Partial System Information

  2023cites this paper
• Event-triggered-based adaptive dynamic programming for distributed formation control of multi-UAV

  2022cites this paper
• A gradient-based algorithm for non-smooth constrained optimization problems governed by discrete-time nonlinear equations with application to long-term hydrothermal optimal scheduling control

  2022cites this paper
• The intelligent critic framework for advanced optimal control

  2022cites this paper
• Interference compensation discrete control based on memory data for a class of nonlinear systems

  2022cites this paper
• Safety-critical Policy Iteration Algorithm for Control under Model Uncertainty

  2022cites this paper
• SOS-based policy iteration for H∞ control of polynomial systems with uncertain parameters

  2022influential citation
• Event‐trigger‐based approximate optimal control of modular robot manipulators using zero‐sum game

  2022cites this paper
• A framework for online, stabilizing reinforcement learning

  2022cites this paper
• Adaptive, Optimal, Virtual Synchronous Generator Control of Three-Phase Grid-Connected Inverters Under Different Grid Conditions—An Adaptive Dynamic Programming Approach

  2022cites this paper
• Bias-policy iteration based adaptive dynamic programming for unknown continuous-time linear systems

  2022cites this paper
• Data-Driven Suboptimal Control for Nonlinear Systems Using State-Dependent Riccati Equation

  2022cites this paper
• Nonlinear H∞ path following control for autonomous ground vehicles via neural network and policy iteration algorithm

  2022cites this paper
• State-Feedback Design for Nonlinear Saturating Systems

  2022cites this paper
• Dynamic Event-Triggered Consensus Control for Multi-Agent Systems Using Adaptive Dynamic Programming

  2022cites this paper
• Constrained Event-Triggered H∞ Control Based on Adaptive Dynamic Programming With Concurrent Learning

  2022cites this paper
• Model-free event-triggered optimal control with performance guarantees via goal representation heuristic dynamic programming

  2022influential citation
• Adaptive Optimal Control of Networked Nonlinear Systems With Stochastic Sensor and Actuator Dropouts Based on Reinforcement Learning

  2022cites this paper
• Optimal output regulation for unknown continuous-time linear systems by internal model and adaptive dynamic programming

  2022cites this paper
• Probabilistic Framework of Howard's Policy Iteration: BML Evaluation and Robust Convergence Analysis

  2022cites this paper
• H∞ optimal control of unknown linear systems by adaptive dynamic programming with applications to time‐delay systems

  2021cites this paper
• Real-Time Energy-Efficient Actuation of Induction Motor Drives Using Approximate Dynamic Programming

  2021cites this paper
• A Neural Network Approach for Real-Time High-Dimensional Optimal Control

  2021cites this paper
• Learning-based control for discrete-time constrained nonzero-sum games

  2021cites this paper
• A tractor-trailer parking control scheme using adaptive dynamic programming

  2021cites this paper
• Model-free reinforcement learning approach to optimal speed control of combustion engines in start-up mode

  2021cites this paper
• Model-Free Composite Control for a Class of Semi-Coupled Two-Time-Scale Systems: An Adaptive Dynamic Programming Approach

  2021cites this paper
• Secure Networked Control Systems Design Using Semi-homomorphic Encryption

  2021cites this paper
• A Neural Network Approach for High-Dimensional Optimal Control Applied to Multiagent Path Finding

  2021cites this paper
• Sum‐of‐squares‐based policy iteration for suboptimal control of polynomial time‐varying systems

  2021influential citation
• H∞ Codesign for Uncertain Nonlinear Control Systems Based on Policy Iteration Method

  2021cites this paper
• Policy-Iteration-Based Learning for Nonlinear Player Game Systems With Constrained Inputs

  2021cites this paper
• Hamiltonian‐driven adaptive dynamic programming for mixed H2/H∞ performance using sum‐of‐squares

  2021influential citation
• Adaptive dynamic programming enhanced admittance control for robots with environment interaction and actuator saturation

  2021cites this paper
• Adaptive Dynamic Programming for Control: A Survey and Recent Advances

  2021cites this paper
• Data-based optimal tracking of autonomous nonlinear switching systems

  2021cites this paper
• Value iteration with deep neural networks for optimal control of input-affine nonlinear systems

  2021cites this paper
• An Adaptive Dynamic Programming Scheme for Nonlinear Optimal Control With Unknown Dynamics and Its Application to Turbofan Engines

  2021cites this paper
• Event-triggered optimal control for nonlinear stochastic systems via adaptive dynamic programming

  2021cites this paper
• Tracking of stabilizing, optimal control in fixed-time based on time-varying objective function

  2021cites this paper
• Further Results on Optimal Tracking Control for Nonlinear Systems With Nonzero Equilibrium via Adaptive Dynamic Programming

  2021cites this paper
• Complete model-free sliding mode control (CMFSMC)

  2021cites this paper
• Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors

  2021cites this paper