Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach

Sepsis is a leading cause of mortality in intensive care units (ICUs) and costs hospitals billions annually. Treating a septic patient is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. Understanding more about a patient's physiological state at a given time could hold the key to effective treatment policies. In this work, we propose a new approach to deduce optimal treatment policies for septic patients by using continuous state-space models and deep reinforcement learning. Learning treatment policies over continuous spaces is important, because we retain more of the patient's physiological information. Our model is able to learn clinically interpretable treatment policies, similar in important aspects to the treatment policies of physicians. Evaluating our algorithm on past ICU patient data, we find that our model could reduce patient mortality in the hospital by up to 3.6% over observed clinical policies, from a baseline mortality of 13.7%. The learned treatment policies could be used to aid intensive care clinicians in medical decision making and improve the likelihood of patient survival.

• Publication year

  2017

• Venue

  Machine Learning in Health Care

• Publication date

  2017-05-23

• Fields of study

  Medicine, Computer Science, Mathematics

• Identifiers
  arXiv 1705.08422
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Introduction to Reinforcement Learning

  2020cited by this paper
• Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016

  2017cited by this paper
• Fluid administration in severe sepsis and septic shock, patterns and outcomes: an analysis of a large national database

  2017cited by this paper
• A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units

  2017cited by this paper
• Mastering the game of Go with deep neural networks and tree search

  2016cited by this paper
• MIMIC-III, a freely accessible critical care database

  2016cited by this paper
• Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach

  2016cited by this paper
• Prioritized Experience Replay

  2015cited by this paper
• Dueling Network Architectures for Deep Reinforcement Learning

  2015influential reference
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

  2015influential reference
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Doubly Robust Off-policy Evaluation for Reinforcement Learning

  2015influential reference
• Deep Reinforcement Learning with Double Q-Learning

  2015cited by this paper
• Sepsis: a roadmap for future research.

  2015cited by this paper
• Population-Level Prediction of Type 2 Diabetes From Claims Data and Analysis of Risk Factors

  2015cited by this paper
• The demise of early goal‐directed therapy for severe sepsis and septic shock

  2015cited by this paper
• Blood Pressure Targets For Vasopressor Therapy: A Systematic Review

  2015cited by this paper
• The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)

  2015influential reference
• Human-level control through deep reinforcement learning

  2015influential reference
• Unfolding physiological state: mortality modelling in intensive care units

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Disease-based Modeling to Predict Fluid Response in Intensive Care Units

  2013cited by this paper
• Prognostic Physiology: Modeling Patient Severity in Intensive Care Units Using Radial Domain Folding

  2012cited by this paper
• Informing sequential clinical decision-making through reinforcement learning: an empirical study

  2010cited by this paper
• Inverse Reinforcement Learning

  2010cited by this paper
• ICU Acuity: Real-time Models versus Daily Models

  2009cited by this paper
• Learning Deep Architectures for AI

  2007cited by this paper
• Sepsis in European intensive care units: Results of the SOAP study*

  2006cited by this paper
• Vasopressors for shock.

  2004cited by this paper
• On-line Q-learning using connectionist systems

  1994influential reference

• Clinician-informed offline reinforcement learning for vasopressor administration optimization in shock management

  2026cites this paper
• medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functions

  2026influential citation
• A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies

  2026cites this paper
• World Model Enhanced Offline Reinforcement Learning for Sequential Intervention Optimization in Acute Kidney Injury

  2026cites this paper
• Toward Robust Agents: A Survey of Adversarial Attacks and Defenses in Deep Reinforcement Learning

  2026cites this paper
• Causally-Aware Safe Reinforcement Learning for Long-Horizon Partially Observable Environments in IoT Systems

  2025cites this paper
• NutriSighT: Interpretable Transformer Model for Dynamic Prediction of Hypocaloric Enteral Nutrition in Mechanically Ventilated Patients

  2025cites this paper
• THEMES: An Offline Apprenticeship Learning Framework for Evolving Reward Functions

  2025cites this paper
• Dynamic reward-augmented ensemble learning for EEG signal classification in major depressive disorder

  2025cites this paper
• Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devices

  2025cites this paper
• Dual-channel batch constrained deep Q-learning for sepsis treatment

  2025cites this paper
• Enhance differential privacy mechanisms for clinical data analysis using CNNs and reinforcement learning

  2025cites this paper
• Off-Policy Evaluation Under Nonignorable Missing Data

  2025cites this paper
• NutriSighT: Interpretable Transformer Model for Dynamic Prediction of Underfeeding Enteral Nutrition in Mechanically Ventilated Patients

  2025cites this paper
• Dynamic Risk Stratification in ICU Sepsis: A Deep Reinforcement Learning Approach Using Synthetic Data

  2025cites this paper
• Methods in dynamic treatment regimens using observational healthcare data: A systematic review

  2025cites this paper
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI

  2025cites this paper
• Smart Imitator: Learning from Imperfect Clinical Decisions

  2025cites this paper
• Treatment, evidence, imitation, and chat

  2025cites this paper
• SAFER: A Calibrated Risk-Aware Multimodal Recommendation Model for Dynamic Treatment Regimes

  2025cites this paper
• Optimizing Sepsis Treatment Through Reinforcement Learning: A Revisitation of Reward and Loss Functions in Dueling Double Deep Q-Network

  2024cites this paper
• Medical Knowledge Integration Into Reinforcement Learning Algorithms for Dynamic Treatment Regimes

  2024cites this paper
• An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning

  2024cites this paper
• Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning

  2024cites this paper
• A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features

  2024cites this paper
• Reinforced Sequential Decision-Making for Sepsis Treatment: The PosNegDM Framework With Mortality Classifier and Transformer

  2024cites this paper
• MiranDa: Mimicking the Learning Processes of Human Doctors to Achieve Causal Inference for Medication Recommendation

  2024cites this paper
• ADT²R: Adaptive Decision Transformer for Dynamic Treatment Regimes in Sepsis

  2024influential citation
• Artificial Intelligence-based Decision Support Systems for Precision and Digital Health

  2024cites this paper
• Corticosteroid sensitivity detection in sepsis patients using a personalized data mining approach: A clinical investigation

  2024cites this paper
• Guideline-informed reinforcement learning for mechanical ventilation in critical care

  2024cites this paper
• Optimizing sepsis treatment strategies via a reinforcement learning model

  2024cites this paper
• A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs

  2024cites this paper
• A drug mix and dose decision algorithm for individualized type 2 diabetes management

  2024cites this paper
• Conservative Evaluation of Offline Policy Learning

  2024cites this paper
• Scalable Personalised Treatment Recommendation in Critical Care

  2024cites this paper
• PrescDRL: deep reinforcement learning for herbal prescription planning in treatment of chronic diseases

  2024cites this paper
• Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare

  2024influential citation
• Learning Personalized Health Recommendations via Offline Reinforcement Learning

  2024cites this paper
• Machine learning approaches for the discovery of clinical pathways from patient data: A systematic review

  2024cites this paper
• AgentMV: A Deep Reinforcement Learning Model For Mechanical Ventilation

  2024cites this paper
• A PopPBPK-RL approach for precision dosing of benazepril in renal impaired patients

  2024cites this paper
• Measurement Scheduling for ICU Patients with Offline Reinforcement Learning

  2024cites this paper
• How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models

  2024cites this paper
• Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning

  2024cites this paper
• Reinforcement Learning Model for Managing Noninvasive Ventilation Switching Policy

  2023cites this paper
• An Offline Time-aware Apprenticeship Learning Framework for Evolving Reward Functions

  2023cites this paper
• Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

  2023cites this paper
• A optimization framework for herbal prescription planning based on deep reinforcement learning

  2023cites this paper
• Cross-Domain Policy Adaptation via Value-Guided Data Filtering

  2023cites this paper
• Kernel Density Bayesian Inverse Reinforcement Learning

  2023cites this paper
• Towards Real-World Applications of Personalized Anesthesia Using Policy Constraint Q Learning for Propofol Infusion Control

  2023cites this paper
• Privacy-Preserving Deep Reinforcement Learning Based On Adaptive Noise For Sepsis Treatment

  2023cites this paper
• Continuous-Time Decision Transformer for Healthcare Applications

  2023cites this paper
• Natural Intelligence as the Brain of Intelligent Systems

  2023influential citation
• Interpretable Skill Learning for Dynamic Treatment Regimes through Imitation

  2023cites this paper
• A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis

  2023influential citation
• Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications

  2023influential citation
• Kernel-Based Distributed Q-Learning: A Scalable Reinforcement Learning Approach for Dynamic Treatment Regimes

  2023cites this paper
• Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

  2023cites this paper
• Causal inference using observational intensive care unit data: a scoping review and recommendations for future practice

  2023cites this paper
• Resource Allocation Reinforcement Learning for Quality of Service Maintenance in Cloud-Based Services

  2023cites this paper
• Analysis of Reward Formulation Based on Mean Arterial Pressure in Reinforcement Learning for Critically Ill Septic Patient

  2023cites this paper
• Towards Safe Propofol Dosing during General Anesthesia Using Deep Offline Reinforcement Learning

  2023cites this paper
• Data-driven decision support for individualised cardiovascular resuscitation in sepsis: a scoping review and primer for clinicians

  2023cites this paper
• Effects of Varying $k$ in $L_{k}$ Norm Similarity Measures on a Reinforcement Learning-Based Clinical Decision Support System

  2023cites this paper
• Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

  2023cites this paper
• Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery

  2023cites this paper
• Reinforcement learning in large, structured action spaces: A simulation study of decision support for spinal cord injury rehabilitation

  2023cites this paper
• Reinforcement Learning for Intelligent Healthcare Systems: A Review of Challenges, Applications, and Open Research Issues

  2023cites this paper
• Bias in Reinforcement Learning: A Review in Healthcare Applications

  2023cites this paper
• Deep Attention Q-Network for Personalized Treatment Recommendation

  2023influential citation
• Value function assessment to different RL algorithms for heparin treatment policy of patients with sepsis in ICU

  2023cites this paper
• Demystifying Complex Treatment Recommendations: A Hierarchical Cooperative Multi-Agent RL Approach

  2023influential citation
• Towards more efficient and robust evaluation of sepsis treatment with deep reinforcement learning

  2023influential citation
• Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care

  2023cites this paper
• Reward poisoning attacks in deep reinforcement learning based on exploration strategies

  2023cites this paper
• Decentralized Multi-Robot Navigation in Unknown Environments via Hierarchical Deep Reinforcement Learning

  2023cites this paper
• Artificial intelligence in perioperative medicine: a narrative review

  2022cites this paper
• Establishment and Implementation of Potential Fluid Therapy Balance Strategies for ICU Sepsis Patients Based on Reinforcement Learning

  2022cites this paper
• Semi-Markov Offline Reinforcement Learning for Healthcare

  2022cites this paper
• Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions

  2022cites this paper
• Adaptive Semi-Supervised Inference for Optimal Treatment Decisions with Electronic Medical Record Data

  2022cites this paper
• Personalization of Mechanical Ventilation Treatment using Deep Conservative Reinforcement Learning

  2022cites this paper
• Guiding Efficient, Effective, and Patient-Oriented Electrolyte Replacement in Critical Care: An Artificial Intelligence Reinforcement Learning Approach

  2022cites this paper
• Hierarchical Imitation Learning via Subgoal Representation Learning for Dynamic Treatment Recommendation

  2022influential citation
• Critic-over-Actor-Critic Modeling: Finding Optimal Strategy in ICU Environments

  2022cites this paper
• A Deep Reinforcement Computation Model for Sepsis Treatment

  2022cites this paper
• Relative sparsity for medical decision problems

  2022influential citation
• Near-Optimal Differentially Private Reinforcement Learning

  2022cites this paper
• Safe Reinforcement Learning for Sepsis Treatment

  2022cites this paper
• Conditional generation of medical time series for extrapolation to underrepresented populations

  2022cites this paper
• Optimal Treatment Regimes for Proximal Causal Learning

  2022cites this paper
• A Framework for Safe Evaluation of Offline Learning

  2022cites this paper
• The treatment of sepsis: an episodic memory-assisted deep reinforcement learning approach

  2022influential citation
• Electronic health records based reinforcement learning for treatment optimizing

  2022cites this paper
• Federated Offline Reinforcement Learning

  2022cites this paper
• Optimizing Substance Use Treatment Selection Using Reinforcement Learning

  2022influential citation
• Artificial Intelligence: A Review of Progress and Prospects in Medicine and Healthcare

  2022cites this paper
• Predicting clinical outcomes using artificial intelligence and machine learning in neonatal intensive care units: a systematic review

  2022cites this paper