Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devices

We study the sequential decision-making problem for automated weaning of mechanical circulatory support (MCS) devices in cardiogenic shock patients. MCS devices are percutaneous micro-axial flow pumps that provide left ventricular unloading and forward blood flow, but current weaning strategies vary significantly across care teams and lack data-driven approaches. Offline reinforcement learning (RL) has proven to be successful in sequential decision-making tasks, but our setting presents challenges for training and evaluating traditional offline RL methods: prohibition of online patient interaction, highly uncertain circulatory dynamics due to concurrent treatments, and limited data availability. We developed an end-to-end machine learning framework with two key contributions (1) Clinically-aware OOD-regularized Model-based Policy Optimization (CORMPO), a density-regularized offline RL algorithm for out-of-distribution suppression that also incorporates clinically-informed reward shaping and (2) a Transformer-based probabilistic digital twin that models MCS circulatory dynamics for policy evaluation with rich physiological and clinical metrics. We prove that \textsf{CORMPO} achieves theoretical performance guarantees under mild assumptions. CORMPO attains a higher reward than the offline RL baselines by 28% and higher scores in clinical metrics by 82.6% on real and synthetic datasets. Our approach offers a principled framework for safe offline policy learning in high-stakes medical applications where domain expertise and safety constraints are essential.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-11-08

• Fields of study

  Medicine, Computer Science, Engineering

• Identifiers
  DOI 10.48550/arXiv.2511.06111arXiv 2511.06111
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies

  2025cited by this paper
• FUSE: Fast Unified Simulation and Estimation for PDEs

  2024cited by this paper
• Data-Driven Simulator for Mechanical Circulatory Support with Domain Adversarial Neural Process

  2024cited by this paper
• Med-Real2Sim: Non-Invasive Medical Digital Twins using Physics-Informed Self-Supervised Learning

  2024cited by this paper
• Policy Regularization with Dataset Constraint for Offline Reinforcement Learning

  2023cited by this paper
• Constrained Policy Optimization with Explicit Behavior Density For Offline Reinforcement Learning

  2023cited by this paper
• Supported Value Regularization for Offline Reinforcement Learning

  2023cited by this paper
• A Comprehensive Review of Mechanical Circulatory Support Devices.

  2022cited by this paper
• Towards Safe Reinforcement Learning with a Safety Editor Policy

  2022cited by this paper
• Forecasting Aortic Pressure Cross-Cohort with Deep Sequence Models

  2022cited by this paper
• Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning

  2022cited by this paper
• Supported Policy Optimization for Offline Reinforcement Learning

  2022cited by this paper
• Reinforcement Learning for Precision Oncology

  2021cited by this paper
• RoFormer: Enhanced Transformer with Rotary Position Embedding

  2021cited by this paper
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization

  2021cited by this paper
• MOReL : Model-Based Offline Reinforcement Learning

  2020cited by this paper
• Mortality in patients with cardiogenic shock treated with the Impella CP microaxial pump for isolated left ventricular failure

  2020cited by this paper
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

  2019cited by this paper
• When to Trust Your Model: Model-Based Policy Optimization

  2019cited by this paper
• Development and internal validation of a new rating method of intra-operative haemodynamic instability

  2019cited by this paper
• Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

  2018cited by this paper
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

  2018cited by this paper
• The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care

  2018cited by this paper
• Billion-Scale Similarity Search with GPUs

  2017cited by this paper
• Deep reinforcement learning for automated radiation adaptation in lung cancer

  2017cited by this paper
• A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units

  2017cited by this paper
• Attention is All you Need

  2017cited by this paper
• Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach

  2017cited by this paper
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

  2016cited by this paper
• State-Space Models

  2013cited by this paper
• Strictly Proper Scoring Rules, Prediction, and Estimation

  2007cited by this paper
• Cardiogenic Shock

  1971cited by this paper

• No citing papers are available for this paper.