Research on LLRF Feedback Control Algorithm Based on Reinforcement Learning

This study proposes a low-level radio frequency (LLRF) feedback control algorithm based on reinforcement learning (RL) using the soft actor–critic (SAC) and proximal policy optimization (PPO) algorithms. These approaches can automatically learn optimal control strategies for multi-input–multi-output (MIMO) systems, thereby reducing interference among variables during closed-loop control and enhancing the overall control precision of the system. To avoid interruption to accelerators when training the RL models, we modeled the relationship between LLRF operator input and klystron output to enable offline training, significantly accelerating the training. Trained RL models were validated at the FELiChEM facility. Experimental results demonstrated that the LLRF control algorithm based on RL shows improvements in both amplitude and phase closed-loop accuracy compared to the traditional univariate proportional–integral (PI) controller.

• Publication year

  2025

• Venue

  IEEE Transactions on Nuclear Science

• Publication date

  2025-10-01

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1109/TNS.2025.3604518
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Development of a digital low-level RF system for the STCF

  2024cited by this paper
• Learning to Do or Learning While Doing: Reinforcement Learning and Bayesian Optimisation for Online Continuous Tuning

  2023cited by this paper
• Commissioning of the next-generation LLRF control system for the Rapid Cycling Synchrotron of the Japan Proton Accelerator Research Complex

  2021cited by this paper
• Reinforcement Learning Algorithms: Analysis and Applications

  2021cited by this paper
• Policy gradient methods for free-electron laser and terahertz source optimization and stabilization at the FERMI free-electron laser at Elettra

  2020cited by this paper
• Basic Reinforcement Learning Techniques to Control the Intensity of a Seeded Free-Electron Laser

  2020cited by this paper
• A review of PID control, tuning methods and applications

  2020cited by this paper
• SciPy 1.0: fundamental algorithms for scientific computing in Python

  2019cited by this paper
• Feedback Design for Control of the Micro-Bunching Instability based on Reinforcement Learning

  2019cited by this paper
• Application of the asynchronous advantage actor–critic machine learning algorithm to real-time accelerator tuning

  2019cited by this paper
• Soft Actor-Critic Algorithms and Applications

  2018cited by this paper
• Hindsight Experience Replay

  2017cited by this paper
• Proximal Policy Optimization Algorithms

  2017cited by this paper
• MIMO PID tuning via iterated LMI restriction

  2016cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Multi-Layer Perceptrons

  2013cited by this paper
• Digital prototype of LLRF system for SSRF

  2008cited by this paper
• LOW-LEVEL RF SYSTEMS FOR SYNCHROTRONS Part I : the low-intensity case

  2005cited by this paper
• Random Forests

  2001cited by this paper
• Low-level RF systems for synchrotrons. Part II: High intensity. Compensation of beam-induced effects

  2000cited by this paper
• Ieee Transactions on Systems, Man and Cybernetics—part C: Applications and Reviews 1 a Survey of Actor-critic Reinforcement Learning: Standard and Natural Policy Gradients

  year unknowninfluential reference

• No citing papers are available for this paper.