Orthogonalized Policy Optimization:Policy Optimization as Orthogonal Projection in Hilbert Space

We propose Orthogonalized Policy Optimization (OPO), a principled framework for large language model alignment derived from optimization in the Hilbert function space L2(pi_k). Lifting policy updates from the probability simplex into L2(pi_k) transforms the nonlinear normalization constraint into a linear orthogonality condition_{pi_k} = 0 on the density fluctuation field v = pi/pi_k - 1. By the Hilbert projection theorem, the unique closed-form update is v_star = (omega_alpha - E[omega_alpha]) / mu, where the subtracted mean acts as a chemical potential enforcing probability conservation. This interpretation reveals advantage z-score normalization as a conservation-law projection rather than a variance-reduction heuristic. OPO cleanly decouples sampling geometry, controlled by the escort exponent alpha, from optimization geometry, governed by the stiffness parameter mu, a separation not attainable under KL-based objectives. The same update can also be derived as a Euclidean mirror-descent step and as the linear-response law of near-equilibrium statistical mechanics, establishing its structural uniqueness within ratio geometry. Structurally, OPO induces constant curvature, non-saturating linear gradient dynamics, and an intrinsic chi-square trust region. Experiments on MATH benchmarks show that the Hilbert projection formulation prevents gradient saturation typical of KL-constrained methods. By sustaining non-vanishing gradients in high-confidence regimes, OPO avoids premature plateaus and achieves stronger long-horizon training rewards and improved out-of-distribution generalization compared to clipping-based baselines.

• Publication year

  2026

• Venue

  Unknown venue

• Publication date

  2026-01-18

• Fields of study

  Computer Science

• Identifiers
  arXiv 2601.12415
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• APO: Alpha-Divergence Preference Optimization

  2025cited by this paper
• ADPO: Anchored Direct Preference Optimization

  2025cited by this paper
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale

  2025cited by this paper
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

  2024cited by this paper
• SimPO: Simple Preference Optimization with a Reference-Free Reward

  2024cited by this paper
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

  2023cited by this paper
• A General Theoretical Paradigm to Understand Learning from Human Preferences

  2023cited by this paper
• Improving proximal policy optimization with alpha divergence

  2023cited by this paper
• Training language models to follow instructions with human feedback

  2022cited by this paper
• A Divergence Minimization Perspective on Imitation Learning Methods

  2019cited by this paper
• Proximal Policy Optimization Algorithms

  2017cited by this paper
• Deep Reinforcement Learning from Human Preferences

  2017cited by this paper
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization

  2016cited by this paper
• Rényi Divergence Variational Inference

  2016cited by this paper
• Information geometry and its applications

  2012cited by this paper
• A General Class of Coefficients of Divergence of One Distribution from Another

  1966cited by this paper
• The fluctuation-dissipation theorem

  1966cited by this paper
• statistical-physics-part-1

  1958cited by this paper

• Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space

  2026influential citation