Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models

Hierarchical Bayesian models based on Gaussian processes are considered useful for describing complex nonlinear statistical dependencies among variables in real-world data. However, effective Monte Carlo algorithms for inference with these models have not yet been established, except for several simple cases. In this study, we show that, compared with the slow inference achieved with existing program libraries, the performance of Riemannian manifold Hamiltonian Monte Carlo (RMHMC) can be drastically improved by applying the chain rule for the differentiation of the Hamiltonian in the optimal order determined by the model structure, and by dynamically programming the eigendecomposition of the Riemannian metric with the recursive update of the eigenvectors at the previous move. This improvement cannot be achieved when using a naive automatic differentiator included in commonly used libraries. We numerically demonstrate that RMHMC effectively samples from the posterior, allowing the calculation of model evidence, in a Bayesian logistic regression on simulated data and in the estimation of propensity functions for the American national medical expenditure data using several Bayesian multiple-kernel models. These results lay a foundation for implementing effective Monte Carlo algorithms for analysing real-world data with Gaussian processes, and highlight the need to develop a customisable library set that allows users to incorporate dynamically programmed objects and to finely optimise the mode of automatic differentiation depending on the model structure.

• Publication year

  2025

• Venue

  Mathematics

• Publication date

  2025-11-09

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.48550/arXiv.2511.06407arXiv 2511.06407
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Debiased Maximum Likelihood Estimators of Hazard Ratios Under Kernel-Based Machine Learning Adjustment

  2025cited by this paper
• Fully Bayesian inference for latent variable Gaussian process models

  2022cited by this paper
• Faster proximal algorithms for matrix optimization using Jacobi-based eigenvalue methods

  2021cited by this paper
• Posterior Contraction for Deep Gaussian Process Priors

  2021cited by this paper
• Scalable Fully Bayesian Gaussian Process Modeling and Calibration With Adaptive Sequential Monte Carlo for Industrial Applications

  2021cited by this paper
• Data Parallel C++: Mastering DPC++ for Programming of Heterogeneous Systems using C++ and SYCL

  2020cited by this paper
• An Introduction to Sequential Monte Carlo

  2020cited by this paper
• C++17 Standard Library Quick Reference: A Pocket Guide to Data Structures, Algorithms, and Functions

  2019cited by this paper
• Introducing an Explicit Symplectic Integration Scheme for Riemannian Manifold Hamiltonian Monte Carlo

  2019cited by this paper
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

  2019cited by this paper
• On sampling from a log-concave density using kinetic Langevin diffusions

  2018cited by this paper
• Underdamped Langevin MCMC: A non-asymptotic analysis

  2017cited by this paper
• Double/debiased machine learning for treatment and structural parameters

  2017cited by this paper
• Markov chain Monte Carlo with the Integrated Nested Laplace Approximation

  2017cited by this paper
• Improved Rectangular Matrix Multiplication using Powers of the Coppersmith-Winograd Tensor

  2017cited by this paper
• Causal inference with a continuous treatment and outcome: Alternative estimators for parametric dose-response functions with applications

  2016cited by this paper
• Bayesian Computing with INLA: A Review

  2016cited by this paper
• GPflow: A Gaussian Process Library using TensorFlow

  2016cited by this paper
• Computationally Efficient Bayesian Learning of Gaussian Process State Space Models

  2015cited by this paper
• Bayesian Spatial Modelling with R-INLA

  2015cited by this paper
• MCMC for Variationally Sparse Gaussian Processes

  2015cited by this paper
• Extending Integrated Nested Laplace Approximation to a Class of Near‐Gaussian Latent Models

  2014cited by this paper
• Hilbert space methods for reduced-rank Gaussian process regression

  2014influential reference
• A Bayesian information criterion for singular models

  2013cited by this paper
• Annealed Importance Sampling Reversible Jump MCMC Algorithms

  2013cited by this paper
• Bayesian Inference and Learning in Gaussian Process State-Space Models with Particle MCMC

  2013cited by this paper
• Generalizing the No-U-Turn Sampler to Riemannian Manifolds

  2013cited by this paper
• Deep Gaussian Processes

  2012cited by this paper
• PAC-Bayesian Bound for Gaussian Process Regression and Multiple Kernel Additive Model

  2012cited by this paper
• A General Metric for Riemannian Manifold Hamiltonian Monte Carlo

  2012influential reference
• INLA or MCMC? A tutorial and comparative evaluation for spatial prediction in log-Gaussian Cox processes

  2012cited by this paper
• A widely applicable Bayesian information criterion

  2012cited by this paper
• Riemann manifold Langevin and Hamiltonian Monte Carlo methods

  2011cited by this paper
• Probabilistic Inference Using Markov Chain Monte Carlo Methods

  2011influential reference
• Information Rates of Nonparametric Gaussian Process Methods

  2011cited by this paper
• Matrix Computations

  2011cited by this paper
• The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo

  2011cited by this paper
• Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

  2009cited by this paper
• The BUGS project: Evolution, critique and future directions

  2009cited by this paper
• Estimating Bayes factors via thermodynamic integration and population MCMC

  2009cited by this paper
• Rates of contraction of posterior distributions based on Gaussian process priors

  2008cited by this paper
• Gaussian Process Dynamical Models

  2005cited by this paper
• Gaussian Processes for Machine Learning

  2005cited by this paper
• Feynman-Kac Formulae: Genealogical and Interacting Particle Systems with Applications

  2004cited by this paper
• Causal Inference With General Treatment Regimes

  2004cited by this paper
• Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data

  2003cited by this paper
• Disease cases and their medical costs attributable to smoking: an analysis of the national medical expenditure survey

  2003cited by this paper
• Kernel independent component analysis

  2003cited by this paper
• Trust Region Methods

  2000cited by this paper
• Reversible jump Markov chain Monte Carlo computation and Bayesian model determination

  1995cited by this paper
• On the limited memory BFGS method for large scale optimization

  1989cited by this paper
• Hybrid Monte Carlo

  1987influential reference
• A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines

  1970cited by this paper
• On the quadratic convergence of the special cyclic Jacobi method

  1966cited by this paper

• No citing papers are available for this paper.