Large-Scale Heteroscedastic Regression via Gaussian Process

Heteroscedastic regression considering the varying noises among observations has many applications in the fields, such as machine learning and statistics. Here, we focus on the heteroscedastic Gaussian process (HGP) regression that integrates the latent function and the noise function in a unified nonparametric Bayesian framework. Though showing remarkable performance, HGP suffers from the cubic time complexity, which strictly limits its application to big data. To improve the scalability, we first develop a variational sparse inference algorithm, named VSHGP, to handle large-scale data sets. Furthermore, two variants are developed to improve the scalability and capability of VSHGP. The first is stochastic VSHGP (SVSHGP) that derives a factorized evidence lower bound, thus enhancing efficient stochastic variational inference. The second is distributed VSHGP (DVSHGP) that follows the Bayesian committee machine formalism to distribute computations over multiple local VSHGP experts with many inducing points and adopts hybrid parameters for experts to guard against overfitting and capture local variety. The superiority of DVSHGP and SVSHGP compared to the existing scalable HGP/homoscedastic GP is then extensively verified on various data sets.

• Publication year

  2018

• Venue

  IEEE Transactions on Neural Networks and Learning Systems

• Publication date

  2018-11-03

• Fields of study

  Mathematics, Computer Science, Medicine

• Identifiers
  DOI 10.1109/TNNLS.2020.2979188arXiv 1811.01179PMID 32275610
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Sparse Gaussian Process Regression Beyond Variational Inference

  2019cited by this paper
• Parametric Gaussian Process Regressors

  2019cited by this paper
• Information Directed Sampling and Bandits with Heteroscedastic Noise

  2018cited by this paper
• Understanding and Comparing Scalable Gaussian Process Regression for Big Data

  2018cited by this paper
• Gaussian Process Conditional Density Estimation

  2018cited by this paper
• A Two-Layer Mixture Model of Gaussian Process Functional Regressions and Its MCMC EM Algorithm

  2018cited by this paper
• When Gaussian Process Meets Big Data: A Review of Scalable GPs

  2018cited by this paper
• Variational Implicit Processes

  2018cited by this paper
• Generalized Robust Bayesian Committee Machine for Large-scale Gaussian Process Regression

  2018cited by this paper
• Remarks on multi-output Gaussian process regression

  2018cited by this paper
• Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models

  2018cited by this paper
• Heteroscedastic Gaussian Process-based System Identification and Predictive Control of a Quadcopter

  2018cited by this paper
• Stochastic Variational Inference for Bayesian Sparse Gaussian Process Regression

  2017cited by this paper
• Non-Stationary Spectral Kernels

  2017cited by this paper
• Laplace approximation and the natural gradient for Gaussian process regression with the heteroscedastic Student-t model

  2017cited by this paper
• A probabilistic data-driven model for planar pushing

  2017cited by this paper
• Stochastic Variational Inference for Fully Bayesian Sparse Gaussian Process Regression Models

  2017cited by this paper
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

  2017cited by this paper
• Gaussian process decentralized data fusion meets transfer learning in large-scale distributed cooperative perception

  2017cited by this paper
• Asynchronous Distributed Variational Gaussian Process for Regression

  2017cited by this paper
• Laplace Approximation for Divisive Gaussian Processes for Nonstationary Regression

  2016cited by this paper
• TensorFlow: A system for large-scale machine learning

  2016cited by this paper
• Chained Gaussian Processes

  2016cited by this paper
• Understanding Probabilistic Sparse Gaussian Process Approximations

  2016cited by this paper
• Taking the Human Out of the Loop: A Review of Bayesian Optimization

  2016cited by this paper
• A Distributed Variational Inference Framework for Unifying Parallel Sparse Gaussian Process Regression Models

  2016cited by this paper
• Variational Fourier Features for Gaussian Processes

  2016cited by this paper
• GPflow: A Gaussian Process Library using TensorFlow

  2016influential reference
• GPz: non-stationary sparse Gaussian processes for heteroscedastic uncertainty estimation in photometric redshifts

  2016influential reference
• Practical Heteroscedastic Gaussian Process Modeling for Large Simulation Experiments

  2016cited by this paper
• Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)

  2015cited by this paper
• Distributed Gaussian Processes

  2015influential reference
• Sensor Calibration and Hysteresis Compensation With Heteroscedastic Gaussian Processes

  2015cited by this paper
• Gaussian Process Random Fields

  2015cited by this paper
• Variational Inference for Heteroscedastic Semiparametric Regression

  2015cited by this paper
• Probabilistic electricity price forecasting with variational heteroscedastic Gaussian process and active learning

  2015cited by this paper
• Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo

  2015cited by this paper
• A Unifying Framework of Anytime Sparse Gaussian Process Regression Models with Stochastic Variational Inference for Big Data

  2015cited by this paper
• Tree-structured Gaussian Process Approximations

  2014cited by this paper
• Retrieval of Biophysical Parameters With Heteroscedastic Gaussian Processes

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014influential reference
• A Metamodel for Estimating Error Bounds in Real-Time Traffic Prediction Systems

  2014cited by this paper
• Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

  2014cited by this paper
• Fast Allocation of Gaussian Process Experts

  2014cited by this paper
• Nested Variational Compression in Deep Gaussian Processes

  2014cited by this paper
• Divisive Gaussian Processes for Nonstationary Regression

  2014cited by this paper
• Expectation propagation for nonstationary heteroscedastic Gaussian process regression

  2014cited by this paper
• Gaussian Processes for Big Data

  2013influential reference
• Building Accurate 3D Spatial Networks to Enable Next Generation Intelligent Transportation Systems

  2013cited by this paper
• A review of deterministic approximate inference techniques for Bayesian machine learning

  2013cited by this paper
• Active learning

  2013cited by this paper
• A framework for evaluating approximation methods for Gaussian process regression

  2012cited by this paper
• Twenty Years of Mixture of Experts

  2012cited by this paper
• Gaussian Process Regression with Heteroscedastic or Non-Gaussian Residuals

  2012cited by this paper
• Heteroscedastic Gaussian process regression using expectation propagation

  2011cited by this paper
• Variational Heteroscedastic Gaussian Process Regression

  2011influential reference
• Gaussian Processes for Machine Learning (GPML) Toolbox

  2010cited by this paper
• Parallel Geostatistics for Sparse and Dense Datasets

  2010cited by this paper
• Variational Learning of Inducing Variables in Sparse Gaussian Processes

  2009cited by this paper
• Kernel Conditional Quantile Estimation via Reduction Revisited

  2009cited by this paper
• Bagging for Gaussian process regression

  2009cited by this paper
• Modelling local and global phenomena with sparse Gaussian processes

  2008cited by this paper
• Local and global sparse Gaussian process approximations

  2007influential reference
• Most likely heteroscedastic Gaussian process regression

  2007cited by this paper
• Variable Noise and Dimensionality Reduction for Sparse Gaussian processes

  2006cited by this paper
• A Unifying View of Sparse Approximate Gaussian Process Regression

  2005cited by this paper
• Healing the relevance vector machine through augmentation

  2005cited by this paper
• Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models

  2005cited by this paper
• Sparse Gaussian Processes using Pseudo-inputs

  2005cited by this paper
• Training Products of Experts by Minimizing Contrastive Divergence

  2002cited by this paper
• Infinite Mixtures of Gaussian Process Experts

  2001cited by this paper
• Sparse Bayesian Learning and the Relevan e Ve tor Ma hine

  2001cited by this paper
• Before the

  2000cited by this paper
• A Bayesian Committee Machine

  2000cited by this paper
• Regression with Input-dependent Noise: A Gaussian Process Treatment

  1997influential reference

• Accurate monocular structured light system calibration based on improved grey wolf optimizer and Gaussian process regression

  2026cites this paper
• Robust and Scalable Multi-Robot Localization Using Stereo UWB Arrays

  2025cites this paper
• Vecchia approximated Bayesian heteroskedastic Gaussian processes

  2025influential citation
• Hierarchical Gaussian processes for characterizing gait variability in multiple sclerosis

  2025influential citation
• Challenges and Opportunities for Statistics in the Era of Data Science

  2025cites this paper
• CATOA: Cooperative Calibration of Timestamp Measurements for Distributed Multi-Robot Localization

  2024cites this paper
• Gaussian process fusion method for multi-fidelity data with heterogeneity distribution in aerospace vehicle flight dynamics

  2024cites this paper
• Accurate Robotic Pushing Manipulation Through Online Model Estimation Under Uncertain Object Properties

  2024cites this paper
• Predicting product quality in continuous manufacturing processes using a scalable robust Gaussian Process approach

  2024cites this paper
• Spatiotemporal Transformation-Based Neural Network With Interpretable Structure for Modeling Distributed Parameter Systems

  2024cites this paper
• An Uncertainty-Aware, Mesh-Free Numerical Method for Kolmogorov PDEs

  2024cites this paper
• Robust parameter design for 3D printing process using stochastic computer model

  2024cites this paper
• Nested Heteroscedastic Gaussian Process for Simulation Metamodeling

  2024influential citation
• Multi-fidelity Bayesian Optimization with Multiple Information Sources of Input-dependent Fidelity

  2024cites this paper
• Heteroscedastic Bayesian optimization using generalized product of experts

  2023cites this paper
• Noise-Aware Sparse Gaussian Processes and Application to Reliable Industrial Machinery Health Monitoring

  2023cites this paper
• Sparsity-Aware Distributed Learning for Gaussian Processes With Linear Multiple Kernel

  2023cites this paper
• Regression with Cost-based Rejection

  2023cites this paper
• Identifying Drivers of Predictive Aleatoric Uncertainty

  2023cites this paper
• Bayesian Optimisation for Robust Model Predictive Control under Model Parameter Uncertainty

  2022cites this paper
• Variational Hierarchical Mixtures for Learning Probabilistic Inverse Dynamics

  2022cites this paper
• Variational Hierarchical Mixtures for Probabilistic Learning of Inverse Dynamics

  2022cites this paper
• A two-layer aggregation model with effective consistency for large-scale Gaussian process regression

  2021cites this paper
• Sequential Bayesian experimental design for estimation of extreme-event probability in stochastic input-to-response systems

  2021cites this paper
• Modulating Scalable Gaussian Processes for Expressive Statistical Learning

  2020cites this paper
• A Variational Infinite Mixture for Probabilistic Inverse Dynamics Learning

  2020cites this paper
• Probabilistic modelling of wind turbine power curves with application of heteroscedastic Gaussian Process regression

  2020influential citation
• A Dual Metamodeling Perspective for Design and Analysis of Stochastic Simulation Experiments

  2019cites this paper
• Distributed Variational Inference-Based Heteroscedastic Gaussian Process Metamodeling

  2019cites this paper
• Achieving robustness to aleatoric uncertainty with heteroscedastic Bayesian optimisation

  2019influential citation
• When Gaussian Process Meets Big Data: A Review of Scalable GPs

  2018cites this paper