Automatic Construction and Natural-Language Description of Nonparametric Regression Models

This paper presents the beginnings of an automatic statistician, focusing on regression problems. Our system explores an open-ended space of statistical models to discover a good explanation of a data set, and then produces a detailed report with figures and natural-language text. Our approach treats unknown regression functions nonparametrically using Gaussian processes, which has two important consequences. First, Gaussian processes can model functions in terms of high-level properties (e.g. smoothness, trends, periodicity, changepoints). Taken together with the compositional structure of our language of models this allows us to automatically describe functions in simple terms. Second, the use of flexible nonparametric models and a rich language for composing them in an open-ended manner also results in state-of-the-art extrapolation performance evaluated over 13 real time series data sets from various domains.

• Publication year

  2014

• Venue

  AAAI Conference on Artificial Intelligence

• Publication date

  2014-02-18

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1609/aaai.v28i1.8904arXiv 1402.4304
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• GEFCom2012 hierarchical load forecasting: Gradient boosting machines and Gaussian processes

  2014cited by this paper
• A fully automatic problem solver with human-style output

  2013cited by this paper
• Gaussian Process Covariance Kernels for Pattern Discovery and Extrapolation

  2013cited by this paper
• Evolution of Covariance Functions for Gaussian Process Regression Using Genetic Programming

  2013cited by this paper
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search

  2013influential reference
• Nonparametric dynamics estimation for time periodic systems

  2013cited by this paper
• Multiresolution Gaussian Processes

  2012cited by this paper
• Exploiting compositionality to explore a large space of model structures

  2012cited by this paper
• Video In Sentences Out

  2012cited by this paper
• A GP-based kernel construction and optimization method for RVM

  2010cited by this paper
• Sparse Spectrum Gaussian Process Regression

  2010cited by this paper
• Sequential Bayesian Prediction in the Presence of Changepoints and Faults

  2010cited by this paper
• Gaussian Process Change Point Models

  2010cited by this paper
• Distilling Free-Form Natural Laws from Experimental Data

  2009cited by this paper
• The discovery of structural form

  2008cited by this paper
• Evolving kernel functions for SVMs by genetic programming

  2007cited by this paper
• Multiple kernel learning, conic duality, and the SMO algorithm

  2004cited by this paper
• Information Theory, Inference, and Learning Algorithms

  2004cited by this paper
• Discovering Admissible Model Equations from Observed Data Based on Scale-Types and Identity Constrains

  1999cited by this paper
• Declarative Bias in Equation Discovery

  1997cited by this paper
• Reconstruction of solar irradiance since 1610: Implications for climate change

  1995cited by this paper
• Basic Statistics for Business and Economics

  1982cited by this paper
• Occam's razor

  1980cited by this paper
• Estimating the Dimension of a Model

  1978influential reference
• Time series analysis, forecasting and control

  1972cited by this paper

• Amortized Spectral Kernel Discovery via Prior-Data Fitted Network

  2026cites this paper
• Automated Model Discovery via Multi-modal & Multi-step Pipeline

  2025influential citation
• A Practitioner's Guide to Automatic Kernel Search for Gaussian Processes in Battery Applications

  2025cites this paper
• Novel inverse predictive system integrated with industrial lubricant information

  2025cites this paper
• Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs

  2025cites this paper
• Forecasting Robust Gaussian Process State Space Models for Assessing Intervention Impact in Internet of Things Time Series

  2025cites this paper
• Automated Machine Learning for Failure Detection Model in Predictive Maintenance

  2025cites this paper
• Offline Changepoint Detection With Gaussian Processes

  2025cites this paper
• Just One Layer Norm Guarantees Stable Extrapolation

  2025cites this paper
• Generating Computational Cognitive Models using Large Language Models

  2025cites this paper
• Scalable Deep Basis Kernel Gaussian Processes

  2025cites this paper
• Efficient scalable deep kernels: unifying deep learning and nonparametric methods for large-scale data analysis

  2025cites this paper
• An Adaptive Hybrid Correlation Kriging Approach for Uncertainty Dynamic Optimization of Spherical-Conical Shell Structure

  2025cites this paper
• On the Laplace Approximation as Model Selection Criterion for Gaussian Processes

  2024influential citation
• Automated Statistical Model Discovery with Language Models

  2024cites this paper
• Industrial Data Modeling With Low-Dimensional Inputs and High-Dimensional Outputs

  2024cites this paper
• Exploratory Visual Analysis for Increasing Data Readiness in Artificial Intelligence Projects

  2024cites this paper
• Measuring Strength of Joint Causal Effects

  2024cites this paper
• Compressing spectral kernels in Gaussian Process: Enhanced generalization and interpretability

  2024cites this paper
• Discovering Structural Regularities in Time Series via Gaussian Processes

  2024cites this paper
• Patterns of time-interval based patterns for improved multivariate time series data classification

  2024cites this paper
• Predicting the elastic properties of Norway spruce by its morphology

  2024cites this paper
• Amortized Inference for Gaussian Process Hyperparameters of Structured Kernels

  2023cites this paper
• Sequential Monte Carlo Learning for Time Series Structure Discovery

  2023cites this paper
• Hierarchical-Hyperplane Kernels for Actively Learning Gaussian Process Models of Nonstationary Systems

  2023influential citation
• A Primer on Bayesian Neural Networks: Review and Debates

  2023cites this paper
• Improved automatic kernel construction for Gaussian process regression in small sample learning for predicting lift body aerodynamic performance

  2023cites this paper
• Ensemble learning of multi-kernel Kriging surrogate models using regional discrepancy and space-filling criteria-based hybrid sampling method

  2023cites this paper
• Identifying Quantifiably Verifiable Statements from Text

  2023influential citation
• UWM Digital Commons UWM Digital Commons

  2023cites this paper
• Beyond Intuition, a Framework for Applying GPs to Real-World Data

  2023cites this paper
• Understanding Insured Behavior through Causal Impact Analysis of IoT Streams

  2023cites this paper
• Eight years of AutoML: categorisation, review and trends

  2023influential citation
• Molecular Design of Fuels for Maximum Spark-Ignition Engine Efficiency by Combining Predictive Thermodynamics and Machine Learning

  2023cites this paper
• A Gaussian Process Decoder with Spectral Mixtures and a Locally Estimated Manifold for Data Visualization

  2023cites this paper
• The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning

  2023cites this paper
• On the Identifiability and Interpretability of Gaussian Process Models

  2023cites this paper
• Leveraging uncertainty estimates and derivative information in Gaussian process regression for efficient collection and use of molecular simulation data.

  2023cites this paper
• Learning material synthesis-process-structure-property relationship by data fusion: Bayesian Coregionalization N-Dimensional Piecewise Function Learning

  2023cites this paper
• Online model selection by learning how compositional kernels evolve

  2023cites this paper
• Gaussian Process Latent Variable Modeling for Few-Shot Time Series Forecasting

  2022cites this paper
• Uncertainty in equation learning

  2022cites this paper
• A Survey of Open Source Automation Tools for Data Science Predictions

  2022cites this paper
• Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes

  2022influential citation
• Trust in Language Grounding: a new AI challenge for human-robot teams

  2022cites this paper
• Automated Model Inference for Gaussian Processes: An Overview of State-of-the-Art Methods and Algorithms

  2022influential citation
• Adaptive Gaussian Process Change Point Detection

  2022cites this paper
• Bayesian Model Selection, the Marginal Likelihood, and Generalization

  2022cites this paper
• Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport

  2022cites this paper
• Online Time Series Anomaly Detection with State Space Gaussian Processes

  2022cites this paper
• Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships

  2022cites this paper
• Efficient prediction of turbulent flow quantities using a Bayesian hierarchical multifidelity model

  2022cites this paper
• A Change Dynamic Model for the Online Detection of Gradual Change

  2022cites this paper
• Online structural kernel selection for mobile health

  2021cites this paper
• Tractable Computation of Expected Kernels by Circuits

  2021cites this paper
• Priors in Bayesian Deep Learning: A Review

  2021cites this paper
• The Minecraft Kernel: Modelling correlated Gaussian Processes in the Fourier domain

  2021cites this paper
• Slow Momentum with Fast Reversion: A Trading Strategy Using Deep Learning and Changepoint Detection

  2021cites this paper
• Inference of dominant modes for linear stochastic processes

  2021cites this paper
• 3CS Algorithm for Efficient Gaussian Process Model Retrieval

  2021influential citation
• Interpretable Actions: Controlling Experts with Understandable Commands

  2021cites this paper
• Evolution of Gaussian Process kernels for machine translation post-editing effort estimation

  2021cites this paper
• Gaussian Process Latent Class Choice Models

  2021cites this paper
• Machine Learning for Cyber Physical Systems

  2021cites this paper
• Automating data science

  2021influential citation
• Combining Deductive and Statistical Explanations in the FRANK Query Answering System

  2021cites this paper
• Structure-Aware Random Fourier Kernel for Graphs

  2021cites this paper
• Evolving Gaussian process kernels from elementary mathematical expressions for time series extrapolation

  2021cites this paper
• GPy-ABCD: A Configurable Automatic Bayesian Covariance Discovery Implementation

  2021influential citation
• Automated Kernel Search for Gaussian Processes on Data Streams

  2021influential citation
• On machine learning and the replacement of human labour: anti-Cartesianism versus Babbage’s path

  2021cites this paper
• Model Selection for Production System via Automated Online Experiments

  2021cites this paper
• Latent-space time evolution of non-intrusive reduced-order models using Gaussian process emulation

  2020cites this paper
• Towards Large-scale Gaussian Process Models for Efficient Bayesian Machine Learning

  2020influential citation
• Compositional kernel learning using tree-based genetic programming for Gaussian process regression

  2020influential citation
• A Survey of Machine Learning Methods and Challenges for Windows Malware Classification

  2020cites this paper
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits

  2020cites this paper
• Towards Gaussian Processes for Automatic and Interpretable Anomaly Detection in Industry 4.0

  2020influential citation
• Linearly Constrained Gaussian Processes with Boundary Conditions

  2020influential citation
• Social Internet of Things: Opportunities and Challenges in Developing Countries

  2020cites this paper
• Large-scale Retrieval of Bayesian Machine Learning Models for Time Series Data via Gaussian Processes

  2020influential citation
• One-Shot Bayes Opt with Probabilistic Population Based Training

  2020cites this paper
• Marginalised Spectral Mixture Kernels with Nested Sampling

  2020cites this paper
• Scalable Approximate Bayesian Computation for Growing Network Models via Extrapolated and Sampled Summaries.

  2020cites this paper
• Accelerating Gaussian Process surrogate modeling using Compositional Kernel Learning and multi-stage sampling framework

  2020cites this paper
• Marginalised Gaussian Processes with Nested Sampling

  2020cites this paper
• Human-like Time Series Summaries via Trend Utility Estimation

  2020cites this paper
• Automatic Gaussian Process Model Retrieval for Big Data

  2020cites this paper
• Gaussian Process Panel Modeling—Machine Learning Inspired Analysis of Longitudinal Panel Data

  2020cites this paper
• Learning Compositional Sparse Gaussian Processes with a Shrinkage Prior

  2020influential citation
• Task-Agnostic Amortized Inference of Gaussian Process Hyperparameters

  2020cites this paper
• Forecasting monthly airline passenger numbers with small datasets using feature engineering and a modified principal component analysis

  2020cites this paper
• Model Construction with Support Vector Machines and Gaussian Processes through Kernel Search

  2019cites this paper
• Comments on: Data science, big data and statistics

  2019cites this paper
• Statistical inference for periodic and partially observable Poisson processes

  2019cites this paper
• Automating Population Health Studies Through Semantics and Statistics

  2019cites this paper
• DeepNotebooks: Deep Probabilistic Models Construct Python Notebooks for Reporting Datasets

  2019cites this paper
• Automating Active Learning for Gaussian Processes

  2019cites this paper
• Machine learning techniques for monitoring the sludge profile in a secondary settler tank

  2019cites this paper
• Deep Gaussian Processes with Importance-Weighted Variational Inference

  2019cites this paper