Freeze-Thaw Bayesian Optimization

In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings. Our method uses the partial information gained during the training of a machine learning model in order to decide whether to pause training and start a new model, or resume the training of a previously-considered model. We specifically tailor our method to machine learning problems by developing a novel positive-definite covariance kernel to capture a variety of training curves. Furthermore, we develop a Gaussian process prior that scales gracefully with additional temporal observations. Finally, we provide an information-theoretic framework to automate the decision process. Experiments on several common machine learning models show that our approach is extremely effective in practice.

• Publication year

  2014

• Venue

  arXiv.org

• Publication date

  2014-06-15

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1406.3896
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Bayesian Optimization with Unknown Constraints

  2014cited by this paper
• Input Warping for Bayesian Optimization of Non-Stationary Functions

  2014cited by this paper
• Multi-Task Bayesian Optimization

  2013cited by this paper
• Practical Bayesian Optimization of Machine Learning Algorithms

  2012cited by this paper
• Improving neural networks by preventing co-adaptation of feature detectors

  2012influential reference
• Random Search for Hyper-Parameter Optimization

  2012cited by this paper
• Exponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations

  2012cited by this paper
• Entropy Search for Information-Efficient Global Optimization

  2011cited by this paper
• A reliable effective terascale linear learning system

  2011cited by this paper
• Convergence Rates of Efficient Global Optimization Algorithms

  2011cited by this paper
• Sequential Model-Based Optimization for General Algorithm Configuration

  2011cited by this paper
• Algorithms for Hyper-Parameter Optimization

  2011cited by this paper
• Portfolio Allocation for Bayesian Optimization

  2010cited by this paper
• Online Learning for Latent Dirichlet Allocation

  2010cited by this paper
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

  2010cited by this paper
• Slice sampling covariance hyperparameters of latent Gaussian models

  2010cited by this paper
• Bayesian optimization for sensor set selection

  2010cited by this paper
• Handling Sparsity via the Horseshoe

  2009cited by this paper
• Practical bayesian optimization

  2008cited by this paper
• Gaussian Processes for Global Optimization

  2008cited by this paper
• Probabilistic Matrix Factorization

  2007cited by this paper
• Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces

  2004cited by this paper
• A Taxonomy of Global Optimization Methods Based on Response Surfaces

  2001cited by this paper
• An algorithmic framework for performing collaborative filtering

  1999cited by this paper
• Application of Bayesian approach to numerical methods of global and stochastic optimization

  1994cited by this paper

• Neural Neural Scaling Laws

  2026cites this paper
• Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps

  2026cites this paper
• Data Complexity-aware Deep Model Performance Forecasting

  2026cites this paper
• Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization

  2026cites this paper
• Empirical Gaussian Processes

  2026cites this paper
• Scaling with Collapse: Efficient and Predictable Training of LLM Families

  2025cites this paper
• You Only Train Once

  2025cites this paper
• Information-theoretic Bayesian Optimization: Survey and Tutorial

  2025cites this paper
• Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework

  2025cites this paper
• Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning

  2025influential citation
• Tune My Adam, Please!

  2025cites this paper
• Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks

  2025cites this paper
• LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought

  2025cites this paper
• Model-Free-Communication Federated Neuroevolution

  2025cites this paper
• MARIO: A Superadditive Multi-Algorithm Interworking Optimization Framework for Analog Circuit Sizing

  2025cites this paper
• ATOM: An Automatic Topology Synthesis Framework for Operational Amplifiers

  2025cites this paper
• Optimizing Sports Predictions Using Model Selection: A Case Study in Reproducible Data-Science

  2025cites this paper
• Zero-Shot Performance Prediction for Probabilistic Scaling Laws

  2025cites this paper
• When to retrain a machine learning model

  2025cites this paper
• A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs

  2024cites this paper
• Efficient Hyperparameter Optimization with Adaptive Fidelity Identification

  2024cites this paper
• Meta-learning from learning curves for budget-limited algorithm selection

  2024cites this paper
• NeuroLGP-SM: Scalable Surrogate-Assisted N euroevolution for Deep Neural Networks

  2024cites this paper
• Automated customization of large-scale spiking network models to neuronal population activity

  2024cites this paper
• Evaluating Learning Potential with Internal States in Deep Neural Networks

  2024influential citation
• FastTuning: Enabling Fast and Efficient Hyper-Parameter Tuning With Partitioning and Parallelism of Search Space

  2024cites this paper
• Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization

  2024cites this paper
• NEvoFed: A Decentralized Approach to Federated NeuroEvolution of Heterogeneous Neural Networks

  2024cites this paper
• FastBO: Fast HPO and NAS with Adaptive Fidelity Identification

  2024cites this paper
• Fast Model Selection and Hyperparameter Tuning for Generative Models

  2024cites this paper
• Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection

  2024cites this paper
• Aioli: A Unified Optimization Framework for Language Model Data Mixing

  2024cites this paper
• In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization

  2024influential citation
• Optimizing contextual bandit hyperparameters: A dynamic transfer learning-based framework

  2024cites this paper
• Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

  2024cites this paper
• NeuroLGP-SM: A Surrogate-assisted Neuroevolution Approach using Linear Genetic Programming

  2024cites this paper
• Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equation

  2024cites this paper
• BBGP-sDFO: Batch Bayesian and Gaussian Process Enhanced Subspace Derivative Free Optimization for High-Dimensional Analog Circuit Synthesis

  2024cites this paper
• Strong convexity-guided hyper-parameter optimization for flatter losses

  2024cites this paper
• Beyond Trend Following: Deep Learning for Market Trend Prediction

  2024cites this paper
• Early Stopping Bayesian Optimization for Controller Tuning

  2024cites this paper
• Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolation

  2024influential citation
• Automated customization of large-scale spiking network models to neuronal population activity

  2023cites this paper
• Hyper-parameter Tuning for Adversarially Robust Models

  2023cites this paper
• Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single

  2023cites this paper
• FedHPO-Bench: A Benchmark Suite for Federated Hyperparameter Optimization

  2023cites this paper
• Can we infer the presence of Differential Privacy in Deep Learning models' weights? Towards more secure Deep Learning

  2023cites this paper
• Single-shot General Hyper-parameter Optimization for Federated Learning

  2023cites this paper
• Multi-Objective Surrogate Modeling Through Transfer Learning for Telescopic Boom Forklift

  2023cites this paper
• Practitioner Motives to Select Hyperparameter Optimization Methods

  2023cites this paper
• Using Pipeline Performance Prediction to Accelerate AutoML Systems

  2023cites this paper
• HyperSTAR: Task-Aware Hyperparameter Recommendation for Training and Compression

  2023cites this paper
• PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

  2023cites this paper
• Genetic algorithm-based hyperparameter optimization of deep learning models for PM_2.5 time-series prediction

  2023cites this paper
• Optimizing Hyperparameters with Conformal Quantile Regression

  2023cites this paper
• MASIF: Meta-learned Algorithm Selection using Implicit Fidelity Information

  2023cites this paper
• Exploring the Advancements and Challenges of Automated Machine Learning

  2023cites this paper
• Pruning during training by network efficacy modeling

  2023cites this paper
• Efficient and Robust Bayesian Selection of Hyperparameters in Dimension Reduction for Visualization

  2023cites this paper
• An improved hyperparameter optimization framework for AutoML systems using evolutionary algorithms

  2023cites this paper
• Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks

  2023influential citation
• Learning Reliable Neural Networks with Distributed Architecture Representations

  2023cites this paper
• Gray-Box Gaussian Processes for Automated Reinforcement Learning

  2023cites this paper
• A stopping criterion for Bayesian optimization by the gap of expected minimum simple regrets

  2023cites this paper
• Learning to Rank Normalized Entropy Curves with Differentiable Window Transformation

  2023cites this paper
• Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round

  2022cites this paper
• Amortized Proximal Optimization

  2022cites this paper
• Meta-learning from Learning Curves: Challenge Design and Baseline Results

  2022cites this paper
• Bayesian Optimization-Based Beam Alignment for MmWave MIMO Communication Systems

  2022cites this paper
• Hyperparameter Optimization Using Iterative Decision Tree (IDT)

  2022cites this paper
• Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach

  2022influential citation
• Cello: Efficient Computer Systems Optimization with Predictive Early Termination and Censored Regression

  2022cites this paper
• IGWO-SS: Improved Grey Wolf Optimization Based on Synaptic Saliency for Fast Neural Architecture Search in Computer Vision

  2022cites this paper
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning

  2022cites this paper
• Syne Tune: A Library for Large Scale Hyperparameter Tuning and Reproducible Research

  2022cites this paper
• Practitioner Motives to Use Different Hyperparameter Optimization Methods

  2022cites this paper
• Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

  2022cites this paper
• Learning adaptive hyper-guidance via proxy-based bilevel optimization for image enhancement

  2022cites this paper
• TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning

  2022cites this paper
• Neural Architecture Search for Energy Efficient Always-on Audio Models

  2022cites this paper
• Supervising the Multi-Fidelity Race of Hyperparameter Configurations

  2022cites this paper
• Analog Circuit Yield Optimization via Freeze–Thaw Bayesian Optimization Technique

  2022influential citation
• Comparative Research of Hyper-Parameters Mathematical Optimization Algorithms for Automatic Machine Learning in New Generation Mobile Network

  2022cites this paper
• Learning curves for decision making in supervised machine learning: a survey

  2022influential citation
• Building High-throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine

  2022cites this paper
• Neural architecture search for energy-efficient always-on audio machine learning

  2022cites this paper
• PriorBand: HyperBand + Human Expert Knowledge

  2022cites this paper
• FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization

  2022cites this paper
• Neural Architecture Search via Proxy Validation

  2022cites this paper
• GloMPO (Globally Managed Parallel Optimization): a tool for expensive, black-box optimizations, application to ReaxFF reparameterizations

  2022cites this paper
• Investigation of Hyperparametry Methods and Kits for Deep Investigation of Hyperparametry Methods and Kits for Deep Neural Networks Neural Networks

  2021cites this paper
• Cost-Efficient Online Hyperparameter Optimization

  2021cites this paper
• Hyper-Parameter Tuning using Bayesian Optimization

  2021cites this paper
• Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

  2021cites this paper
• Faster & More Reliable Tuning of Neural Networks: Bayesian Optimization with Importance Sampling

  2021cites this paper
• Genealogical Population-Based Training for Hyperparameter Optimization

  2021cites this paper
• Improved Penalty Method via Doubly Stochastic Gradients for Bilevel Hyperparameter Optimization

  2021cites this paper
• Faster Improvement Rate Population Based Training

  2021cites this paper
• Classification of Alzheimer's Disease Using Gaussian-Based Bayesian Parameter Optimization for Deep Convolutional LSTM Network

  2021cites this paper
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

  2021cites this paper