SCIENCE@home

Practical Bayesian Optimization of Machine Learning Algorithms

Jasper Snoek,H. Larochelle,Ryan P. Adams

Published 2012 in Neural Information Processing Systems

ABSTRACT

The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.

PUBLICATION RECORD

  • Publication year

    2012

  • Venue

    Neural Information Processing Systems

  • Publication date

    2012-06-13

  • Fields of study

    Mathematics, Computer Science

  • Identifiers
    arXiv 1206.2944
  • External record

    Open on Semantic Scholar

  • Source metadata

    Semantic Scholar

CITATION MAP

EXTRACTION MAP

CLAIMS

CONCEPTS

REFERENCES

Showing 1-22 of 22 references · Page 1 of 1

CITED BY

Showing 1-100 of 9021 citing papers · Page 1 of 91

123491
Next