Skip to content
The methodOpen data
Back to the collection

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar

2016 · arXiv

AlgorithmsStatistics & probabilityMachine learning
Read the paper
HN score
0
Mentions
4
Readers
3
Months
4
What’s in a score?

From the abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

abhgh
Neutral mention · rule
You are not doing away with the test/validation set. You are doing away with: 1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1] 2. you might still do grid search but you want to preferentially allocate resources to parameter exploration based on how promising they seem. Hyperband - [2]- is an example. 3. you can do both - you can be smart about picking parameters, but for the ones you pick you can preferentially allocate resources. Ex *Bayesian Optimization and Hyperband* (BOHB) [1] Bayesian Optimization using Gaussian…
Read the full comment
Processes is an example of this. https://arxiv.org/pdf/1206.2944.pdf Here's a library that helps you do this: https://github.com/JasperSnoek/spearmint. But there are other techniques in this family like Randomized Online Aggressive Racing (ROAR), Deep Network for Global Optimization (DNGO), Tree-Structured Parzen Estimators (TPE), etc [2] Hyperband: https://arxiv.org/abs/1603.06560 [3] BOHB: https://arxiv.org/abs/1807.01774
Original comment Troubling Trends in Machine Learning Scholarship
kirillseva
Neutral mention · rule
the tweet you link to is architectural search, or figuring out what the neural network should look like (along with its parameters). If self-driving car ML is "data scientists replacing a driver with computers", architecture search is computers replacing data scientists. $1.2M is roughly 3-4 data scientists' annual compensations at a company like Google or Uber. Finding hyperparameters for an existing architecture is much easier because you would typically deal with a much lower cardinality in hyperparameter space. SigOpt is helping with that. Google has an internal solution that is able to take advantage of parameters within their job scheduling infrastructure (they…
Read the full comment
actually have a bunch of them, the one I'm talking about is called hyperband [0]). CERN, as far as I know, was using hyperopt [1]. Avant uses an open-source version of SigOpt that we wrote in 2016 called loop [2]. [0] - https://arxiv.org/pdf/1603.06560.pdf [1] - http://hyperopt.github.io/hyperopt [2] - https://github.com/avantoss/loop
Original comment Uber and Waymo Reach Settlement
kirillseva
Neutral mention · rule
This is a fairly well represented technique in bayesian hyperparameter optimization, where you train a meta-classifier that keeps track of the parameter space. Kind of like a manager model, if you will, that learns to intelligently optimize exploration vs exploitation so that a team of workers will arrive at the global optimum. Back when Yahoo! was a real company they used a technique called "multi-armed bandits" to learn what ads to show. [1] More recently, there's a number of off-the-shelf packages available that you can trivially integrate into your ML pipeline to optimize hyperparameters of your models, I'll include the links below. [1] multi-armed bandits…
Read the full comment
https://www.theregister.co.uk/2011/09/23/yahoo_core_personal... [2] tree of parzen estimators https://jaberg.github.io/hyperopt/ [3] hyperband - what google uses in their internal ML toolkits AFAIK https://arxiv.org/pdf/1603.06560.pdf [4] (shameless plug) gaussian process based hyperparameter optimization service https://github.com/avantoss/loop
Original comment Population-based training of neural networks

Keep following the thread.

More in Algorithms

More from the same topics.