Jasper Snoek, Oren Rippel, Kevin Swersky
Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, Ryan P. Adams
2015 · arXiv
Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization. In this work, we explore the use of neural networks as an alternative to GPs to model distributions over functions. We show that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we apply to large scale hyperparameter optimization, rapidly finding competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
While this is mostly true there is has been some research to use non-GP models for Bayesian Optimization or the general Sequential Model Based Optimization (SMBO) problem. Some examples: * SMAC (using Random forests) - [1] * Using Deep Neural Networks - [2] * Tree Structured Parzen Estimators - [3] [1] https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf [2] https://arxiv.org/pdf/1502.05700.pdf [3] https://papers.nips.cc/paper/4443-algorithms-for-hyper-param...
It has been done already: http://arxiv.org/abs/1502.05700 Edit: Although what they seem to describe is replacing GPs with neural networks in Bayesian optimization which is supposedly more efficient. Since the point of Bayesian optimization is to limit the number of times you have to evaluate a new set of hyperparameters, I am not sure how useful it is to "be able to scale" (i.e. even if maintaining the GP is O(n^3) with the number of evaluations, the costly part should be to evaluate the hyperparameters in the first place) but I haven't read the paper so they may show some high dimensional hyperparameter cases where performing a lot of evaluations pays off.
That may be. And I don't think kernels will ever be a hot topic. But part of that is that they are a very old topic (you can see Gauss referring to them in slide 22 of http://math.iit.edu/~fass/590/notes/Notes590_Ch1.pdf). I think that kinda prevents funding agencies from pushing too hard for research into them, since there is a more likely quick return on investment into a new topic. And I don't begrudge them that - funding should be given to people with new ideas which have the potential to revolutionize things. I'd also mention that there are people trying to bridge the gap between how GPs work and the benefits of neural nets (http://arxiv.org/abs/1502.05700). Probably…
another thing holding them back is their high barrier to entry - you can't do much with the theory of kernels unless you know statistics and functional analysis (and of course numerical analysis and linear algebra,) which is one of the reasons I had difficulty finding research students. To draw a parallel to numerical analysis (where I am much more comfortable than machine learning,) this is why finite element methods are much more popular than, say, boundary element methods for PDEs, despite the fact that boundary element methods are better in many circumstances. But when implementing things, especially in industry, simplicity and robustness count for something and those are not strong points of kernels. But kernels haven't gone anywhere yet, and they've been fundamental to analysis since at least David Hilbert, so they are not going anywhere any time soon. Deep learning ... we'll see. Maybe it has the legs to stick around or maybe it'll be swept aside by the next hot thing. 8 years ago all the research funding was going into uncertainty quantification, 5 years ago compressed sensing was the hot idea (thanks Terry Tao,) and now deep learning is going to light the world on fire. All of those are, and should be, hotter topics than kernels, but we'll see what we're talking about in 5 years. It won't be kernels (unless something crazy happens and I become the leader of the US, in which case I have a great idea for a new reality show) but I don't know that it will be deep learning either.
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.