Barret Zoph, Quoc V. Le
2016 · arXiv
Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
8 selected excerpts · 8 total mentions
Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables). 1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268
Cool, was thinking about this subject earlier today. Abstract: >This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent…
architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578 Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...
This question isn't silly! People have actually trained neural networks to generate other neural networks. The "designer" network is trained using reinforcement learning, with a reward signal that depends on the efficacy of the networks it creates. See: Neural Architecture Search with Reinforcement Learning (https://arxiv.org/abs/1611.01578) It looks like they trained the "designer" network by trainning 15,000 deep learning models (800 at a time in parallel over 800 GPUs)
Jeff Dean talks about AutoML using RL and in the paper " Neural Architecture Search with Reinforcement Learning" it also talks about this. Also it seems different from more traditional hyperparameter optimization because it makes novel cells. So the structure of the network isn't limited to our existing library of layers/cells. https://arxiv.org/abs/1611.01578 https://youtu.be/HcStlHGpjN8?t=2073
Yeah, I'm confused that this is the top comment; it's factually incorrect. NASNet is an example of a result of AutoML. To quote the Google blogpost on NASNet: >In Learning Transferable Architectures for Scalable Image Recognition, we apply AutoML to the ImageNet image classification and COCO object detection dataset... AutoML was able to find the best layers that work well on CIFAR-10 but work well on ImageNet classification and COCO object detection. These two layers are combined to form a novel architecture, which we called “NASNet”. [https://research.googleblog.com/2017/11/automl-for-large-sca..., November 2017] In contrast AutoML is, as the nytimes article describes,…
"a machine-learning algorithm that learns to build other machine-learning algorithms". More specifically, from the Google blogpost about AutoML: >In our approach (which we call "AutoML"), a controller neural net can propose a “child” model architecture, which can then be trained and evaluated for quality on a particular task...Eventually the controller learns to assign high probability to areas of architecture space that achieve better accuracy on a held-out validation dataset, and low probability to areas of architecture space that score poorly. [https://research.googleblog.com/2017/05/using-machine-learni..., May 2017] Quoc, Barret, and others have been working on ANN-architecture-design systems for a while now (see: https://arxiv.org/abs/1611.01578), and AutoML specifically was done before announcing NASNet. Saying that NASNet is "the actual research behind AutoML" is drawing the causal arrow backwards.
There has been some work related to what you are describing: Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474 Related (but less so), there are also some papers about learning neural network architectures: Designing Neural Network Architectures using Reinforcement Learning https://arxiv.org/abs/1611.02167 Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578
Deep learning architectures built by machines (so we no longer have to design architecture to solve problems) https://arxiv.org/abs/1611.01578 Transfer Learning (so we need less data to build models) http://ftp.cs.wisc.edu/machine-learning/shavlik-group/torrey... Generative adversarial networks (so computers can get human like abilities at generating content) https://papers.nips.cc/paper/5423-generative-adversarial-net...
Other equally exciting papers that relates to learning to learn in DL. "Neural Architecture Search with Reinforcement Learning" https://arxiv.org/abs/1611.01578 "RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning" https://arxiv.org/abs/1611.02779 "Designing Neural Network Architectures using Reinforcement Learning" https://arxiv.org/abs/1611.02167
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.