Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le
2017 · arXiv
Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, NASNet achieves 2.4% error rate, which is state-of-the-art. On ImageNet, NASNet achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
We have an example of AdaNet on GitHub that uses several complex CNN layers (conv2d, pooling, linearly-separable layers) as part of the NASNet-A architecture (presented in https://arxiv.org/abs/1707.07012): https://github.com/tensorflow/adanet/blob/master/adanet/exam... AdaNet supports high level APIs (e.g tf.layers, tf.losses), so it can support RNN cells as well.
Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables). 1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268
This is not "AI that builds AI". The actual research behind AutoML is called NASNet (https://arxiv.org/pdf/1707.07012.pdf), and all it is simply: we found two good neural network layers (called NASNet normal cells / reduction cells in the paper) that work well on many different image datasets. It's a very cool research result. But it's not something that will replace AI researchers.
Practical example, neural architecture search to discover convnets that require less computation, https://arxiv.org/abs/1707.07012 "Learning Transferable Architectures for Scalable Image Recognition"
This is good background but modern architecture are significantly better: Learning Transferable Architectures for Scalable Image Recognition https://arxiv.org/abs/1707.07012
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.