Esteban Real, Chen Liang, David R. So, Quoc V. Le
2020 · arXiv
Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
"AutoML-Zero: Evolving Machine Learning Algorithms From Scratch" (2020) https://arxiv.org/abs/2003.03384 https://scholar.google.com/scholar?cluster=11748751662887361... How does this compare to MOSES (OpenCog/asmoses) or PLN? https://github.com/opencog/asmoses https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=%22... (2007)
I think you are underselling the potential of a model which deeply understand programming. Imagine combining such a model with something like AutoML-Zero: https://arxiv.org/abs/2003.03384 It may not be 'creative', but used as tab-completion, it's not being rewarded or incentivized or used in any way which would expose its abilities towards creating a new sort algorithm.
Here is a paper they referenced in the article: 2003) AutoML-Zero: Evolving Machine Learning Algorithms From Scratch - https://arxiv.org/abs/2003.03384. Although I could not finish reading all of it yet, I think here algorithms mutating is another dimension which can mutate, which then effects of course new topologies, weights, and other factors.
The paper [1] cites Koza among a total of 102 citations. "An early example of a symbolically discovered optimizer is that of Bengio et al. [8], who represent F as a tree: the leaves are the possible inputs to the optimizer (i.e. the xi above) and the nodes are one of {+, −, ×, ÷}. F is then evolved, making this an example of genetic programming [36]. Our search method is similar to genetic programming but we choose to represent the program as a sequence of instructions—like a programmer would type it—rather than a tree. " "[36]" is "Koza, J. R. and Koza, J. R. Genetic programming: on the programming of computers by means of natural selection. MIT press, 1992." [1] https://arxiv.org/pdf/2003.03384.pdf
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.