This paper pursues the insight that large language models (LLMs) trained to generate code can vastly improve the effectiveness of mutation operators applied to programs in genetic programming (GP). Because such LLMs benefit from training data that includes sequential changes and modifications, they can approximate likely changes that humans would make. To highlight the breadth of implications of such evolution through large models (ELM), in the main experiment ELM combined with MAP-Elites generates hundreds of thousands of functional examples of Python programs that output working ambulating robots in the Sodarace domain, which the original LLM had never seen in pre-training. These examples then help to bootstrap training a new conditional language model that can output the right walker for a particular terrain. The ability to bootstrap new models that can output appropriate artifacts for a given context in a domain where zero training data was previously available carries implications for open-endedness, deep learning, and reinforcement learning. These implications are explored here in depth in the hope of inspiring new directions of research now opened up by ELM.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Genetic programming systems have periodically made improvements to algorithms (dating back decades). Whether LLM-powered GP, which is effectively what this is, will be a step change or an evolution of that is still an open question I think. I'm also a little wary of reading too much into the recursive self-improvement idea, because "the GP system can use GP to improve the GP system itself!" is a very old idea that just has never worked, although I realize that isn't proof that it won't eventually work.
Some related work from a different company: https://sakana.ai/ai-cuda-engineer/
And some academic papers kind of in this space: https://arxiv.org/abs/2206.08896, https://arxiv.org/abs/2302.12170, https://arxiv.org/abs/2401.07102
I am sorry but that's nonsense.
I quoted the paper "Evolution through Large Models" written in collaboration between OpenAI and Anthropic researchers
"In other words, the model learns to predict plausible changes to code from examples of changes made to code by human programmers."
https://arxiv.org/pdf/2206.08896
> The idea that models can only write code if they've seen code that does the exact same thing in the past
How do you get "code that does the exact same thing" from "predicting plausible changes?"
The key ingredients are not the humans but the feedback they carry to the model. Humans are embodied and can test ideas in the real world, LLMs need some kind of special deployment to achieve that. It just so happens that chat rooms are such a deployment.
For example, AlphaZero started from scratch and only had feedback from the self-play game outcomes, but that was enough to reach superhuman level. It was the feedback that carried insights and taught the model.
You can make a parallel to the scientific method: you have two stages, ideation and validation. Ideation alone is not scientific. Validation is what makes or breaks ideas. LLMs without a validation system are just…
Read the full comment
like scientists without a lab.
We're not that smart, as demonstrated by the large number of ideas that don't pan out, we can churn ideas fast but we learn from their outcomes, we can't predict outcomes from the beginning and skip validation.
Here is an example of LLMs discovering useful ideas by feedback, even when they are completely outside their training distribution:
"Evolution through Large Models" https://arxiv.org/abs/2206.08896
This works because the task proposed by this paper is easy to test, so there is plenty of feedback. But the LLM still needs to apply ingenuity to optimize it, you can't brute force it by evolutionary methods alone.
It's overfitting.
Some people say they find LLMs very helpful for coding, some people say they are incredibly bad.
I often see people wondering if the some coding task is performed well or not because of availability of code examples in the training data. It's way worse than that. It's overfitting to diffs it was trained on.
"In other words, the model learns to predict plausible changes to code from examples of changes made to code by human programmers."
https://arxiv.org/abs/2206.08896
When you can run experiments quickly it becomes feasible to use ML and evolutionary methods to do novel discoveries, like AlphaTensor's better matrix multiplication than Strassen, and AlphaZero's move 37, upturning centuries of game strategy.
The paper "Evolution through Large Models" shows the way. Just use LLMs as genetic mutation operators. Evolutionary methods are great at search, LLMs are great at intuition but get stuck on their own, they combine well. https://arxiv.org/abs/2206.08896
The interplay between LLMs and Evolutionary Algorithms, despite differing in objectives and methodologies, share a common pursuit of applicability in complex problems. Meanwhile, EA…
Read the full comment
can provide an optimization framework for LLM's further enhancement under black box settings, empowering LLM with flexible global search capacities.
Since chatGPT was first released hundreds of millions of people have been using it for assistance, and the model outputs influenced their actions, maybe even supported scientists to make new discoveries. The LLM text is filtered through people and ends up as real world consequences and discoveries that are reported in text, and get in the next training set closing the loop.
Trillions of AI tokens per month do this slow feedback game. AI speeds up the circulation of useful information and ideas in human society, and AI feedback gets filtered by the contact with people and the real world.
No but some model serving tools like llama.cpp do their best. It's just a matter of choosing the right serving tools. And I am not sure LLMs could not optimize their memory layout. Why not? Just let them play with this and learn. You can do pretty amazing things with evolutionary methods where the LLMs are the mutation operator. You evolve a population of solutions. (https://arxiv.org/abs/2206.08896)
This is the "Anti Stochastic Parrot" paper. I am happy to finally read a philosophical paper on LLMs.
I just have a small quip.
> While LLMs show promise in various forms of task generalization, their participation in the ratcheting process of cultural learning thus appears contingent on further advancements in these areas, which might lie beyond the reach of current architectures.
I don't think cultural transmission with LLMs is that incipient.
Evolution through Large Models - https://arxiv.org/abs/2206.08896
We have seen papers where LLMs are used as "search heads" or "mutation operators" in evolutionary methods. They can rely on their language proficiency to explore…
Read the full comment
the most promising leads, spanning a vast combinatorial space.
And another example of "cultural transmission": GPT-4 has its paws all over Mistral and LLaMA finetunes. Plenty of cultural transmission between big and small LLMs. Not to mention Phi, a model trained purely on synthetic GPT-4 data. Synthetic data has been estimated to be 5x more efficient than human data, you can make a model 1/5 size with similar performance.
> A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way in an environment which is hostile and dynamic.
I agree. Autonomy is hard. But a weaker version of it is possible - self replication in software. An AI model can generate text that can train new models from scratch[1]. They can also generate the model code, explain it, make meaningful changes[2], monitor the training run and evaluate the "child" models[3].
So AI can "pull everything from inside" to make a child, no external materials needed, but any human generated or synthetic data can be used as well. AI is…
Read the full comment
capable of doing half the job of self replication, but can't make GPUs, and probably won't be able to do it autonomously for a long time. High end GPUs are so hard to make no company or even country controls the whole stack, it only works through global cooperation.
[1] TinyStories https://arxiv.org/abs/2305.07759
and Microsoft's Phi-1 using generated data
[2] Evolution through Large Models https://arxiv.org/abs/2206.08896
[3] G-Eval: NLG Evaluation using GPT 4 with Better Human Alignment https://arxiv.org/abs/2303.16634
LLMs can be used to propose candidates for chemical, medical, coding, and basically any domain where you got enough data to create a model but it is exponentially hard to search for solutions. They can be the core module in an agent that learns by RL or evolutionary methods - so the quality only depends on how much data the agent can generate. We know that when an agent can do proper exploration like AlphaGo Zero it can reach superhuman levels, it builds its own specialised dataset as it runs.
For example: Evolution through Large Models - https://arxiv.org/abs/2206.08896
Here is a paper doing evolutionary approaches on top of LLM generating code. It seems LLMs are remarkably good at learning from feedback.
> Evolution through Large Models
https://arxiv.org/abs/2206.08896
I think the source of creativity and knowledge is the world. Without access to the world we can't do science. LLMs are like brains in a jar, but when they get tools they can integrate external signals and actually become creative and even discover knowledge - like [1] and AlphaGo's move 37 (it has access to the go table, which is the "world" of Go)
But this "access to the world" needs massive scale, we are billions of humans all experiencing the world, AI needs probably a similar number of agents doing massive trials and search. AlphaGo sure did need lots of self-play games, not just a couple. AlphaTensor learned to improve matrix multiplication by the same method. Biological evolution produced us the same way.
It's an open-ended exploration problem. A closed system can't do it, and the scale needed makes it expensive.
[1] - Evolution through Large Models - https://arxiv.org/abs/2206.08896
Have I got a link for you:
> Evolution through Large Models
They are using a diff-LM trained on GitHub commits to generate mutations to a population of problem solutions expressed as code.
https://arxiv.org/abs/2206.08896
The non intuitive bit is why are these models surpassing the original programming? It's because they got a secondary source for learning - the feedback they get from generating solutions and testing them. Learning through feedback, be it reinforcement, evolutionary or gradient based, is what can push AI forward.
AlphaZero started with no human knowledge by pure self-play and beat all humans in a few days of practice. It learned from game outcomes as a feedback source. AI can learn from many things, not just from humans. But if it doesn't require humans it usually requires massive computation, simulation or search, maybe even real world testing which can get expensive.
> If a model were to return a new thing (not found in the training set) it would be down-weighted
LLMs can also learn by reinforcement learning and evolutionary techniques. They don't do just next token prediction, and they can even generate their own training data.
Evolution through Large Models
https://arxiv.org/abs/2206.08896
In their paper "Evolution through Large Models", CarperAI shows how you can use diffs to evolve code, while running the code against a test or an environment for validation.
https://arxiv.org/abs/2206.08896
The idea is: LLMs know how to modify code in semantically useful ways. Evolutionary algorithms are great at search, but don't learn new mutations by themselves. So combine them together to generate new data and retrain the models.
So the old system is: scrape human code and train on it. the new system is: generate code, keep the good parts and retrain. It only costs electricity and is open-ended.
There are two ways to learn: from imitation (language models are stochastic parrots) and from validation (problem solving, playing games). Many people only remember the first and forget the second. But the second one has the key to increasing intelligence.
I don't know what the authors mean by "current algorithmic frame of AI research" because it is very diverse. But evolutionary methods work well with neural networks, they are not just next token predictors.
> Evolution through Large Models
https://arxiv.org/abs/2206.08896
It is true that simply using a dataset or a bunch of environments is not enough.
> General Intelligence Requires Rethinking Exploration
https://arxiv.org/abs/2211.07819
But when sufficient data can be generated from acting in the environment, then a model can surpass human level, like AlphaZero.
While I agree with the authors that large language models only trained on text lack the ability to distinguish "possible worlds" from reality, I think there is a path ahead.
Large language models might be excellent candidates for evolutionary methods and RL. They need to learn from solving language problems on a massive scale. But problem solving could be the medicine that cures GPT-3's lack of grounding.
For example: "Evolution through Large Models" https://arxiv.org/abs/2206.08896
They need learning from validation to complement learning from imitation.
AI is not just that. That's a myth.
Yes, when you train on a human generated corpus you're going to get an interpolative AI. But when you train on a dataset generated by AI, you can surpass human level.
AlphaZero started, as the name says, from zero. No examples of how we play the game. In just three days of self-play it surpassed the best human. How was that possible? We got a 2000 year head start, larger brains, and many players, not just one model. AlphaZero did it by creating its own training data.
Here is an example of using code language models for this kind of dataset-creation by play.
> Evolution through Large Models
https://arxiv.org/abs/2206.08896
By the way,…
Read the full comment
GPT-3 and chatGPT are not simple interpolative language models. They have also been trained on many tasks, problems, and codes, which is the kind of data that will awaken the skills of the model to a whole new level. What I am saying is that we can auto-generate more problem data and increase their skills.
> If we had a way of differentiating the wheat from the chaff
This is the key to AGI. We need verification systems, they can be a code execution environment, a database of facts, a math symbolic engine, a physical simulation, a game, or real world lab experiments. These verifiers will produce signal that can be used by the language models to improve. The cheaper and faster verification is, the faster we can iterate. Generating ideas is cheap, proof matters.
Just remember AlphaZero a bit - it started from scratch, playing against itself, in a few hours it surpassed human level. Go simulation and verification is trivial. The board is just a matrix. So learning from massive search and verification is a proven path to super-human level.
Here is a related paper:
> Evolution through Large Models
https://arxiv.org/abs/2206.08896
TL;DR - Models can learn from massive validation, not just massive text prediction.
I agree our current crop of Copilot, GPT-3 and chatGPT can introduce subtle bugs, and that's a big problem. Checking for such bugs takes time. They code like someone who read a lot and has good memory, but never implemented anything complex.
But there is no reason to be the same in the next iteration. Language models can be fine-tuned on code execution, that means giving sufficient exposure to trial and error to learn how to avoid such bugs.
Having a way to test its outputs is the key to self improvement. It works the same with AlphaGo - the model is generating its own training data, and…
Read the full comment
this process can eventually surpass human level. It just needs tons of compute to iron out a lot of bad ideas.
Alternatively, the model could use a code execution environment at runtime, too. So it could iterate a few times on an error, eventually using web search if it cannot solve it alone, instead of simply failing and leaving it to the human to fix. It is pretty good at proposing fixes and will recover most of the times, like human devs.
I'm leaving this for further reading. It's fascinating.
> Evolution through Large Models
> Large language models (LLMs) trained to generate code can vastly improve the effectiveness of mutation operators applied to programs in genetic programming.
https://arxiv.org/abs/2206.08896
The paper is from OpenAI, Ken Stanley who works on opendedness is the last author.
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.