Emergent Abilities of Large Language Models

of our brain is central to our reasoning abilities. Maybe in some way, the autoregressive nature of LLMs is similar to our predictive brain. For a collection of emergent properties in LLMs, I recommend this paper [2] [1]https://arxiv.org/abs/2201.11903.pdf [2]https://arxiv.org/pdf/2206.07682.pdf

relevant. "There is no sense in which they are mathematically destined to eventually program well" - Yes there is and this belies and ignorance of the literature and how things work - Again: RL has been around forever. Scaling laws have held empirically up to the largest scales we've tested. There are known RL scaling laws for both training and test time. It's ludicrous to state there is "no sense" in this, on the contrary, the burden of proof of this is squarely on yourself because this has already been studied and indeed is the primary reason why we're able to secure the eye-popping funding: contrary to popular HN belief, a trillion dollars of CapEx spend is based on rational evidence-based decision making. > "How good are these types of algorithms at generalization" There is a tremendously large literature and history of this. ULMFiT, BERT ==> NLP task generalization; https://arxiv.org/abs/2206.07682 ==> emergent capabilities, https://transformer-circuits.pub/2022/in-context-learning-an... ==> demonstrated circuits for in context learning as a mechanism for generalization, https://arxiv.org/abs/2408.10914 + https://arxiv.org/html/2409.04556v1 ==> code training produces downstream performance improvements on other tasks > Verifiable domains are interesting. It is unquestionably why agents have come first for coding. But if you've played with claude you may have experienced it short-circuiting failing tests, cheating tests with code that does not generalize, writing meaningless tests, and at long last if you turn it away from all of these it may say something like "honest answer - this feature is really difficult and we should consider a compromise." You say this and ignore my entire argument: you are right about all of your observations, yet - Opus 4.6 compared to Sonnet 3.x is clearly more generalizable and less prone to these mistakes - Verifiable domain performance SCALES, we have no reason to expect that this scaling will stop and our recursive improvement loop will die off. Verifiable domains mean that we are in alphago land, we're learning by doing and not by mimicking human data or memorizing a training set.

difficult to train for solving calculations. Where they will do well on problems with the same number of digits but this will degrade rapidly as number of digits increase. It can be weird to read some of these papers as there will sometimes be periodic relationships with the number of digits but that should give us information about how they're encoding the problems. But that lack of transferability indicates that despite the problem solving and what we'd believe is actually just the same problem, doesn't mean it is. So you have to be really careful here, because us humans are really fucking good at generalization (yeah, we also suck, but a big part is our proficiency makes us recognize where we lack. But also, this is more a "humans can" more than "humans do" type of thing. So be careful when comparing). This generalization is really because we're focused around building causal relationships, while on the other hand the ML algorithms are build around compression (i.e. fitting data). Which, if you notice, is the same issue I was pointing to above. > Ie, you are the Oracle and whatever model is being trained doesn't know the answer, only if it is right or wrong. But I don't know if the reward function must be binary or on a scale. This entirely depends on the problem. We can construct simple problems that both illustrate success as well as failure. What you really need to think about here is the information gain from the answer. If you check how to calculate that, you will see the dependence (we could get into Bayesian Learning or experiment design but this is long enough). But let's think of a simple example in the negative direction. If I ask you to guess where I'm from, you're going to have a very hard time pinning down the exact location. Definitely in this example there is a efficient method, but our ML learning algorithms don't start with prior knowledge about strategies and so they aren't going to know to binary search. If you gave that to the model, you baked in that information. This is a tricky form of information leakage. It can be totally fine to bake in knowledge, but we should be aware of how that changes how we evaluate things (we always bake in knowledge btw. There is no escaping this). But most models would not have a hard time if instead we played "hot/cold", because the information gain is much higher. We've provided a gradient to the solution space. We might call this hard and soft labels, respectively. I picked this because there's a rather famous paper about emergent abilities (I fucking hate this term[0]) in ML models[1], and a far less famous counter to it[2]. There's a lot of problems with [1] that require a different discussion but [2] shows how a big part of the issue is how many of the loss landscapes are fairly flat and so when feedback is discrete the smaller models just wonder around that flat landscape needing to get lucky to find the optima (btw, this also shows that technically this can be done too! But that would require different training methods and optimizers). But when giving them continuous feedback (i.e. you're wrong, but closer than your last guess), they are able to actually optimize. A big criticism of the work is that it is an unfair comparison because there are "right and wrong" answers here, but it'd be naive to not recognize that some answers are more wrong than others. Plus, their work shows a clear testable way we can confirm or deny if this works or not. We schedule learning rates, there's no reason you cannot schedule labels. In fact, this does work. But also look at the ways they tackled these problems. They are entirely different. [1] tries to do proof by evidence while [2] uses proof by contradiction. Granted, [2] has an easier problem since they only need to counter the claims of [1], but that's a discussion about how you formulate proofs. So I'd be very careful when using the recent advancements in ML as a framework for modeling reasoning. The space is noisy. It is undeniable that we've made a lot of advancements but there is some issues with what work gets noticed and what doesn't. A lot does come down to this proof by evidence fallacy. Evidence can only bound confidence, it can unfortunately not prove things. But this is helpful and well, we can bound our confidence to limit the search space before we change strategies, right? I picked [1] and [2] for a reason ;) And to be clear, I'm not saying [1] shouldn't exist as a paper or that the researchers were dumb for doing it. Read back on this paragraph, because we've got multiple meta layers here. It's good to place a flag in the ground, even if it is wrong, because you gotta start somewhere, and science is much much better at ruling things out than ruling things in. We more focus on proving things don't work until there's not much left and then accept those things (limits here too, but this is too long already). I'll leave with this, because now there should be a lot of context that makes this much more meaningful: https://www.youtube.com/watch?v=hV41QEKiMlM [0] It significantly diverges from the terminology used in fields such as physics. ML models are de facto weakly emergent by nature of composition. But the ML definition can entirely be satisfied by "Information was passed to the model but I wasn't aware of it" (again, same problem: exhaustive testing) [1] (2742 citations) https://arxiv.org/abs/2206.07682 [2] (447 citations) https://arxiv.org/abs/2304.15004

predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models. Version history (for relevant dates): [v1] Wed, 15 Jun 2022 17:32:01 UTC (59 KB) [v2] Wed, 26 Oct 2022 05:06:24 UTC (88 KB)

means that LeaderBoards and test scores for LLMs provide a low-confidence metric for what the propose to be testing. The paper's title is itself a continuation of the phenomenon where papers are named after the landmark paper "Attention Is All You Need". [4] A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results. 1. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... 2. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... 3. https://arxiv.org/abs/2206.07682 4. https://arxiv.org/abs/1706.03762 5. https://arxiv.org/abs/2306.11644

incapable of doing $X > $Y or displaying numeracy, which I refuted by showing an example of an LLM doing greater than comparisons, and subtracting a quantity in different units ($50k -> 50,000). Now you are substituting a much narrower claim, that an LLM is stucturally incapable of symbolic manipulation and “really modeling numbers”. This might be so! But it’s not required for basic numeracy, “tricks” as you put it, or whatever else GPT has learned, can objectively get us to median human performance. Even going way back to GPT-2 there are mechanistic interpretability papers investigating how greater-than is implemented, eg https://arxiv.org/abs/2305.00586. And there is work that suggests that LLMs do some sort of phase transition to gain numeracy skills: https://arxiv.org/pdf/2206.07682.pdf. Your objection about working memory is also odd. Chain of thought reasoning strategies use the context as the working memory and have been demonstrated to improve performance on numeracy tasks. But again, if you are retreating to a very narrow claim that the model can’t do precise calculations in a single inference step, then sure, that’s technically plausible, but that’s a way higher bar than displaying basic numeracy, and doesn’t justify the incredulity in your GP comment.

communication of the idea that larger models can do more than smaller models, which should be a surprise to no one. Stop spreading this nonsense. I'm not saying that LLMs aren't impressive. I'm just saying this breathless fantasy where they're doing totally unexpected and unexplained things that are beyond human understanding, is totally false. Since this is controversial and seemingly most HN folks can't hold a conversation with any nuance, if you don't use the word "shape" in your response to this post, I'm simply going to point out that you didn't read the post you're responding to, and therefore shouldn't be responding. If you stop reading at the first chance you see to correct something, go away--you're dragging down the level of the conversation.

prompted 'normally' and then smooth scaling when an inner-monologue prompt is used). Similarly, 'inverse scaling' turns out to be 'U-shaped' when tested on PaLM ( https://arxiv.org/abs/2211.02011v3#google ), but no one had any good way of predicting what tasks would show inverse scaling nor whether PaLM would be adequate. There's a lot we do not know about DL scaling right now, which means we definitely do not know how consciousness does or doesn't work in DL scaling. Not that we have any idea how to benchmark it right now, nor is anyone doing so, so of course it may not need to emerge at all... There's also concerning scaling trends like power-seeking or self-preservation ( https://arxiv.org/abs/2212.09251#anthropic ), results which may sting a little more right now if you've been reading the Bing Sydney transcripts - at present, those outputs are 'just' imitation/memorization but there is no way to know at what threshold the memorization is replaced by generalization and becomes genuine agency. (Sufficiently advanced imitation is indistinguishable from the real thing.)