On the Measure of Intelligence

system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. https://arxiv.org/abs/1911.01547

who coded it for this specific challenge. The point of ARC-AGI is to test the intelligence of AI systems in novel, but simple, environments. Having a human give it more powerful tools in a harness defeats the purpose. You should go back and read the original ARC-AGI paper to see what this is about+. Are you upset about the benchmark because frontier LLM models do so poorly exhibiting the ability to generalize when the benchmarks are released? + https://arxiv.org/abs/1911.01547

runway. - Take the "bitter lesson" seriously: most of the successes in modern machine learning are due to deep learning and the ability to leverage vasts amounts of compute and data. I want to study how DL/LLM models reason, using ARC as a benchmark (https://arcprize.org/, https://arxiv.org/abs/1911.01547). Techniques I want to investigate include test time compute, active inference, meta-learning, self-play, or any technique that would be considered "system 2". With that in mind, today I'm looking for the following role: - ML research engineer Compensation will include flexible options scheme based on results. Interest and knowhow in the following is a plus: Deep learning, transformer models, reasoning, world models, learning-to-learn, mechinterp, bayesian methods, probabilistic programming. I will mostly consider candidates having an EU passport for visa reasons. To apply, email me at not_a_cat at fastmail dot com.

any attempt to use that concept, so at the very least we can know that the tasks solved so far do not need any core knowledge priors to be solved. But, just to be perfectly clear: when results are posted, they are measured by simple comparison of the target output grids with the output grids generated by a system. Not by comparing the method used to solve a task. Also, if I may be critical: you can find this information all over the place online. It takes a bit of reading I suppose, but it's public information.

because he became skilled at chess in one human lifetime, starting out with only basic priors about the world like objectness, agentness, and some others I can't remember. In contrast, DeepBlue is not as intelligent, because it started with many more priors specific to chess, and AlphaZero is also not as intelligent, because it acquired its chess skill much less efficiently, requiring the equivalent of thousands of human lifetimes. I don't remember if the paper considers whether 3.5 billion years of "pre-training" counts in this regard, though. [1] https://arxiv.org/abs/1911.01547

much more complicated of course, but the principles remain. > I subscribe to the view we think of the human mind as a pattern matching computer simply because this is the current major tech, much like people in the past thought of "humors" or "steam machines". I think some of the analogies are useful, to a point, but I don't think there's hard evidence the mind is like a neural net (irony notwithstanding) or a pattern marching GPT-like algorithm. I don't think the computational approach is an interesting question. No one thinks brains operate like software neural networks (not sure what "pattern marching GPT-like algorithm" means - it is just a neural network). That doesn't matter because all computational methods are ultimately equivalent. I think outcomes on metrics like benchmarks is important, and while some benchmarks have issues that some approaches exploit I think things like Chollet's "On the Measure of Intelligence" (https://arxiv.org/abs/1911.01547) are reasonable frameworks for discussion of progress. > Re: Thinking Fast and Slow, I see there are serious doubts about the validity of the book's foundations and conclusions, and that it's been challenged. This in inaccurate. There are some reasonable criticisms of the studies on the priming affect (one chapter of the book). These and Kahneman's response (he accepts the criticisms) are outlined in https://replicationindex.com/category/thinking-fast-and-slow... and https://retractionwatch.com/2017/02/20/placed-much-faith-und... These issues don't detract from the overall theme of the book about the two systems of decision making (rational and reflex) and how often we use the reflex decision system but convince ourselves we are using the rational system. If you can point at any additional doubts about the books conclusions I'd appreciate a reference.

skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power." He argues that we should move towards evaluating "Intelligence as skill-acquisition efficiency". I agree with him. We should move away from benchmarks that involve training and evaluating algorithms on the same datasets. This is indeed more or less "curve fitting". Instead we should focus on benchmarking how efficient algorithms are at solving tasks involving completely new datasets, preferably even unknown to the developers. For example, language model GPT-2 was trained to predict next word given some previous words. After that training GPT-2 was able to do things that were unrelated like question answering, translating etc. GPT-2 is of course doing that very badly, and requires GB of training data, but it is a step towards skill-acquisition efficiency and away from what everyone sees as curve-fitting. We should benchmark models so that we select for these that are able to do solve tasks they were not build to solve.