Stéphane d'Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, François Charton
2022 · arXiv
Symbolic regression, i.e. predicting a function from the observation of its values, is well-known to be a challenging task. In this paper, we train Transformers to infer the function or recurrence relation underlying sequences of integers or floats, a typical task in human IQ tests which has hardly been tackled in the machine learning literature. We evaluate our integer model on a subset of OEIS sequences, and show that it outperforms built-in Mathematica functions for recurrence prediction. We also demonstrate that our float model is able to yield informative approximations of out-of-vocabulary functions and constants, e.g. $\operatorname{bessel0}(x)\approx \frac{\sin(x)+\cos(x)}{\sqrt{πx}}$ and $1.644934\approx π^2/6$. An interactive demonstration of our models is provided at https://symbolicregression.metademolab.com.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
Nothing prevents it, no. Transformers are certainly capable of learning mathematical tasks; consider [1] as an example, which uses big but regular token lengths. Alternatively you could just scale 'till the problem solves itself. [1] https://arxiv.org/abs/2201.04600
Sure: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Both of these models can generalize to numbers it have not seen.
tl;dr - They model operations (e.g. multiply, add, cosine, sqrt) as tokens, and use a transformer to predict a recursive function from (incomplete) integer sequences (and float sequences, separately). This is not dissimilar from e.g. DALL-E where they model images as NLP-style tokens and use a transformer to predict image tokens (operations, respectively) from text tokens (list of ints/floats, respectively). This demo lets you interact with the model. Pretty cool stuff. Refreshing to see symbolic problems being tackled with deep learning. Yannic Kilcher video with one of the authors: https://youtu.be/1HEdXwEYrGM arxiv: https://arxiv.org/abs/2201.04600
Take a look at this paper: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 If you look at embedding visualization it is very clear that the model learns order of numbers. (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) There is also: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Again, looking at visualizations the model very clearly grasps the structure of the function it models.
More from the same topics.
William P. Thurston
In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.