Guillaume Lample, François Charton
2019 · arXiv
Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
I'm not sure anyone is claiming that ChatGPT is going to steal their job. I think most people are just extrapolating the ridiculously steep rate of progression and pondering what the future holds. Sure, maybe we're about to hit a wall. But I'm not seeing any evidence of that as it stands. Also, it's not clear to me why it's impossible for an LLM to learn symbolic manipulation? https://arxiv.org/abs/1912.01412
It's a model mismatch, not an inherent impossibility. A calculator needs to have an adaptive number of intermediate steps. Usually our models have fixed depth, but in auto-regressive modelling the tape can become longer as needed by the stepwise algorithm. Recent models show LMs can do arithmetic, symbolic math and common sense chain-of-thought step by step reasoning and reach much higher accuracies. In other words, we too can't do three digit multiplication in our heads reliably, but can do it much better on paper, step by step. The problem you were mentioning is caused by the bad approach - LMs need intermediate reasoning steps to get from problem to solution, like us. We just need to ask them to produce the whole reasoning chain. - Chain of Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903 - Deep Learning for Symbolic Mathematics https://arxiv.org/abs/1912.01412
It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one, mostly based on repeating whatever we have read about in popular science literature without any intuitive understanding. GPT-3 understanding of physical world is similarly bad. It's just simply a modality not available to the model. But GPT-3 is able to do few-shots learning. If you show it couple of times how to do something it will try to repeat after you showing some rudimentary understanding of what you are trying to do. It's even able to learn to do simple…
analogies [0]. GPT-3 is able to do very, very simple arthritics like addition and subtractions with 2-3 numbers. Actually even a lot smaller models trained on arthritics are able to do impressive feats like symbolic integration and solving differential equations. > Deep Learning for Symbolic Mathematics [1]: Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica. What is true is that GPT-3 is not very good at following instructions. It would not be able to execute an algorithm. So, it would not learn arithmetic from instructions if you would not include explicit examples. [0] https://medium.com/@melaniemitchell.me/can-gpt-3-make-analog... [1] https://arxiv.org/abs/1912.01412
They don't actually use trees in the paper though. https://arxiv.org/abs/1912.01412 They express everything in reverse polish notation, exactly so the transformer network can work on streams and not care about the nesting levels. It's a bit odd that this article doesn't talk about that.
To me, the article somewhat misses the point of what's interesting here. Using ASTs to represent equations, or even whole programs has plenty of precedents in ML/AI. I'd have liked to know how exactly they translate these trees to a representation suitable for an ANN. Fortunately, the paper seems to be easy to find and access (it's [1], I guess). [1] https://arxiv.org/abs/1912.01412
More from the same topics.
William P. Thurston
In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.