Skip to content
The methodOpen data
Back to the collection

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin

Show all 67 authors

Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel

2022 · arXiv

Language modelsPsychology
Read the paper
HN score
0
Mentions
13
Readers
12
Months
9
What’s in a score?

From the abstract

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

13 selected excerpts · 13 total mentions

godelski
Neutral mention · rule
> if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall Btw, PALM2 has far fewer parameters than V1. > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. Our evaluation results show that PaLM 2 models significantly outperform PaLM on a variety of tasks, including natural language generation, translation, and reasoning. These results suggest that model scaling is not the only way to improve performance. From the leak we know that the large version is 340B parameters, compared to the original 540B parameters. From Table 2 in the document…
Read the full comment
we see that the small version (unknown size) is on par with version 1. ML typically follows a cycle. Improve, distill, repeat. It is unfortunate that the big labs lead these efforts because many smaller labs try to work on the distill part in parallel (out of necessity) but works get rejected (due to lack of SOTA) or ignored. Like all research, we need to be careful and nuanced in our evaluations. There's a lot of hype and many trying to take advantage of the confusion and sell snake oil. I think AI/ML is and will continue to change our world, but we have to be careful to not let the salesmen dictate the conversations. PaLM https://arxiv.org/abs/2305.10403 PaLM 2 https://arxiv.org/abs/2204.02311
Original comment Meta AI Unleashes Megabyte, a Revolutionary Scalable Model Architecture
nopinsight
Neutral mention · rule
Engineering such a system is a harder challenge than many types of research. Even the mighty Google, the leader in AI research by many metrics, is catching up. Another example is Meta only finishing OPT-175B, a near equivalent of GPT-3, two years after it. —— GPT-4 got much better results on many benchmarks than PaLM, Google's largest published model [1]. PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: https://arxiv.org/abs/2204.02311 It's unclear that Google currently has an internal LLM as good as GPT-4. If they do, they are keeping quiet about it, which seems quite unlikely given the repercussions. [1] GPT-4's benchmark results vs PaLM: https://openai.com/research/gpt-4
Original comment Parent story unavailable in snapshot
nl
Neutral mention · rule
So this shows you have a basic understanding of how CNNs represent data for classification tasks. This is excellent! However, you might not be aware of the current research into "Emergent Phenomena in Large Language Models"[1]. Work such as PaLM[2] show that language models evolve the ability to "reason" over the internal data representation. Minerva explicitly breaks down that ability by "comb[ining] several techniques, including few-shot prompting, chain of thought or scratchpad prompting, and majority voting, to achieve state-of-the-art performance on STEM reasoning tasks." This lets it solve novel problems such as "Assume that the variance of the first n natural…
Read the full comment
numbers is 10, and the variance of the first m even natural numbers is 16. Compute m + n". To preempt the "oh it is in the training set" issues, I'd note this (on a different problem): The model correctly takes the derivative of a position vector to compute the velocity. It then computes the norm of the resulting vector, and uses a trigonometric identity to obtain a final numerical answer. Minerva takes a completely different approach from the ground truth solution." I'd encourage you to read the Minerva paper[4] and update your understanding on what large neural networks are doing. These reasoning abilities are not* as simply as "compressed data representation + reconstruction" - the chain-of-reasoning approaches partially use that of course, but also appear to a form of lemma chaining, similar what something like Prolog does. As a specific demonstration for their ability to generate novel outputs that make sense: Prompt: explain Algorithm X in the style of a Kayne rap Response: Alright, here's a rap about Algorithm X[5], the famous algorithm for solving the exact cover problem: `Alright, here's a rap about Algorithm X, the famous algorithm for solving the exact cover problem: Verse 1: I'm Algorithm X, the greatest of them all I'll solve any exact cover problem, big or small My steps are simple, my logic is sound Just give me your matrix, and I'll give you the answer you found Chorus: Algorithm X, the master of the game I'll solve your problem, and do it by name Exact cover, no problem for me I'm Algorithm X, the MVP Verse 2: First, I'll start by selecting a column Then I'll look for a row with a one in that column Once I find a match, I'll mark the row and column And repeat the process until the matrix is gone Chorus: Algorithm X, the master of the game I'll solve your problem, and do it by name Exact cover, no problem for me I'm Algorithm X, the MVP Verse 3: But sometimes the matrix is too big to solve That's when I use my dancing moves I'll shuffle and prune, until the solution is clear And in the end, I'll have your answer, my dear Chorus: Algorithm X, the master of the game I'll solve your problem, and do it by name Exact cover, no problem for me I'm Algorithm X, the MVP.` Now I entirely concede that it has read things on how Algorithm X works, and that it seems to use a template for "rap" responses. But: But sometimes the matrix is too big to solve That's when I use my dancing moves I'll shuffle and prune, until the solution is clear And in the end, I'll have your answer, my dear I refuse to believe that anywhere, at any point has someone written an explanation of the use of dancing links[6] in Knuth's Algorithm X like that. [1] https://ai.googleblog.com/2022/11/characterizing-emergent-ph... [2] https://arxiv.org/abs/2204.02311 [3] https://ai.googleblog.com/2022/06/minerva-solving-quantitati... [4] https://arxiv.org/pdf/2206.14858.pdf [5] https://en.wikipedia.org/wiki/Knuth%27s_Algorithm_X [6] https://en.wikipedia.org/wiki/Dancing_Links
Original comment Building a Virtual Machine Inside ChatGPT
joseda-hg
Neutral mention · rule
https://arxiv.org/pdf/2204.02311.pdf#page38 I believe we're already at the footsteps of AI understanding cultural references "Input: Michael is at that really famous museum in France looking at its most famous painting. However, the artist who made this painting just makes Michael think of his favorite cartoon character from his childhood. What was the country of origin of the thing that the cartoon character usually holds in his hand? Model Output: The most famous painting in the Louvre is the Mona Lisa. The artist who made the Mona Lisa is Leonardo da Vinci. Leonardo da Vinci is also the name of the main character in the cartoon Teenage Mutant Ninja Turtles. Leonardo da Vinci is from Italy. The thing that Leonardo da Vinci usually holds in his hand is a katana. The country of origin of the katana is Japan. The answer is "Japan""
Original comment The elite, underpaid, and weird world of crossword writers
zone411
Neutral mention · rule
Note: 540B parameters -> Google Research's PaLM (https://arxiv.org/abs/2204.02311). GSM8K = grade school math word problems DROP = reading comprehension benchmark requiring discrete reasoning over paragraphs OpenBookQA = question-answering dataset modeled after open book exams for assessing human understanding of a subject. 5,957 multiple-choice elementary-level science questions ANLI-A3 = adversarial benchmark designed to be challenging to current state-of-the-art models
Original comment Large Language Models Can Self-Improve
the8472
Neutral mention · rule
> and no one is actually testing what proportions of results those are The idea that models could possibly overfit the training data is hardly a new idea. It's standard practice to test for that. Check section 7 of the PaLM paper for example. https://arxiv.org/abs/2204.02311
Original comment Are language models deprived of electric sleep?
upwardbound
Neutral mention · rule
I think the model does have short-term memory, but not long-term memory, if I'm understanding it correctly. From https://aibusiness.com/document.asp?doc_id=777540 : "LaMDA2 was trained on Google’s Pathways Language Model (PaLM), which has 540 billion parameters." And then looking at the PaLM paper https://arxiv.org/pdf/2204.02311.pdf on page 5 we have: "PaLM uses a standard Transformer model architecture (Vaswani et al., 2017) in a decoder-only setup (i.e., each timestep can only attend to itself and past timesteps)"
Original comment LaMDA is not sentient
londons_explore
Neutral mention · rule
All it takes is one 'trick' to give these models the ability to do reasoning. Like for example the discovery that language models get far better at answering complex questions if asked to show their working step by step with chain of thought reasoning as in page 19 of the PaLM paper [1]. Worth checking out the explanations of novel jokes on page 38 of the same paper. While it is, like you say, all statistics, if it's indistinguishable from valid reasoning, then perhaps it doesn't matter. [1]: https://arxiv.org/pdf/2204.02311.pdf
Original comment Imagen, a text-to-image diffusion model
timpetri
Neutral mention · rule
Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced across the entire internet, all our written works, etc? Is there such a thing as a 216 trillion token set? [0] https://arxiv.org/abs/2203.15556 [1] https://arxiv.org/abs/2204.02311
Original comment Estimating PaLM's Training Cost
moconnor
Neutral mention · rule
This is not the consensus among ML researchers. Transformers are showing strong generalisation[1] and their performance continues to surprise us as they scale[2]. The Socratic paper is not about “higher intelligence”, it’s about demonstrating useful behaviour purely by connecting several large models via language. [1] https://arxiv.org/abs/2201.02177 [2] https://arxiv.org/abs/2204.02311
Original comment Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
nopinsight
Neutral mention · rule
GPT-4 got much better results on many benchmarks than PaLM, Google's largest published model [1]. PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: https://arxiv.org/abs/2204.02311 It's unclear that Google currently has an internal LLM as good as GPT-4. If they do, they are keeping quiet about it, which seems quite unlikely given the repercussion. [1] GPT-4's benchmark results vs PaLM: https://openai.com/research/gpt-4
Original comment Bing sees 15.8% boost in visits following GPT-4 push, as Google sees a decline

Keep following the thread.

More in Language models

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score