Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu
Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving
2019 · arXiv
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
That started years ago. Exactly when depends what you count, but part of the path leading to ChatGPT was getting a bunch of dumb systems to work together to train a smarter system: https://arxiv.org/pdf/1909.08593 fig 1, and also section 4.4 for what can go wrong
i highlighted this last night which seems to be making the rounds - https://twitter.com/swyx/status/1699369076529971545 Alec's CV (https://www.linkedin.com/in/alecradford/) seems to be: - 2011-2016 BSc Eng from Olin College - 2013+ started a data/AI consultancy as a sophomore that turned into a vague startup/product? - 2015 first paper on GANs coauthored with soumith https://arxiv.org/pdf/1511.06434.pdf%C3 - 2016 first GAN paper under openai email https://proceedings.neurips.cc/paper_files/paper/2016/file/8... - 2017 the generative reviews paper mentioend in the Wired article, with Ilya https://arxiv.org/pdf/1704.01444.pdf - 2017 coauthor on PPO paper (precursor to…
instructgpt/rlhf) https://arxiv.org/pdf/1707.06347.pdf). - 2018 lead author on GPT1 https://www.mikecaptain.com/resources/pdf/GPT-1.pdf - 2019 lead author on GPT2 https://insightcivic.s3.us-east-1.amazonaws.com/language-mod... blog https://openai.com/research/better-language-models - 2019 coauthor on RLHF https://arxiv.org/pdf/1909.08593.pdf) - 2020 coauthor on gpt3 https://arxiv.org/abs/2005.14165 - 2020 coauthor on scaling laws https://arxiv.org/pdf/2001.08361.pdf%E4%B8%AD%E5%BE%97%E5%88... - 2021 coauthor on DallE http://proceedings.mlr.press/v139/ramesh21a/ramesh21a.pdf - 2021 coauthor on Codex https://arxiv.org/pdf/2107.03374.pdf?trk=public_post_comment... - 2023 lead author on Whisper https://proceedings.mlr.press/v202/radford23a/radford23a.pdf name a more successful 7 year CS career post undergrad... Update: FYI openai just announced a "developer day" in Nov - somehow not blessed by the HN gods https://news.ycombinator.com/item?id=37408234
My understanding is that there’s no RLHF model sitting on top of ChatGPT. There’s a separate reward model during training, but that reward model is just used to fine-tune the decoder model. This is sort of confusing in the literature, because they will switch from calling it a “language model” to a “policy”, once they’ve used PPO, but if you trace the citations back to [1], you can see that the policy is initialized with a pretrained language model (in that case, GPT-2). What I don’t know is how they’re applying RLHF after user fine-tuning. Are they redoing the PPO with the original reward model after tuning on your input? Are they just letting it slide, and hoping that the fine-tuning doesn’t cause the model to forget the RLHF? It’s unclear from what I’ve read. [1] https://arxiv.org/pdf/1909.08593.pdf
Reinforcement learning in general didn't come out of AI alignment work, but RHLF in particular did. The initial idea and paper [1] were from AI alignment folks, as was most of the later development [2][3][4][5]. Overview: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoug... [1] Deep Reinforcement Learning from Human Preferences https://arxiv.org/abs/1706.03741 [2] Fine-Tuning Language Models from Human Preferences https://arxiv.org/abs/1909.08593 [3] Learning to summarize from human feedback https://arxiv.org/abs/2009.01325 [4] Recursively Summarizing Books with Human Feedback https://arxiv.org/abs/2109.10862 [5] Training language models to follow instructions with human feedback https://arxiv.org/abs/2203.02155
What's particularly nice is if you can plug in a classifier for things like esthetics based on human ratings, along the lines of 'learning from human preferences' https://arxiv.org/abs/1909.08593 but better - why spend the enormous effort running PPO to brute force the classifier to obtain desired text or image output, when you can just backprop through it and let the classifier itself tell you how exactly to improve the inputs? (I was writing up this idea as a proposal for improving preference learning at https://www.gwern.net/GPT-2-preference-learning#optimization... based on my experiences idly waiting for PPO to run and being annoyed by the divergences, and then Uber just... blogs it out. Like a madman.)
> Lots of developers here, but also in the media, believe GPT is the precursor for AGI. Which is ridiculous; it's just an incredible token predictor. Its language capabilities might be useful, but AGI won't be possible without a self-correcting model that can actually reason. Even Altman hinted at that. I think part of this is that all three letters of the initialism mean different things to different people. To me? The domain is broad rather than narrow, so it's got the "G" of "AGI". Definitely agree it is not a complete solution, and thus that reasonable people will say "no Ben this isn't general what are you talking about". I also definitely agree isn't superhuman, but…
I think superhuman ability across all of its domain is an unreasonable requirement for AGI and what I would instead call that ASI… but I'm also a linguistic descriptivist, so it doesn't matter if the consensus is unreasonable[0] just what the consensus actually is. Also: my understanding from "Fine-Tuning Language Models from Human Preferences"[1] (in particular the summary version of the bug, "The True Story of How GPT-2 Became Maximally Lewd"[2]) is that these models are capable of self-correcting behaviour — we just don't do that live because that makes it much harder to research and to use (every time it even seems like the ChatGPT model changes, someone complains). > But it will keep getting stuck on the accuracy part I suspect otherwise. Some — not all, but some — domains have easily testable results. In these domains, the generated output can be quickly tested, and used to update the model. This is already done with human preferences, but I'm thinking more along the lines of a coding assistant which gets feedback from a compiler, or a maths assistant which gets feedback from… maths isn't my field, I'm just going to say MATLAB or the Wolfram Language and hope these are both as highly regarded as Photoshop. [0] this happens a lot: tautologies such as "ATM machine"; or redefinitions such that "PC" excludes MacOS and all phones and tablets; that Elon Musk can only be called "African-American" by pedants (including me, despite being a descriptivist); exactly how effective and how altruistic is "Effective Altruism" etc. [1] https://arxiv.org/pdf/1909.08593.pdf [2] https://forum.effectivealtruism.org/posts/5mADSy8tNwtsmT3KG/...
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.