Skip to content
The methodOpen data
Back to the collection

Deep reinforcement learning from human preferences

Paul Christiano, Jan Leike, Tom B. Brown

Show all 6 authors

Miljan Martic, Shane Legg, Dario Amodei

2017 · arXiv

Machine learningRoboticsPsychology
Read the paper
HN score
0
Mentions
5
Readers
5
Months
5
What’s in a score?

From the abstract

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

godelski
Neutral mention · rule
Jack has a lot of really bad takes and frequently makes lots of mistakes. Honestly, I don't know why people take him seriously. I mean you can go read his blog post that's pinned where he argues that there's no new ideas and it is all data. He makes the argument that architecture doesn't matter, which is just so demonstrably false that it is laughable. He's a scale maximalist. I also expect an AI researcher from a top university to not make such wild mistakes > 3. RLHF: first proposed (to my knowledge) in the InstructGPT paper from OpenAI in 2022 I mean if you go read the instruct paper on page 2 you'll see | Specifically, we use reinforcement learning from human…
Read the full comment
feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020) to fine-tune GPT-3 to follow a broad class of written instructions (see Figure 2). Where in Christiano you'll find || Our algorithm follows the same basic approach as Akrour et al. (2012) and Akrour et al. (2014) I mean this is just obviously wrong. It is so obviously wrong it should make the person saying it second guess themselves (which is categorically the same error you're pointing out). I'm sure we can trace the idea back to the 80's if not earlier. This is the kind of take I'd expect a non-researcher to have, but not someone with two dozen NLP papers. The Instruct-GPT paper was just the first time someone integrated RLHF into a LLM (but not a LM). Maybe a better article is the one he wrote on Super Intelligence From First Principles. As usual, when someone says "First Principles" you bet they're not gonna start from First Principles... I guess this makes sense in CS since we index from 0 [0] https://arxiv.org/abs/2203.02155 [Christiano et al] https://arxiv.org/abs/1706.03741 [Stiennon eta al] https://arxiv.org/abs/2009.01325 [Akrour et al (2012)] https://arxiv.org/abs/1208.0984
Original comment Curious about the training data of OpenAI's new GPT-OSS models? I was too
jefftk
Neutral mention · rule
Reinforcement learning in general didn't come out of AI alignment work, but RHLF in particular did. The initial idea and paper [1] were from AI alignment folks, as was most of the later development [2][3][4][5]. Overview: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoug... [1] Deep Reinforcement Learning from Human Preferences https://arxiv.org/abs/1706.03741 [2] Fine-Tuning Language Models from Human Preferences https://arxiv.org/abs/1909.08593 [3] Learning to summarize from human feedback https://arxiv.org/abs/2009.01325 [4] Recursively Summarizing Books with Human Feedback https://arxiv.org/abs/2109.10862 [5] Training language models to follow instructions with human feedback https://arxiv.org/abs/2203.02155
Original comment Microsoft lays off an ethical AI team as it doubles down on OpenAI
sgillen
Neutral mention · rule
Thanks for your response, I had actually heard about Yuri before and was hoping you would respond to this post! I have a lot of questions for you if you don't mind, maybe you don't want to answer any/all of them that's fine. 1. How do you handle constraining the AI such that it's enjoyable? If you read through the comments here you'll notice just about everyone is saying the hard part is not making a good AI but in making a fun AI. Do you know of any research in this area? most examples I see are all about making the best AI possible. It seems like it's really hard to find a good reward signal to use for "fun". Have you seen this [0,1] paper on learning from human…
Read the full comment
preferences? does Yuri attempt to do something similar? It seems like it would be labor intensive for the play testers. 2. Does Yuri make use of imitation learning at all? I.E learning from lots of human data when it's available to bootstrap the learning process? 3. Do you let the game designers impose any structure on the AI? I.E. "I want three phases for the boss where he progressively gets more aggressive" or use RL just for certain small parts of the behavior? A well defined path finding algorithm with the RL deciding where the agent tries to go? stuff like that? I have a ton of questions about the technical details side of Yuri too, what frameworks are you using, which RL algorithms etc etc, not sure if you are willing to share those or not. [0] https://arxiv.org/abs/1706.03741 [1] https://blog.openai.com/deep-reinforcement-learning-from-hum...
Original comment Ask HN: Game devs, what is the hardest part about designing an AI for your game?
gwern
Neutral mention · rule
> Qualitative items such as the ones listed are important for customer focused environments, however, I'm not sure if AI can account for such factors. One of the nice things you can do with optimization problems is plug humans into the loop as oracles. Often, 'we know it when we see it', and we can do pairwise comparisons of 2 possibilities. So you can train a ML model based on win/loss comparisons and it'll learn to take into account the softer qualitative aspects via preference learning. A recent example you might remember from the press: "Deep reinforcement learning from human preferences" https://arxiv.org/abs/1706.03741 , Christiano et al 2017…
Read the full comment
https://deepmind.com/blog/learning-through-human-feedback/ https://blog.openai.com/deep-reinforcement-learning-from-hum... But also "Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces" https://arxiv.org/abs/1709.10163 , Warnell et al 2017. You can even pair it with EEG or brain scans for implicit ranking: "Towards personalized human AI interaction - adapting the behavior of AI agents using neural signatures of subjective interest" https://arxiv.org/abs/1709.04574 , Shih et al 2017.
Original comment Evolving Floorplans

Keep following the thread.

More in Machine learning

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score