Yizhong Wang, Yeganeh Kordi, Swaroop Mishra
Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi
2022 · arXiv
Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
I think most of the instruction fine-tuning methods for oss models stem from Alpaca, so it should be included: https://crfm.stanford.edu/2023/03/13/alpaca.html And the one referenced in there on synthetic data generation: https://arxiv.org/abs/2212.10560
It has existed for years: Self-Instruct: Aligning Language Models with Self-Generated Instructions https://arxiv.org/abs/2212.10560 airoboros: using large language models to fine-tune large language models https://github.com/jondurbin/airoboros
Since nobody is actually recommending papers, here's an incomplete reading list that I sent out to some masters students I work with so they can understand the current research (academic) my little team is doing: Paper reference / main takeaways / link instructGPT / main concepts of instruction tuning / https://proceedings.neurips.cc/paper_files/paper/2022/hash/b... self-instruct / bootstrap off models own generations / https://arxiv.org/pdf/2212.10560.pdf Alpaca / how alpaca was trained / https://crfm.stanford.edu/2023/03/13/alpaca.html Llama 2 / probably the best chat model we can train on, focus on training method. / https://arxiv.org/abs/2307.09288 LongAlpaca / One of…
many ways to extend context, and a useful dataset / https://arxiv.org/abs/2309.12307 PPO / important training method / idk just watch a youtube video Obviously these are specific to my work and are out of date by ~3-4 months but I think they do capture the spirit of "how do we train LLMs on a single GPU and no annotation team" and are frequently referenced simply by what I put in the "paper reference" column.
There have been other approaches to make instruction tuning easier. This one proceeds largely on self generated synthetic data: https://arxiv.org/abs/2212.10560
For those skeptical of the above comment, this technique absolutely works and powers production-grade models like Anthropic’s Claude. There’s plenty of literature on this, but here are a couple papers that might be helpful for people doing their own training: - Constitutional AI: by Anthropic, an “RLAIF” technique that creates the preference model for “finding errors” based on a set of around 70 “principles” the AI uses to check its own output, not human feedback like in ChatGPT. This technique taught the Claude bot to avoid harmful output with few to no manual harmfulness labels! https://arxiv.org/abs/2212.08073. Not sure if there’s a HuggingFace implementation with…
LoRA / PEFT yet like there is for regular RLHF, so somebody may need to implement this for Llama still - Self-Instruct: Creates artificial training data on instruction tuning from an untuned base model, from a tiny seed of prompts, and filters out the bad ones before fine-tuning. Manages to approach Instruct-GPT performance with only ~100 human labels. https://arxiv.org/abs/2212.10560
Alpaca uses Self-Instruct[0] which is better than just the pre-training but I wouldn't expect it to be at the level of ChatGPT (RLHF) in terms of human-friendly prompting. OpenAssistant should make it close to ChatGPT (from GPT-3.5 version) if the LLaMA is as powerful as claimed. [0]: https://arxiv.org/abs/2212.10560
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.