Neutral mention · rule
Most current "AI" models essentially compress and then store the information, often to the point where training data can be extracted [1][2] [1]: https://arxiv.org/abs/2012.07805 [2]: https://arxiv.org/abs/2301.13188
Back to the collection
Extracting Training Data from Large Language Models
Nicholas Carlini, Florian Tramer, Eric Wallace
Show all 12 authors
Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel
2020 · arXiv
- HN score
- 0
- Mentions
- 3
- Readers
- 3
- Months
- 3
From the abstract
It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
Neutral mention · rule
> Would it be OK if I launched a tech preview of my AI poem writer companion that would copy lines but also complete stanzas from famous poets, rock bands and singer-songwriters? Yes it would be if it only happened ~0.1% of the time and if quoting verbatim wasn't the intended function of the system but merely a side-effect. In fact, that's what artists sometimes do deliberately. It's what happens with other GANs as well and all that needs to happen is to educate users about the possibility of this. As long as you don't take ownership of the output produced by your AI (and neither do Microsoft), it's at the discretion of the user what they use the generated content for and in which context. It has been demonstrated that training data can be extracted from any large NLP model [0] so this wouldn't come as a surprise either. [0] https://arxiv.org/abs/2012.07805 https://towardsdatascience.com/openai-gpt-leaking-your-data-...
Neutral mention · rule
Yes, it has been shown [1] that these models can memorize information even if it only appears once in the training data. It could potentially cause issues with privacy and especially copyright. However, in this case it's not relevant because they didn't release the model weights. [1] https://arxiv.org/abs/2012.07805
Keep following the thread.
More in Language modelsMore from the same topics.
2023arXiv
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
60 mentions14 months
3.5HN score
2020arXiv
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
83 mentions35 months
3HN score
2025arXiv
Design Patterns for Securing LLM Agents against Prompt Injections
Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.
4 mentions4 months
2.5HN score
2023arXiv
Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.
24 mentions5 months
2HN score