Milad Nasr, Nicholas Carlini, Jonathan Hayase
Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, Katherine Lee
2023 · arXiv
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and emit training data at a rate 150x higher than when behaving properly. Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
> Like if there is no way to trace it back to the original material, does it make sense to regulate it? Training data extraction has seen some success, tracing should be possible for at least some of it https://arxiv.org/abs/2311.17035
Yes. https://arxiv.org/abs/2403.06634 https://arxiv.org/abs/2311.17035 (I just have these ones off the top of my head because I'm a Nicholas Carlini fan and we interviewed him about these attacks.)
I had the exact same idea after seeing Google point out that you can[0] get ChatGPT to regurgitate verbatim training data by asking it to repeat the same word over and over again[1]. I'm glad to see someone else actually bring it to fruition. This, of course, brings two additional questions: 1. Is this "AI, hold the AI" approach more energy-efficient than having gradient descent backpropagation compress a bunch of training data into a model that can then be run on specialized AI coprocessors? 2. Will this result wind up being evidence in the ongoing lawsuits against OpenAI and Stability AI? [0] Could. OpenAI now blocks generation if you fill the context window with a single word. [1] https://arxiv.org/abs/2311.17035
Commercial musicians do not? You sure about that? Entire legal firm empires have been built on the licensing, negotiations, and fees that make up the industry. I'm ain't talking about some dude on YouTube or Soundcloud. Few people care about some rando on Soundcloud. Those moles aren't big enough to whack. Vanilla Ice and MC Hammer were. OpenAI is as well. There's even a company that specializes in sample clearance: https://sampleclearance.com More info: https://www.soundonsound.com/sound-advice/sample-clearance Also: >not turned into a part of a pattern used to generate new sounds like what AI generators do with images This is demonstrably false. Multiple individuals have repeatedly been able to extract original images from AI generators. Here's one-- Extracting Training Data from Diffusion Models https://arxiv.org/abs/2301.13188 Text, too: https://arxiv.org/abs/2311.17035
Sorry for any confusion, my comment was meant to refer to the linked paper: https://arxiv.org/abs/2311.17035 I think it’s valid work, but the original tweet seems to call a prompt based paper into question. At least enough to clarify where he would stand on an example like this.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.