Extracting Training Data from Diffusion Models

The tweet/paper co-author posted the paper (https://arxiv.org/abs/2301.13188) yesterday on HN (https://news.ycombinator.com/item?id=34596187) and ironically the top comment there is referencing to this exact tweet thread (which was posted yesterday as well). Evidence for the metacircular evaluation of HN comments? I think the paper is well worth the read, it's not particularly long (much is references and appendices), and nicely written, with at least a quick bit on most things I would think to test as part of something like this. Good stuff.

Then they should easily fall within the regulation section posted earlier. If you cannot see the difference between BitTorrent and Ai models, then it's probably not worth engaging with you. But Ai model have been shown to reproduce the training data https://gizmodo.com/ai-art-generators-ai-copyright-stable-di... https://arxiv.org/abs/2301.13188

If 1 in 1,000 generations will randomly produce memorized CSAM that slipped into the training set then yeah, it's pretty damn unsafe to use. Producing memorized images has precedent[0]. Is it unlikely? Sure, but worth validating. [0] https://arxiv.org/abs/2301.13188

Commercial musicians do not? You sure about that? Entire legal firm empires have been built on the licensing, negotiations, and fees that make up the industry. I'm ain't talking about some dude on YouTube or Soundcloud. Few people care about some rando on Soundcloud. Those moles aren't big enough to whack. Vanilla Ice and MC Hammer were. OpenAI is as well. There's even a company that specializes in sample clearance: https://sampleclearance.com More info: https://www.soundonsound.com/sound-advice/sample-clearance Also: >not turned into a part of a pattern used to generate new sounds like what AI generators do with images This is demonstrably false. Multiple individuals have repeatedly been able to extract original images from AI generators. Here's one-- Extracting Training Data from Diffusion Models https://arxiv.org/abs/2301.13188 Text, too: https://arxiv.org/abs/2311.17035

https://www.technologyreview.com/2023/02/03/1067786/ai-model... The approach I've seen is to prompt for people with unusual names, they're often be only a single source image in the input data set that gets reproduced by the AI. I've seen examples with the AI "generated" images and the source image side by side - I'll try and find them. EDIT: Link to the paper https://arxiv.org/abs/2301.13188 and this articles has the example images: https://www.theregister.com/2023/02/06/uh_oh_attackers_can_e... Look at the Ann Graham Lotz image and tell me that isn't the source image being reproduced in a lossy manner.

> Compression implies the input can be reconstructed from the output (lossy or not), in the case of these ml models the input is the training data and the output is the model. You can't reconstruct even a fraction of that training data using the model alone therefore it is not compression even in the most lossy sense. It's already proven that you can reconstruct at least a small fraction of the training set from diffusion models. It's something quite well known, so could we not die on this hill? [1] https://twitter.com/Eric_Wallace_/status/1620449942090420224 [2] The paper: https://arxiv.org/abs/2301.13188 [3] Relevant HN thread: https://news.ycombinator.com/item?id=34596187

Most current "AI" models essentially compress and then store the information, often to the point where training data can be extracted [1][2] [1]: https://arxiv.org/abs/2012.07805 [2]: https://arxiv.org/abs/2301.13188

My mistake! I mis-remembered the original Google paper[0], which stated only that the image was in the Stable Diffusion training set. The complete set of "matches" they got were interesting, even while the number was very low. 0. https://arxiv.org/abs/2301.13188

Every agent training the model on their proprietary data has to have access to the model form in some way (otherwise how would they train it?) For this reason, one must assume that the model form is known to the adversary. With this, the question becomes: is it possible to reconstruct training data from a trained model? We already know that, at least for some image models, the answer to that question is "yes": https://arxiv.org/pdf/2301.13188.pdf

IANAL either, but there are attacks you can use to extract the original training data from the models: https://arxiv.org/abs/2301.13188. I’d guess these attacks will get better with time. So in some ways, you can argue that Stability is also directly redistributing the original images (albeit in a compressed format).

Sometimes, you're just wrong. "Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images. In this work, we show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models, ranging from photographs of individual people to trademarked company logos. We also train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy. Overall, our results show that diffusion models are much less private than prior generative models such as GANs, and that mitigating these vulnerabilities may require new advances in privacy-preserving training. " https://arxiv.org/abs/2301.13188