Skip to content
The methodOpen data
Back to the collection

Quantifying Memorization Across Neural Language Models

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski

Show all 6 authors

Katherine Lee, Florian Tramer, Chiyuan Zhang

2022 · arXiv

Language modelsSecurity & cryptography
Read the paper
HN score
0
Mentions
6
Readers
6
Months
6
What’s in a score?

From the abstract

Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

johndough
Neutral mention · rule
I am not arguing that Kimi did not distill Opus or Sonnet (which they almost certainly have, as opposed to Fable or Sol). I am arguing that even if they did not distill, the model could still identify itself as Opus or Sonnet. That being said, I'd like to point out a few things: - The second link (to claude-sonnet-4-2025051) had 17k matches, not just 3k. - grep.app does not index the entirety of GitHub, so the real number of occurrences of model identifiers is even higher. SourceGraph has a larger index, but the number is so high that it exceeds their result limit of 10k: https://sourcegraph.com/search?q=claude-opus-4-5-20251101&pa... - 1000 occurrences are already plenty…
Read the full comment
for memorization. Here is a paper that shows over 80% "extractability" with 1000 occurrences for a 6B model. It also shows that extractability scales with model size: https://arxiv.org/pdf/2202.07646 - LLMs have a "mosaic memory", which means that they can combine information from different parts of the training data. https://arxiv.org/pdf/2405.15523 For example, one piece of training data might mention "I am Opus", another might mention "Opus (claude-opus-4-5-20251101)" and another 10000 "claude-opus-4-5-20251101", where the later occurrences strengthen the generation probability of "I am Opus". - RL can change the model identity easily, so all of the above may not matter at all.
Original comment The Kimi K3 Moment
mwcampbell
Neutral mention · rule
I'm not making an argument from grievance about my own code being plagiarized. I actually don't care if my own code is used without even the attribution required by the permissive licenses it's released under; I just want it to be used. I do also write proprietary code, but that's not in the training datasets, as far as I know. But the training datasets do include code under a variety of open-source licenses, both permissive and copyleft, and some of those developers do care how their code is used. We should respect that. As for our tendency to disrespect the copyrights of art, clearly we've always been in the wrong about this, and we should respect the rights of artists.…
Read the full comment
The fact that we've been in the wrong about this doesn't mean we should redouble the offense by also plagiarizing from other programmers. And there is evidence that LLMs do plagiarize when generating code. I'll just list the most relevant citations from Baldur Bjarnason's book _The Intelligence Illusion_ (https://illusion.baldurbjarnason.com/), without quoting from that copyrighted work. https://arxiv.org/abs/2202.07646 https://dl.acm.org/doi/10.1145/3447548.3467198 https://papers.nips.cc/paper/2020/hash/1e14bfe2714193e7af5ab...
Original comment Migrating away from Rust
1vuio0pswjnm7
Neutral mention · rule
"Can anyone find the filing?" https://arxiv.org/pdf/2202.07646 Personally, I would not rely on blogs like "developer-tech.com" for unbiased information about "AI" litigation. I would read the order and draw own conclusions.^1 (Note the plaintiffs are attempting to file an interloctory appeal re: the dimissal of the DMCA claims.) 1 https://ia904708.us.archive.org/6/items/gov.uscourts.cand.40... No doubt I am in the minority but I am more interested in the contract claims, which have survived dismissal, than in the DMCA ones. If plaintiffs can stop the "training" of "AI" through contract, then in theory DMCA violations can be avoided.^2 2 For example, license terms which specifically prohibit using the licensed source code to train language models. Is there a "fair use" defense to contract liability.^3 3 https://cdn.ca9.uscourts.gov/datastore/opinions/2006/05/16/0...
Original comment Judge dismisses majority of GitHub Copilot copyright claims
versteegen
Neutral mention · rule
> They just show it to the AI, it doesn't store it - just data about it. The same as you when you look at something, be it a painting or a chair. In general for machine learning, yes, but deep learning models with large numbers of parameters such as transformers and stable diffusion do actually memorise part of their training data verbatim. The problem is mainly duplicate data points. E.g. https://arstechnica.com/information-technology/2023/02/resea... https://arxiv.org/pdf/2202.07646.pdf Quantifying Memorization Across Neural Language Models
Original comment PG: We can't all use AI. Someone has to generate the training data
ijk
Neutral mention · rule
You are probably very aware of it, but just to highlight the importance of this for people who aren't aware: data duplication degrades the training and makes memorization (and therefore plagiarism, in the technical sense) more likely. For language models, this includes near-similarities, which I'd guess would extend to images. Quantifying Memorization Across Neural Language Models https://arxiv.org/abs/2202.07646 Deduplicating Training Data Makes Language Models Better https://arxiv.org/abs/2107.06499 https://twitter.com/arankomatsuzaki/status/14154721921003397... https://twitter.com/katherine1ee/status/1415496898241339400
Original comment Exploring 12M of the 2.3B images used to train Stable Diffusion

Keep following the thread.

More in Language models

More from the same topics.