We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/google-research/deduplicate-text-datasets.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
There are benchmarks that are decided beforehand and similar sentences are removed from even the first stage of training. This is useful for tracking model performance and comparing different choices. e.g. see section 'Contamination of downstream tasks' of [1].
Every decent AI lab does this, else the benchmark result couldn't be trusted. OpenAI publishes results of ~20 benchmarks[2] and it is safe to assume they have made reasonable attempt to remove it from training set
[1]: https://arxiv.org/pdf/2107.06499
[2]: https://openai.com/index/hello-gpt-4o/
The OP’s paper reads:
> Document Deduplication … We use the MinHash algorithm to compute 13-gram Jaccard similarities to determine which documents are near-duplicates of each other (Lee et al., 2021a). … We define two documents to be too similar when their Jaccard similarity exceeds 0.8, and randomly remove one of them.
The inline referenced [Lee et al](https://arxiv.org/pdf/2107.06499) goes into detail, and notes that one of the algorithms is based on:
> 4.2 Approximate Matching with MinHash … perform approximate deduplication based on matching entire examples. This method, which we call NEARDUP, is a good complement to the exact substring matching, especially for web…
Read the full comment
crawl text, as it handles the very common case of docu- ments being identical except for interspersed templated fields.
I’ve searched to my Peter Principle limit here, and was able to find [another implementation of MinHash](https://web.eecs.utk.edu/~jplank/plank/classes/cs494/494/not...) that used the first eight bytes of the hash, and so I am just going to assume that’s what they needed in order to meet their similarity KPI. Hopefully somebody here knows better/is smarter.
Deduplicating Training Data Makes Language Models Better
https://arxiv.org/pdf/2107.06499
Quantifying Memorization Across Neural Language Models
https://arxiv.org/pdf/2202.07646.pdf
You are probably very aware of it, but just to highlight the importance of this for people who aren't aware: data duplication degrades the training and makes memorization (and therefore plagiarism, in the technical sense) more likely. For language models, this includes near-similarities, which I'd guess would extend to images.
Quantifying Memorization Across Neural Language Models
https://arxiv.org/abs/2202.07646
Deduplicating Training Data Makes Language Models Better
https://arxiv.org/abs/2107.06499
https://twitter.com/arankomatsuzaki/status/14154721921003397...
https://twitter.com/katherine1ee/status/1415496898241339400
Not sure which of the two papers you’re referring to. The Anthropic paper [1] clearly shows an example where not deduping leads to serious quality degradation.
However it’s not clear if training for more than one epoch on deduped and well balanced data would help. I personally think it should and the reason people don’t do it might be because it’s too expensive.
[1] https://arxiv.org/pdf/2107.06499.pdf
The controversy here is specifically around Stochastic Parrots, by among others Timnit Gebru and "Schmargaret Smitchell". Timnit had recently been fired by Google for a controversy that started when leadership prevented her and Margaret from being coauthors on the paper (which is critical of Google in some ways) and Mitchell was fired a few months later.
Google[0] claims the paper didn't meet the bar for publication and ignored too much relevant research. The authors obviously dispute these claims (I generally side with the authors that the "ignored research" line is a stretch).
The then amusing thing is that a number of Google authored or coauthored papers still cite it,…
Read the full comment
for various reasons[1][2] (but there's another like half dozen or so Google papers at least in the ~500 citations). So the entire controversy is that the paper wasn't up to academic standards, except that empirically it clearly was.
> Were they reworked at all after being told they weren't up to standard?
There was an attempt by the authors to respond to the internal criticism, but this wasn't met constructively.
[0]: https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQ...
[1]: https://arxiv.org/pdf/2107.06499.pdf
[2]: https://arxiv.org/pdf/2109.01652.pdf
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.