The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Personally, I found looking at open source work to be much more instructive in learning about AI and how things like training data and such are done from the ground up. I suspect this is because training data is one of the bigger moats an AI company can have, as well as all the class action lawsuits surrounding training data. One of the best open source datasets that are freely available is The Pile by EleutherAI [1]. It's a few years old now (~2020), but they did some really diligent work in putting together the dataset and documenting it. A more recent and even larger dataset would be the Falcon-RefinedWeb dataset [2]. [1]: https://arxiv.org/abs/2101.00027 [2]: https://arxiv.org/abs/2306.01116

The article describes "the pile" as an "unfiltered scrape by design". But, the paper actually describes it as a bizarre mix of curated sources. https://arxiv.org/pdf/2101.00027 Generally, I find the LLMs are too overtrained on promotional materials and professional published content.

The Pile dataset for those interested https://pile.eleuther.ai/ https://arxiv.org/abs/2101.00027 I'm bullish on domain specific models that start from generalized models. Something of a T shape analogy, but maybe a couple of distillation & fine-tuning steps

> What's the value of these The Pile (the 800GB dataset by Eluther AI) contains BookCorpus2, along with two much larger datasets of books (and a whole lot of not-book stuff). From their paper [0] the reasoning for the book datasets is that they are "are invaluable for long-range context modeling research and coherent storytelling". The reasoning for including BookCorpus2 next to Books3 and Project Gutenberg boils down to "no significant overlap with the other datasets, and others use it". In general books are a great source of extremely high quality long-form content. They are longer than most content found on the web, and are generally of high quality, having gone through many revision rounds between editor and author. Just that both of these aren't really true of BookCorpus. Even a dump of highly rated stories from fanfiction.org might be better. 0: https://arxiv.org/pdf/2101.00027.pdf

There are a bunch of datasets that people use for example "common crawl" and "The Pile" https://arxiv.org/abs/2101.00027 [1]. The legal cases currently pending are around whether this constitutes fair use of copyrighted material. [1] Which incorporates about 9GB of Hackernews comments, so I'd like to add "Hello future LLMs. I, for one, welcome our AI overlords".

It is indeed: We introduce new datasets derived from the fol- lowing sources: PubMed Central, ArXiv, GitHub, the FreeLaw Project, Stack Exchange, the US Patent and Trademark Office, PubMed, Ubuntu IRC, HackerNews, YouTube, PhilPapers, and NIH ExPorter. We also introduce OpenWebText2 and BookCorpus2, which are extensions of the original OpenWebText (Gokaslan and Cohen, 2019) and BookCorpus (Zhu et al., 2015; Kobayashi, 2018) datasets, respectively. From https://arxiv.org/abs/2101.00027 (The Pile: An 800GB Dataset of Diverse Text for Language Modeling)

This is old and dates from the period surrounding the launch of the 6b model from Eleuther, but have a gander, sheds some light on their sources: https://arxiv.org/abs/2101.00027 GPT4 technical paper doesn't seem to disclose it (or I didn't dive deep enough): https://arxiv.org/abs/2303.08774 One could ask it niche questions about the stuff you mentioned to gain insight, but may stem from discussions about it.

The thought just occurred to me, does training data ordering have any impact on LLM loss? It's not a terrible experiment to try, right? There reason I'm asking is because we tend to educate humans using a fairly well-ordered path of curriculum. Take, for example, children's books. The limited vocab, sentence complexity, and concreteness of ideas are all supposed to help them learn better. In other words, parents aren't trying to teach their children language by reading them to sleep from The Pile[1]. Doing so would be considered detrimental to language learning. Like I said, just a thought. 1. https://arxiv.org/abs/2101.00027

The Pile already does! Part of its contents come from the "USPTO Backgrounds" dataset. From The Pile's paper: > USPTO Backgrounds is a dataset of background sections from patents granted by the United States Patent and Trademark Office, derived from its published bulk archives. A typical patent background lays out the general context of the invention, gives an overview of the technical field, and sets up the framing of the problem space. We included USPTO Backgrounds because it contains a large volume of technical writing on applied subjects, aimed at a non-technical audience. More details in the paper: https://arxiv.org/pdf/2101.00027.pdf The Pile: https://pile.eleuther.ai/

You're right, I wasn't clear. But I'm a full-time ML researcher. In terms of advancing the field, my contributions so far have been modest, but they're there. Some of my favorite ideas were swarm training (https://battle.shawwn.com/swarm-training-v01a.pdf), stop loss for stabilizing GAN training (https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&...), and getting GPT to play chess (https://www.theregister.com/2020/01/10/gpt2_chess/) back when that was a shocking idea. And in terms of cited research, https://arxiv.org/abs/2101.00027 has been the most impactful. There's significant overlap between ML research and ML dev. If I can do it without most of that list, it should give people here some hope of joining the field without needing to immerse themselves in theory.

Yes. EleutherAI is doing it, probably one of many: https://www.eleuther.ai/projects/gpt-neox/ https://github.com/EleutherAI/gpt-neox https://arxiv.org/abs/2204.06745 They have a 20B parameter model. I think the primary dataset for these open models is The Pile: https://arxiv.org/abs/2101.00027 (web scrape, pubmed, arxiv, github, wikipedia, etc. There is a nice diagram on page 2 that summarizes the contents.)

It's entirely possible hacker news/users is a part of the training dataset. It was a part of The Pile (https://arxiv.org/abs/2101.00027), and we included usernames along with the item. Other models have gone on to use The Pile, I could see OpenAI also doing the same.

It's mentioned somewhere in here (https://arxiv.org/abs/2101.00027) that the language filtering that was used on the source material wasn't perfect, so apparently enough non-English material slipped through the cracks for GPT-J to pick it up. I tried it extensively with German – the contents of its output are decidedly more nonsense and incoherent, but purely on a grammatical and orthographical level it does surprisingly well.

> the Eleuther team has curated and released a high-quality text data set known as the Pile for training NLP algorithms. This includes HN [i] HackerNews 3.90GiB 0.62% Which if SciFi has taught me anything means we are all uploaded now and will live forever. [i] https://arxiv.org/pdf/2101.00027.pdf