Skip to content
The methodOpen data
Back to the collection

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

Guilherme Penedo, Quentin Malartic, Daniel Hesslow

Show all 9 authors

Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay

2023 · arXiv

Language modelsMachine learning
Read the paper
HN score
1.5
Mentions
4
Readers
3
Months
4
What’s in a score?

From the abstract

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

IshanMi
Strong recommendation · rule
Personally, I found looking at open source work to be much more instructive in learning about AI and how things like training data and such are done from the ground up. I suspect this is because training data is one of the bigger moats an AI company can have, as well as all the class action lawsuits surrounding training data. One of the best open source datasets that are freely available is The Pile by EleutherAI [1]. It's a few years old now (~2020), but they did some really diligent work in putting together the dataset and documenting it. A more recent and even larger dataset would be the Falcon-RefinedWeb dataset [2]. [1]: https://arxiv.org/abs/2101.00027 [2]: https://arxiv.org/abs/2306.01116
Original comment DBRX: A new open LLM
weinzierl
Neutral mention · rule
Similarly HN is already an important data source for large language model training. For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded. From what I understand, it was excluded because it went into the final dataset unvetted. RefinedWeb was used for the Falcon model. https://arxiv.org/pdf/2306.01116.pdf
Original comment A steep rise of Hacker News in Google rankings
weinzierl
Neutral mention · rule
I appreciate that attitude and value it myself, but I like to point out that it is not without risk. If the world around HN (including its community) changes, stasis can damage or kill it as well. Specifically regarding the issue of the original posting: - HN is already an important data source for large language model training. [1] - To the best of my knowledge there is no freely downloadable and current data dump of HN. [2] - The HN-API does not offer all the data that scraping can get. For example, if a post had ever hit the front page or the highest front page position reached, is an interesting data point that is missing. - The Algolia-HN-API has the same…
Read the full comment
limitations. In my opinion this will lead to increased usage of the API and increased scraping which all costs money. HN might be forced to find a solution for this. [1] For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded. From what I understand, it was excluded because it went into the final dataset unvetted. RefinedWeb was used for the Falcon model. https://arxiv.org/pdf/2306.01116.pdf [2] The closest thing is probably the Google BigQuery "bigquery-public-data.hacker_news" dataset. It claims to be updated daily, but really is from late September 2022. Also I could not find the download link which other data sets offered on BigQuery have. Does anyone know if I can download the complete thing anyhow? https://console.cloud.google.com/bigquery?p=bigquery-public-...
Original comment Twitter now requires an account to view tweets

Keep following the thread.

More in Language models

More from the same topics.