LLaMA: Open and Efficient Foundation Language Models

Then, you should check out papers like https://arxiv.org/abs/2302.13971 and https://arxiv.org/abs/2307.09288 In the paper covering the original Llama they explicitly list their data sources in table 1 - including saying that they pretrained on the somewhat controversial books3 dataset. The paper for Llama 2 also explicitly says they don't take data from Meta's products and services; and that they filter out data from sites known to contain a lot of PII. Although it is more coy about precisely what data sources they used, like many such papers are.

The most detailed answer to that I've seen is the original LLaMA paper, which described exactly what that model was trained on (including lots of scraped copyrighted data) https://arxiv.org/abs/2302.13971 Llama 2 was much more opaque about the training data, presumably because they were already being sued at that point (by Sarah Silverman!) over the training data that went into the first Llama! A couple of things I've written about this: - https://simonwillison.net/2023/Aug/27/wordcamp-llms/#how-the... - https://simonwillison.net/2023/Apr/17/redpajama-data/

Because the training data/model size/compute tradeoff derived from that paper is highly suboptimal (too many parameters) compared to the ones from the later Deepmind scaling laws [1]. And then Meta researchers recommended using even smaller models, to trade-off training- and inference-time compute [2] (which I thought was pretty obvious if you care about more than just benchmarks). [1] https://arxiv.org/abs/2203.15556 Training Compute-Optimal Large Language Models [2] https://arxiv.org/abs/2302.13971 LLaMA: Open and Efficient Foundation Language Models

Llama: https://arxiv.org/pdf/2302.13971.pdf You can clearly see in table on second page that higher quality data is trained on more than 1 epoch. Most Open LLM's do this.

The Llama 1 paper [1] was one of the earlier models to question the assumption that more params = better model. Since then they've released Llama 2 and this post is offering more evidence that reinforces their hypothesis. I wouldn't say it was an oversight by other labs that they missed this. It's easier to just increase params on a model over the same training set instead of gathering a larger training set necessary for a smaller model. And at first, increasing model size did seem to be the way forward, but we've since hit diminishing returns. Now that we've hit that point, we've begun exploring other options and the Llamas are early evidence of another way forward. [1] https://arxiv.org/abs/2302.13971

It is very expensive to train these base models so a smaller size is more practical if you aren’t a big company with hundreds of powerful GPUs at hand. Table 15 from LLaMA paper[1] has some insightful figures: it took 135,168 GPU hours to train the 13B version and a bit more than 1M GPU hours for the 65B version. And we are talking about A100 80GB GPUs here (expensive and scarce). Not everyone can afford these kinds of trainings (especially if it takes a few attempts; e.g. if you’ve got a bug in the tokenizer) [1] https://arxiv.org/pdf/2302.13971.pdf

I don't think Facebooks moves were heavily strategically thought out. Instead, I think a small team of a few researchers made LLaMA and published a paper about it[1], and then realised to get any citations for their paper they'd need to let other researchers have the weights. Citations are like money in the academic world. Every researcher wants citations. Executives and managers care less about that stuff. [1]: https://arxiv.org/abs/2302.13971

First link: https://github.com/ggerganov/llama.cpp Which in turn has the following as the first link: https://arxiv.org/abs/2302.13971 Is it really quicker to ask here than just browse content for a bit, skimming some text or even using Google for one minute?

They are likely doing some interpolation for 200B or benchmarking it in wrong way. e.g. Hellaswag accuracy for llama 7b is 0.76[1], but it is written 0.56 in the repo. Even at 200B tokens, it is higher than 0.56 for llama looking at the charts. [1]: https://arxiv.org/pdf/2302.13971.pdf

LLama 65B used 2048 80G A100s for 21 days: > When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM.[1] Note that you probably need to budget for double to triple that because things go wrong and it usually takes multiple starts to get a good training run. Smaller models are cheaper though. [1] https://arxiv.org/pdf/2302.13971.pdf

Page 4 https://arxiv.org/abs/2302.13971 When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM. This means that training over our dataset containing 1.4T tokens takes approximately 21 days. At $4/GPU-hour per A100 80GB GPU, that's $4 * 2,048 * 21 * 24 = $4,128,768.

https://arxiv.org/pdf/2302.13971.pdf table 15. 1770394 A100-80GB hours to train the entire model suite at the going rate for cloud 8xA100-80GBs (~$12/hr if you could actually get capacity) is ~$2.6M, under extremely optimistic assumptions. YMMV on bulk pricing ;) "the more you buy the more you save"

> ChatGPT competitors of EU origin? ChatGPT is not research. If you want real research: https://ai.facebook.com/blog/large-language-model-llama-meta... Edit: end paper is https://arxiv.org/abs/2302.13971 > Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

It's worth pointing out that size isn't everything. From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals. Those interested in doing more comparisons can look at https://crfm.stanford.edu/helm/latest/?group=core_scenarios and https://paperswithcode.com/paper/llama-open-and-efficient-fo... to see where it sits (with some GPT 3.5 and 4 numbers here: https://paperswithcode.com/paper/gpt-4-technical-report-1) I'd agree the secret sauce for how great the newest services perform is probably in the fine-tuning. We're seeing almost daily releases of fine-tuning data sets, training methods and models (at lower and lower costs) so I'm personally pretty optimistic that we'll be seeing some big improvement in self-hosted LLM performance pretty quickly. [1] https://ar5iv.labs.arxiv.org/html/2302.13971#:~:text=Table%2....

Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0]. Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right? [0] https://arxiv.org/pdf/2302.13971.pdf