Textbooks Are All You Need

While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ``` From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. For those interested in reading more about various considerations for coding models, I highly recommend reading the MSR phi-1 paper: https://arxiv.org/pdf/2306.11644.pdf Looking forward to if they ever publish code/model/dataset since it has extremely strong performance trained on a very small number of tokens very manageable 1.3B and 350M parameter models.

Amusingly enough, someone from a frontier lab could probably answer this empirically. Years ago Microsoft demonstrated training LLMs on synthetic text-- books rewritten by LLMs to be more concise and more accurate. https://arxiv.org/abs/2306.11644 It is well known that Anthropic extensively uses these synthetic text in training Claude. You could probably get good 50tile, 90tile, 99 etc numbers just from the size of the training materials on Anthropic servers.

Phi was pretty revolutionary for its time. The accompanying paper https://arxiv.org/abs/2306.11644 has been pretty instrumental in pushing the idea of highly curated synthetic dataset to train model. The lead scientist behind Phi-1 is now VP at OpenAI and leading their own synthetic training dataset effort.

You are probably right that there is massive disparity between the two re: compute. With that said, it is conceivably possible to use a larger number of weaker chips rather than fewer bigger ones, and come out ahead per unit-time. Also, given their strategy seems to be trying to do as much on-device as possible, they are targeting smaller models, so they likely have less of a packing problem with smaller chips. So in the goal of producing a strong model, it could go either way; especially for smaller models, data seems to be much more important than compute, as per the “Textbooks are all you need” paper (https://arxiv.org/abs/2306.11644), especially if you are not looking for a fully generalist model.

>Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. Great stuff. >More broadly, I agree with your sentiment that there is a lot of value in considering the best ways to structure the data we share with LLMs. Especially in the context of coding. As the experiments on PHI-1 and PHI-2 from microsoft show, training data make a difference. The "textbooks is all you need" moto means better structured data, more clear data make a difference. https://arxiv.org/abs/2306.11644

GPT is better than an average human at coding. GPT is worse than an average human at recognizing bounds of its knowledge (i.e. it doesn't know that it doesn't know). Is it fundamental? I don't think so. GPT was trained largely on random internet crap. One of popular datasets is literally called The Pile. If you just use The Pile as a training dataset, AI will learn very little reasoning, but it will learn to make some plausible shit up, because that's the training objective. Literally. It's trained to guess the Pile. Is that the only way to train an AI? No. E.g. check "Textbooks Are All You Need" paper: https://arxiv.org/abs/2306.11644 A small model trained on high-quality dataset can beat much bigger models at code generation. So why are you so eager to use a low-quality AI trained on crap? Can't you wait few years until they develop better products?

I don’t think it’s as much about day dreaming but more so about garbage in, garbage out. We know LLMs perform way better when you use high quality input data [0]. Having your input data be skewed towards one genre isn’t great. And thats not even getting into the quality of these books. [0] https://arxiv.org/abs/2306.11644

>> Wrong in what context Read above. I wrote what exactly is wrong with it. >> There's no no-context objective version here. Exactly. And that is why such texts must be excluded. https://arxiv.org/abs/2306.11644

Wasnt there a paper a few months back, Textbooks are all you need. yes found it https://arxiv.org/abs/2306.11644 So search engines in their traditional sense will be obsolete anyway. 1) GPT-4 and other such LLMs will generate textbooks and manuals for every conceivable topic. 2) These textbooks will be 'dehallucinated' and curated by known experts on particular topics, who have reputations to maintain. The experts' names will be advertised by the LLM provider. 3) People will search for stuff by chatting with the LLMs, which will in turn provide citations for the chat output from the curated textbooks.

Sounds vaguely like the paper “textbooks are all you need”? Though they are not explicitly trying to remove the copyright taint. https://arxiv.org/abs/2306.11644

phi-1[0] is only 1.3 billion parameters and performs very well in coding tasks - small models have a massive amount of potential [0] - https://arxiv.org/abs/2306.11644

> The 24GB of VRAM should keep it relevant for a bit too If anything, I think models are going to shrink a bit, because assumptions around small models reaching capacity during traiing don’t seem fully accurate in practice[0]. We’re already starting to see some effects, like Phi-1[1] (a 1.3B code model outperforming 15B+ models), and BTLM-3B-8K[2] (a 3B model outperforming 7B models) [0]: https://espadrine.github.io/blog/posts/chinchilla-s-death.ht... [1]: https://arxiv.org/pdf/2306.11644.pdf [2]: https://www.cerebras.net/blog/btlm-3b-8k-7b-performance-in-a...

You are totally right - both more and better matters. There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind - https://arxiv.org/abs/2306.11644

> even fine-tuning using AI causes the models to quickly degrade Really? My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results, but happy to see some counter examples 0: https://arxiv.org/abs/2305.07759 1: https://arxiv.org/abs/2306.11644

The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models. https://arxiv.org/abs/2306.11644

Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder) [1] https://arxiv.org/abs/2306.11644