LIMA: Less Is More for Alignment

The LIMA paper, I think, would be a good place to start https://arxiv.org/abs/2305.11206 You can create inputs for DPO/ORPO synthetically which is a huge one as previously it would require gigantic investments https://arxiv.org/abs/2402.10379 There's also the gemma2 paper has advanced SOTA in distil; on a side-note, there's many reasons for it but vocab_size and good sizes 9b/27b, IMHO it's currently the best model for i.e. Ukrainian. in fact, I prefer it to anything else there's, including the much larger llama's—by a mile! The model is a triumph of synthetic datasets. https://arxiv.org/abs/2408.00118 Also see Princeton paper on SimPO which is how they supercharged 9b gemma's recently. https://arxiv.org/abs/2405.14734

I think you may be misunderstanding what fine tuning does. It does not teach the model new knowledge. In fact, Meta has a paper out that argues you only need a data set of 1000[1] to achieve pretty good alignment (fine-tuning) results. (100M is way overkill.) For knowledge retrieval, you need RAG (usually using the context window). [1] https://arxiv.org/pdf/2305.11206.pdf

LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463 Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models https://arxiv.org/abs/2307.14430

I don't think it's so cut and dry. The article paints a somewhat simplistic picture of how the best performing LLMs work. The unsupervised pre-trained networks are indeed data hungry, but the secondary supervised learning stages actually can get by with a far smaller set of highly curated prompt-response data, e.g. LIMA (https://arxiv.org/abs/2305.11206). Another factor is that generative data distributed online may be quite high quality (because people find it interesting enough to share), so it's plausible this could actually improve model performance. Some LLMs have been trained with data from other models with good results, e.g. supposedly Bard with GPT-4 prompt-response pairs and GPT-4 with Whisper transcripts of YouTube (https://twitter.com/amir/status/1641219919202361344/photo/1). Of course, there could be trolling or misinformation that "poisons" the data, and that is a problem (whether synthetic or organic)!

Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of fine-tuning: 3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base model] https://arxiv.org/abs/2305.11206 4. QLoRA: Efficient Finetuning of Quantized LLMs [showing that LLMs can now be fine-tuned quickly on consumer-grade GPUs] https://arxiv.org/abs/2305.14314 —- Adding up these developments (all of which occurred during the span of one week), I don’t see how huge, slow, general-purpose models maintain their relevance in the long term, when a lean, domain-focused model is right there within reach of every application developer.

People are literally doing this for free. You said that there is no individuals, and no non-profits, that can do this. Which is laughable. They're doing it right now: https://arxiv.org/abs/2305.11206 It may even be the case that all of that RLHF training that OpenAI does simply lessens the quality of generations, as suggested by one of their own papers and the paper above.

Quality matters just as much as quantity. LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463