Language Modeling Is Compression

This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2

> Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms. gzip can be used as a (not very good) LLM-like text and image generator: https://arxiv.org/abs/2309.10668

Well, they couldn't generate 2D artwork if they weren't capable of working with multiple output dimensions. An interesting thing about transformers is that they are world-class at compressing 2D image data even when not trained on anything but textual language ( https://arxiv.org/abs/2309.10668 ). Whether that notion is relevant for 3D content would be two or three figures over my pay grade, though.

Language modeling is compression, whittle down graph to reduce duplication and data with little relationship: https://arxiv.org/abs/2309.10668 Let’s say everyone agrees to refer to one hosted copy of a token “cat”, and instead generate a unique vector to represent their reference to “cat”. Blam. Endless unique vectors which are nice and precise for parsing. No endless copies of arbitrary text like “cat”. Now make that your globally distributed data base to bootstrap AI chips from. The data driven programming dream where other machines on the network feed new machines boot strap. American tech industry is IBM now. Stuck on recent success of web SaaS and way behind the plans of AI.

Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668 So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable. It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.

Wasn't there that thing about how large LLM's are essentially compression algorithms (https://arxiv.org/pdf/2309.10668)? Maybe that's where this article is coming from, is the idea that finetuning "adds" data to the set of data that compresses well. But that indeed doesn't work unless you mix in the finetuning data with the original training corpus of the base model. I think the article is wrong though in saying it "replaces" the data - it's true that finetuning without keeping in the original training corpus increases loss on the original data, but "large" in LLM really is large and current models are not trained to saturation so there is plenty of room to fit in finetuning if you do it right.

Yeh. Data driven model syncing machines will be what kills languages, software stacks. All the chip world talk of single function machines and how tech is now energy constrained industry, means pruning the state we store; most software is software to deliver software. Single function pipeline hardware platforms will act as little more than sync engines from models. I mean it’s is 2025. Still write software like it’s the 1970s. So high tech. Edit: https://arxiv.org/abs/2309.10668 From LLMs to energy models that transform electromagnetic geometry. Saving what’s needed to recreate cat pics and emails to mom. Pruning the tail as interest dips with generational churn.

This already happened with Linux where people with no understanding of electrical engineering got jobs memorizing abstractions. I look forward to a data driven system future where a few functions transform the machines electromagnetic geometry to solve a task based upon the most efficient energy model for solving a task as we continue to compress from the model all the non-essential syntax sugar of modern software. https://arxiv.org/abs/2309.10668

Oh well, you should tell the folks at DeepMind and Meta about these objective facts then so they don’t waste any more time doing research: https://arxiv.org/html/2309.10668v2 Maybe apply for a job there too, since you’re obviously so far ahead of everyone in understanding this problem space.

This is actually an excellent question ! And quite an active field of research. This is one paper you could read - it's about text and not films, but same idea: "LANGUAGE MODELING IS COMPRESSION" - https://arxiv.org/pdf/2309.10668

Vapnik: https://www.cs.princeton.edu/courses/archive/spring13/cos511... https://engineering.columbia.edu/files/engineering/vapnik.pd... https://www.learningtheory.org/learning-has-just-started-an-... https://nautil.us/teaching-me-softly-234576/ The main paper: https://gwern.net/doc/reinforcement-learning/exploration/act... It sounds kinda crazy (is there really that much far transfer?), but you know, I think it would work... He just needed to use LLMs instead: https://arxiv.org/abs/2309.10668#deepmind

All we’re doing is engineering new data compression and retrieval techniques: https://arxiv.org/abs/2309.10668 Are we sure there’s anything “net new” to find within the same old x86 machines, within the same old axiomatic systems of the past? Math is a few operations applied to carving up stuff and we believe we can do that infinitely in theory. So “all math that abides our axiomatic underpinnings” is valid regardless if we “prove it” or not. Physical space we can exist in, a middle ground of reality we evolved just so to exist in, seems to be finite; I can’t just up and move to Titan or Mars. So our computers are coupled to the same constraints of observation and understanding as us. What about daily life will be upended reconfirming decades old experiment? How is this not living in sunk cost fallacy? When all you have is a hammer… I’m reminded of Einstein’s quote about insanity.

Yes, next token prediction is exactly what you need for compression. Many papers on this, recent one https://arxiv.org/pdf/2309.10668.pdf

Yes! Prompts are super finicky. You have to create a prompt/function that for a wide set of inputs, generates a token sequence that will perpetually expand in a manner that corresponds to an externally observed truth. Way too often it feels like you have to shove a universal decoding sequence into a prompt. “Talk your steps, list your clues, etc.” Just trying to luck into a prompt that keeps decompressing the model/ generating the next token that ensures the next token is true.* I recall there was a paper with a relevant title recently… https://arxiv.org/abs/2309.10668 Basically - LLMs don’t reason , they regurgitate. If they have the right training data, and the right prompt, they can decompress the training data into something that can be validated as true ——- * Also this has to be done in a limited context window, there is no long term memory, and there is no real underlying model of thought.