GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Can anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model? I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it. [0] https://github.com/turboderp/exllamav2#exl2-quantization [1] https://arxiv.org/abs/2210.17323

> LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though) From the GPTQ paper https://arxiv.org/abs/2210.17323: "... with negligible accuracy degradation relative to the uncompressed baseline"

Training uses gradient descent, so you want to have good precision during that process. But once you have the overall structure of the network, https://arxiv.org/abs/2210.17323 (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy. It seems you can cut down further for larger models. For the 13B Llama-based ones, going below 5 bits per parameter is noticeably worse, but for 30B models you can do 4 bits. The same group did another paper https://arxiv.org/abs/2301.00774 which shows that in addition to reducing the precision of each parameter, you can also prune out a bunch of parameters entirely. It's harder to apply this optimization because models are usually loaded into RAM densely, but I hope someone figures out how to do it for popular models.

Define "comprehensive?" There are some benchmarks here: https://www.reddit.com/r/LocalLLaMA/comments/1248183/i_am_cu... and here: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i... Check out the original paper on quantization, which has some benchmarks: https://arxiv.org/pdf/2210.17323.pdf and this paper, which also has benchmarks and explains how they determined that 4-bit quantization is optimal compared to 3-bit: https://arxiv.org/pdf/2212.09720.pdf I also think the discussion of that second paper here is interesting, though it doesn't have its own benchmarks: https://github.com/oobabooga/text-generation-webui/issues/17...

For this specific implementation here's info from llama.cpp repo: Perplexity - model options 5.5985 - 13B, q4_0 5.9565 - 7B, f16 6.3001 - 7B, q4_1 6.5949 - 7B, q4_0 6.5995 - 7B, q4_0, --memory_f16 According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2]. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa [2] https://arxiv.org/abs/2210.17323

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as they run inference, but they clearly know more than me about the internals), bringing it within reach of phone memory. So, iPhones are not as far off the computational capacity to run these models as you'd think. Memory (and to a greater extent, battery and cooling) are the limiting factors. iPads even less so, given they run M1 chips and have much larger batteries & much more RAM https://arxiv.org/abs/2210.17323

> https://arxiv.org/abs/2210.17323 I've read the paper and to be honest I'm not sure what to make of it. Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users. If you look at the tables in the appendix A.4 with some more relevant benchmarks you'll sometimes find that straight RTN 4 bit quantisation beats both GPTQ and even full 16 bit original! No explanation of it is given in the paper.

As far as I know, yes. https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the model. It makes loading it in ram on consumer computers feasible. The number of parameters stays the same.

They're using GTPQ -- here you go: https://arxiv.org/abs/2210.17323 . The authors benchmarked two families of models over a wide range of numbers of params.

See: https://arxiv.org/abs/2210.17323 Q: Doesn't 4bit have worsen output performance than 8bit or 16bit? A: GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16. https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png

WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization. You cannot tell the difference and if you think you do you're wrong, because it barely even registers on any benchmark. (Note: llama.cpp's 4bit is naive, not GPTQ, and sucks but they are refactoring it to use GPTQ quantization) References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022] https://github.com/ggerganov/llama.cpp/issues/9 - llama.cpp: GPTQ Quantization (3-bit and 4-bit) #9 https://github.com/qwopqwop200/GPTQ-for-LLaMa/ - 4 bits quantization of LLaMa using GPTQ

Decrease in accuracy is negligible and decreases as model size increases. That is, larger models quantize even better than smaller models. https://arxiv.org/abs/2210.17323

From what I can tell, the quantization being used there is extremely naive. If they were using techniques like GPTQ, presumably it would work much better. https://arxiv.org/pdf/2210.17323.pdf

When you click on the strip link to preorder the tinybox, it is advertised as a box running LLaMA 65B FP16 for $15000. To be fair, the previous page has a bit more details on the hardware. I can run LLaMA 65B GPTQ4b on my $2300 PC (built from used parts, 128GB RAM, Dual RTX 3090 @ PCIe 4.0x8 + NVLink), and according to the GPTQ paper(§) the quality of the model will not suffer much at all by the quantization. Just saying, open source is squeezing an amazing amount of LLM goodness out of commodity hardware. (§) https://arxiv.org/abs/2210.17323

Yes, but only minimally. Not enough for any human to notice. However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead. References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]

Relevant: Since LLaMA leaked on torrent, it has been converted to Huggingface weights and it has been quantisized to 8bit for less vram requirements. A few days ago it has also been quantisized to 4bit and 3bit is coming. The quantization method they use is from the GPTQ paper ( https://arxiv.org/abs/2210.17323 ) which leads to almost no quality degradation compared to the 16bit weights. 4 bit weights: Model, weight size, vram req. LLaMA-7B, 3.5GB, 6GB LLaMA-13B, 6.5GB, 10GB LLaMA-30B, 15.8GB, 20GB LLaMA-65B, 31.2GB, 40GB Here is a good overall guide for Linux and Windows: https://rentry.org/llama-tard-v2#bonus-4-4bit-llama-basic-se... I also wrote a guide how to get the bitsandbytes library working on windows: https://github.com/oobabooga/text-generation-webui/issues/14...

text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1] LLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization. LLaMA-30B fits on a 24GB* consumer video card with no output performance loss, beating GPT-3 175B. Multi-GPU support[2] means LLaMA-65B, rivaling PaLM-540B, runs on 2x3090. *Further improvements in active development will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention). [0] https://github.com/qwopqwop200/GPTQ-for-LLaMa [1] https://arxiv.org/abs/2210.17323 [2] https://github.com/oobabooga/text-generation-webui/issues/14... For LLaMA set-up instructions, including GPTQ 4bit, refer to this wiki article: https://github.com/oobabooga/text-generation-webui/wiki/LLaM...