Skip to content
The methodOpen data
Back to the collection

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer

2022 · arXiv

Language models
Read the paper
HN score
0
Mentions
6
Readers
6
Months
5
What’s in a score?

From the abstract

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

make3
Neutral mention · rule
There are a very large number of quantization schemes in existence, definitely not just one, & they all have potentially very different ideas and schemes. LLM8 was introduced before https://arxiv.org/abs/2208.07339 of the same first and last author as QLora & is still what can be used in Huggingface Transformers with the `load_in_8bits` parameter. The idea was just to quantize all weights to 8 bits except a few outliers, which are kept in original precision. This scheme kept the computations extremely accurate, and was really fast to do. I haven't read the new paper, but I assume they came up with a more advanced fast distributional setup.
Original comment QLoRA: Efficient Finetuning of Quantized LLMs
chuckcode
Neutral mention · rule
Thanks for details! Few follow up questions: - I've seen neural nets using int8 for matrix multiplication to reduce memory size [1]. Do you think something similar could be useful in the ANN space? - Do you know of any studies using Faiss looking at speed/cost tradeoffs of RAM vs flash vs Disk for storage? - Are there recommended ways to update Faiss index with streaming data, e.g. updating the vectors continuously? Seems like more and more use cases for Faiss as neural nets become more and more core to workflows. Would like to try and figure out the configurations that are optimized to minimize carbon usage in addition to latency and recall metrics. [1] https://arxiv.org/abs/2208.07339
Original comment Billion-Scale Approximate Nearest Neighbor Search [pdf]
v64
Neutral mention · rule
If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software." I'm very thankful we have researchers like this further democratizing access to this data and prying it out of the hands of the gatekeepers who wish to monetize it. [1] https://github.com/tloen/llama-int8 [2] https://arxiv.org/abs/2208.07339
Original comment Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
eternalban
Neutral mention · rule
Somewhat amazed, dang, that this topic is not discussed more widely here or elsewhere. There is a lot of HPC and DS expertise out there which lacks understanding of ML system architecture (in the sense of the deployed machinery in toto). Her follow up post [1] is also recommended for those who (like me, are experienced but not in ML) finally had things click because of the OP writeup: Large Transformer Model Inference Optimization (2023) https://lilianweng.github.io/posts/2023-01-10-inference-opti... A very cool cite from that article is LLM.int8(): https://arxiv.org/abs/2208.07339
Original comment How to Train Large Models on Many GPUs?
monkmartinez
Neutral mention · rule
Full disclosure; I am not a pro ML'er. I dabble as a hobby and I am trying to learn as much as possible. Question; Does the innovation need to happen on the hardware side? I ask because there is a concerted effort, at least for inference, to reduce memory requirements and provide more access to giant LLMs.[0] It seems like everyone for the past few years has just thrown more and more compute at the wall to see what sticks. Where does it stop? 3 or 4 trillion parameter models that cost a $Billion to train? Is there a law of diminishing returns? Have we reached the apogee or will it be the 15 trillion parameter that ate $100Billion? Accessibility to all manner of models,…
Read the full comment
for knuckle heads like myself, is relatively new. Maybe I should rephrase, accessibility to interesting and FUN models that mere mortals can use and hack on is relatively new. Stable Diffusion is nothing short of spectacular for just having some fun with ML/DL. My naïve thought is that, similar to most natural systems, smaller and more specialized models that can be "glued" together as a way forward. In my mind this makes more sense than these ridiculously large language models. That is to say, purpose built or fine--tuned models that can communicate seems like a good idea. GPT-NeoX "communicates" with to Bloom and makes a gaggle of small models that can be fine-tuned further. If regular people with a consumer GPU can train/fine-tune a model at home, this happens a lot faster. [0] https://arxiv.org/abs/2208.07339
Original comment Hot Chips 34: AMD’s Instinct MI200 Architecture

Keep following the thread.

More in Language models

More from the same topics.