The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Amusing that the link to an article on ternary numbers was posted by Mr Triplett. (: The article is well-written and illustrated. The technique described is used in llama.cpp for running language models like BitNet b1.58 whose weights are stored as ternary types. > ..in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1} > significantly more cost-effective in terms of latency, memory, throughput, and energy consumption The original paper on this technique was published in Feb 2024. (Also linked from the article) The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits - https://arxiv.org/abs/2402.17764 However, since then there have only been a few other models using ternary weights. I get the impression that there are factors not considered in the paper which make it less practical than it seemed.

fair, i think i was referring more to 1.58 bit architecture in general since the original paper (Figure 3) shows that we eliminate FP16 multiplication and addition just for INT8 addition. I need to dive deeper into bonsai overall if it differs https://arxiv.org/pdf/2402.17764

Didn't this paper demonstrate that you only need 1.58 bits to be equivalent to 16 bits in performance? https://arxiv.org/abs/2402.17764

> It doesn't show it can execute it at scale. Without writable state and bit exact ops, Well, modern LLM coding agent products (eg. Claude Code) are able to store state in files in the current repository. So, you could have the model keep the "CPU State", and the files in the repository be the "RAM". Also, could this https://arxiv.org/html/2402.17764v1 possibly reduce errors when doing inference? There is no floating point operations

Ternary quantized weights for LLMs are a thing. Most of the weights in Bitnet b1.58[1] class models[2][3] are ternary (-1/0/1). [1]: https://arxiv.org/abs/2402.17764 [2]: https://huggingface.co/tiiuae/Falcon-E-3B-Instruct [3]: https://huggingface.co/microsoft/bitnet-b1.58-2B-4T

The original BitNet paper (https://arxiv.org/pdf/2310.11453) BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1), but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764) The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits This seems to be first source of the confounding of "1-bit LLM" and ternary weights that I could find. In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}.

Repo with demo video and benchmark: https://github.com/microsoft/BitNet "...It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption..." https://arxiv.org/abs/2402.17764

That was just improper rounding from 1.58 to 1.5. They say 1.58 in other places and explicitly link to https://arxiv.org/abs/2402.17764

For reference, this is a real prototype for Bitnet 1.58b which uses (-1,0,1) as the weights which simplifies the matrix multiplication [0]. [0] https://arxiv.org/abs/2402.17764

No problem, glad you found it interesting, I think it's pretty exciting stuff. This is a more recent paper by the same people, where they tweak slightly to ternary (-1, 0, 1), and they seem to be able to maintain network performance: https://arxiv.org/pdf/2402.17764

> Then there's a trend towards much shorter numbers. 16 bit floating point? 8 bit? 2 bit? 1 bit? There was that recent paper titled "The Era of 1-bit LLMs" [0] which was actually suggeting a 1.58 bit LLM (2 bits in practice). > Someone reading this is probably writing it in VHDL right now, or will be soon. Yeah, I think I'm in the "will be soon" camp - FPGA board has been ordered. Especially with the 2-bit data types outlined in that paper [0] and more details in [1]. There's really a need for custom hardware to do that 2-bit math efficiently. Customizing one of the simpler open source RISC-V integer implementations seems like something to try here adding in the tiled matrix registers and custom instructions for dealing with them (with the 2 bit data types). [0] https://arxiv.org/abs/2402.17764 [1] https://github.com/microsoft/unilm/blob/master/bitnet/The-Er...

In both the '1-bit Model' and '2-bit Model' tables, the forward time (sec) for Llama2-7B with FP16 (full precision) is 0.1 s, whereas it's ~0.231, ~0.257, ~0.353 s respectively for HQQ (1-bit) / HQQ+ (1-bit) / Quip# (2-bit) meaning the FP16 model has ~3x lower inference time. On the contrary, in the BitNet b1.56 paper [0] the authors report their 7b model has 2.9x reduced inference latency. It's not clear to me what's happening here. Can someone explain why the 1/2bit HQQ/HQQ+ models are so much slower than the BitNet b1.56 models? [0] https://arxiv.org/pdf/2402.17764.pdf

> - Training isn’t done at 4-bits, to date this small size has only been for inference. Wasn't there a paper from Microsoft two weeks ago or so where they trained on log₂(3) bits? Edit: https://arxiv.org/pdf/2402.17764.pdf

Read the pdf https://arxiv.org/pdf/2402.17764.pdf they call it 1-bit everywhere. I don't know why do they do this, 1-bit seems to be a very wrong name for {-1, 0, 1}.