BitNet: Scaling 1-bit Transformers for Large Language Models

https://arxiv.org/pdf/2310.11453 The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model. You can build it and run some models, but the selection is limited because it has to be trained from scratch. I imagine inference speed is faster compared with modern PTQ (4- and 8-bit quants) though.

It's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3] [0] https://arxiv.org/abs/1603.01025 [1] https://par.nsf.gov/servlets/purl/10163070 [2] https://arxiv.org/abs/2310.11453 [3] https://arxiv.org/abs/1606.01981

I wonder how well BitNet (ternary weights) would work for this. It seems like a promising way forward for constrained hardware. https://arxiv.org/abs/2310.11453 https://github.com/cpldcpu/BitNetMCU/blob/main/docs/document...

The original BitNet paper (https://arxiv.org/pdf/2310.11453) BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1), but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764) The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits This seems to be first source of the confounding of "1-bit LLM" and ternary weights that I could find. In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}.

Section 2.2 from your paper, with less clarity and more obfuscation: >While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process. https://arxiv.org/pdf/2310.11453 The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it. Uncharitably you might say that they aren't hiding the lead 10 paragraphs in.

Not a stupid question at all. Early quantization approaches just quantized a high-bit-precision pre-trained model - no need to calculate gradients on the quantized weights. BitNet[1] changed the game by training a low-precision model from scratch. It achieves this by keeping high precision in the gradients, optimizer state, and in "latent weights" which are then quantized on the fly. I don't really understand the finer details of how this works, so check out the paper if you're interested. This article's approach is interesting. They start by quantizing a pre-trained high-precision model, and then they fine-tune the quantized model using LoRA (which dramatically improves the performance of the quantized model). They don't talk about the bit depth of the values in the LoRA matrices, so it may be that those are higher bit-depth. [1] https://arxiv.org/pdf/2310.11453.pdf

So for the uninitiated (me), does this mean the input is not a float (i.e. is quantized on input), such that all the math can be done with int operations? This seems almost too good to be true. Edit: Answering my own question, yes. The details are in the original bitnet paper: https://arxiv.org/abs/2310.11453

Does this help? https://arxiv.org/abs/2310.11453 It seems to have more details (it's the paper before the linked one) about the actual training, but I'm scanning it and this isn't my field so maybe it's too light also.

Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.