Skip to content
The methodOpen data
Back to the collection

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer

2023 · arXiv

Language modelsMachine learningDistributed systems
Read the paper
HN score
0
Mentions
10
Readers
7
Months
6
What’s in a score?

From the abstract

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

10 selected excerpts · 10 total mentions

godelski
Neutral mention · rule
In that paper they frequently say "integer quantization" for a reason. They relax to quantization because it's natural 4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization[15] which is an information-theoretically optimal data type that ensures each quantization bin has an equal number of values assigned from the input tensor. - QLoRA: Efficient Finetuning of Quantized LLMs https://arxiv.org/abs/2305.14314 3. Float8 Quantized Fine-tuning, for speeding up fine-tuning by dynamically quantizing high precision weights and activations to float8, similar to pre-training in float8. -…
Read the full comment
https://docs.pytorch.org/ao/stable/eager_tutorials/finetuning.html Or if you want an example of going the other way, here's dequantization and quantization but with normal precision: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_n... Or it's the same quanta in quantum mechanics (which isn't integer!) If you think I'm being pedantic, well... yes... the thread is about math, a formal and pedantic language
Original comment TorchLean: Formalizing Neural Networks in Lean
danielhanchen
Neutral mention · rule
I took a skim through it in the morning - I like the LoRA Learns Less and Forgets Less paper more https://openreview.net/forum?id=aloEru2qCG - it has much more signal in a few pages - also the original QLoRA paper from Dettmers https://arxiv.org/abs/2305.14314 has so many more important morsels. But all in all, the review is a reasonable "manual" I guess. I would have liked maybe more instructive comprehensive practical examples, and maybe more mention of other OSS packages for finetuning :))
Original comment Guide to Fine-Tuning LLMs
anon373839
Neutral mention · rule
Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of fine-tuning: 3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base model] https://arxiv.org/abs/2305.11206 4. QLoRA: Efficient Finetuning of Quantized LLMs [showing that LLMs can now be fine-tuned quickly on consumer-grade GPUs] https://arxiv.org/abs/2305.14314 —- Adding up these developments (all of which occurred during the span of one week), I don’t see how huge, slow, general-purpose models maintain their relevance in the long term, when a lean, domain-focused model is right there within reach of every application developer.
Original comment Falcon 40B LLM (which beats Llama) now Apache 2.0
jonathan-adly
Neutral mention · rule
I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with reasonable effort. 0. https://arxiv.org/abs/2305.14314 1. https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi 2. https://huggingface.co/timdettmers/guanaco-33b-merged
Original comment Ask HN: What's the best self hosted/local alternative to GPT-4?
kamranjon
Neutral mention · rule
I'd be really curious what the authors of the recent (3 days ago) paper on QLora would think of this article? https://arxiv.org/abs/2305.14314 - they claim "Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" Particularly this statement seems relevant: "We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT."
Original comment The False Promise of Imitating Proprietary LLMs
MacsHeadroom
Neutral mention · rule
Guanaco-65B[0] using Basaran[1] for your OpenAI compatible API. (You can use any ChatGPT front-end which lets you change the OpenAI endpoint URL.) [0] https://huggingface.co/TheBloke/guanaco-65B-HF A QLoRA finetune of LLaMA-65B by Tim Dettmers from the paper here: https://arxiv.org/abs/2305.14314 [1] https://github.com/hyperonym/basaran
Original comment Ask HN: What's the best self hosted/local alternative to GPT-4?

Keep following the thread.

More in Language models

More from the same topics.