Edward J. Hu, Yelong Shen, Phillip Wallis
Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
2021 · arXiv
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
18 selected excerpts · 18 total mentions
> Anyone have any ideas / knowledge on how they deploy little incremental fixes to exploited jailbreaks, etc? LoRa[1] would be my guess. For detailed explanation I recommend the paper. But the short explanation is that it is a trick which lets you train a smaller, lower dimensional model which when you add to the original model it gets you the result you want. 1: https://arxiv.org/abs/2106.09685
In this token-mania frenzy that has taken hold of the industry, I guess solutions like "soul document" and "system prompts" will continue for a while, and once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3]. The other explanation may be that these AI labs may be expecting more government scrutiny, and "here's a document" would probably go better than "here's some vector representation of our values" when talking to politicians. [1] https://arxiv.org/abs/2106.09685 [2] https://vgel.me/posts/representation-engineering/ [3] https://transformer-circuits.pub/2024/scaling-monosemanticit...
You should look into LoRA, it’s a partial retraining method, doesn’t require nearly as much as retraining the whole model. It’s different from what this paper is suggesting. The self improvements in this paper even sets the rules for the improvements, basically creating new data out of what it has. LoRA paper: https://arxiv.org/abs/2106.09685
> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a trick to make it faster and/or be able to do it on lower end hardware. LoRA doesn't add "isolated subnetworks" - LoRA parameters are added to the original weights! Here's the equation for the forward pass from the original paper[1]: h = W_{0} * x + ∆W * x =…
W_{0} * x + B * A * x where "W_{0}" are the original weights and "B" and "A" (which give us "∆W_{x}" after they're multiplied) are the LoRA adapter. And if you've been paying attention it should also be obvious that, mathematically, you can merge your LoRA adapter into the original weights (by doing "W = W_{0} + ∆W") which most people do, or you could even create a LoRA adapter from a fully fine-tuned model by calculating "W - W_{0}" to get ∆W and then do SVD to recover B and A. If you know what you're doing anything you can do with LoRA you can also do with full-finetuning, but better. It might be true that it's somewhat harder to "damage" a model by doing LoRA (because the parameter updates are fundamentally low rank due to the LoRA adapters being low rank), but that's a skill issue and not a fundamental property. [1] -- https://arxiv.org/pdf/2106.09685
> When you finetune with lora, you're updating maybe 5% of the parameters I'm not sure I understand this comment. The LoRA paper[1] specifically says that all of the pretrained weights remain frozen. > keeping the pre-trained weights frozen Specifically, the LoRA paper differentiates itself from updating some parameters by stating > Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks. 1. https://arxiv.org/pdf/2106.09685
Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference. The key is that the matrices A and B have a very low rank[2] compared to W_0. Training is then done by treating W_0 as constant, and only updating the A and B matrices. The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between other layers in a neural network. The "experts" can be just some simple matrices or…
neural networks in their own right. The Lamini model seems to combine these ideas, where they use several "experts" layered between the BA matrices. However instead of just a simple Softmax to select the "experts" they use (chunked?) cross-attention[4], and a much larger number of "experts" compared to the MoE paper. From what I gather the cross-attention allows the expert selection to react to the context, unlike the more naive plain Softmax gating approach. Similarly to LoRA, they train the Lamini model by keeping the pre-trained LLM weights constant. From what I can gather they do a bit of training on the cross-attention layer but then freeze that too, to avoid the cross-attention layer favoring the same "experts" all the time. The idea then is to train the "experts" and the LoRA layer on facts until the combined model (W above) gets each fact correct (zero loss). Thus when the model "sees" a keyword in a sentence, a set of "experts" will steer/adjust the output of the pre-trained LLM to output the correct fact. Or at least that's how I imagine it works. What's less clear to me is what exactly the "experts" are and how they are combined. Since they're used to adjust the weights of the combined model, I it makes the most sense to my un-trained eye that they're simple matrices as mentioned in the MoE paper. Given they're layered between the low-rank portions of the LoRA section, they're necessarily relatively small matrices, so having millions of these "expert" matrices doesn't add too many parameters overall. And while they're portrayed as stacked in the Lamini paper, suggesting matrix multiplication, matrices are generally not commutative[5]. So to the un-trained eye it seems likely the "experts" are simply added like in the MoE paper. But yeah, I'm very much not an expert and the paper was more like a conference poster at best, lacking a lot of detail, so this might be all gibberish and I'd appreciate being corrected. [1]: https://arxiv.org/abs/2106.09685 [2]: https://en.wikipedia.org/wiki/Rank_factorization [3]: https://arxiv.org/abs/1701.06538 [4]: https://arxiv.org/abs/2112.04426 [5]: https://en.wikipedia.org/wiki/Commuting_matrices
This seems to be an order of magnitude better than LoRa (https://lora-alliance.org/ not https://arxiv.org/abs/2106.09685). LoRa doesn't have all the features this one does like OFDM, TDM, FDM, and HARQ. I didn't know there's spectrum dedicated for DECT use.
From https://arxiv.org/abs/2106.09685: > LoRA: Low-Rank Adaptation of Large Language Models > An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer…
architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency.
Somewhat yes. See "LoRA": https://arxiv.org/abs/2106.09685 They're not composable in the sense that you can take these adaptation layers and arbitrarily combine them, but training different models while sharing a common base of weights is a solved problem.
Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685 Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...
I haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers. Check out https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA (https://arxiv.org/pdf/2106.09685.pdf) That should get you started.
>when it is well-known in the field that parameter-efficient fine-tuning always pays a cost in terms of performance relative to full fine-tuning The LoRA paper clearly states the performance of the method "LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. ": https://arxiv.org/abs/2106.09685
I think your first statement is incorrect. Lora seems to be a method to fine-tune and optimize the weights of models like Alpaca. It is not a different dataset. This reduces model sizes and therefore also compute costs. See the abstract of https://arxiv.org/pdf/2106.09685.pdf
They address prompt tuning's issues in the paper: "The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods." https://ar5iv.labs.arxiv.org/html/2106.09685 This is key imo: "More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task".
LoRA is an alternative to traditional fine tuning (which is usually done on specific layers as you mentioned). To quote the LoRA paper[1]: > We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach. LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen It's truly revolutionary: It basically lets you create a very small "diff" which you apply yo an existing model and it is suddenly fine tuned. These diff models are very small (5M for example). [1] https://arxiv.org/abs/2106.09685
What's fun is that the recent references to LoRA stands for Low-Rank Adaptation [1], not Low-Resource Adapters[2] (??? don't know if this even exists), but because Low-Rank Adaptation came out in 2021, chatgpt only explains what Low-Resource Adapters is, not Low-Rank Adaptation, which is what is being used in recent break throughs. My own questioning of chatgpt on LoRA returns "Logistic Regression Algorithm" instead. Looks like it's too new to ChatGPT. [1] https://arxiv.org/abs/2106.09685 [2] https://rentry.co/vafkn
Picked the wrong one. LoRA, Low-rank Adaptation of LLMs (https://arxiv.org/pdf/2106.09685.pdf), consists in adapting the weights of a big neural network to a target task (here, answering to instructions). It doesn't touch the weights of the original model, but rather adds the product of two low-rank matrices to select layers. The weights from those matrices are learnable. The method allows to adapt big models on (relatively) low-memory GPUs.
>they perform a little worse. Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.