Paper
LoRA: Low-Rank Adaptation of Large Language Models
First arXiv submission: · ICLR 2022
Adapts a pretrained model with low-rank weight updates while keeping the original weights fixed.
What it does
Reduces the parameters trained and stored for each task. Evaluates adaptation across several language-model families.
- Conditioning
- Training examples for the target task.
- Program form
- Low-rank matrices that update selected model weights.
- Execution
- The language model applies the low-rank updates during inference.
Related work
PAW uses LoRA as a program representation and generates the adapter with a trained compiler.
Limitations
A separate adapter is trained for each task.
Related terms: LoRA, low-rank adaptation, parameter-efficient fine-tuning, adapters.
Related methods
Program form and compiler
LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.
Cite this paper
Published version
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations, 2022.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2106.09685, initial submission 2021-06-17.