# LoRA: Low-Rank Adaptation of Large Language Models

Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen.

Adapts a pretrained model with low-rank weight updates while keeping the original weights fixed.

ArXiv initial submission: 2021-06-17.

[Original paper](https://arxiv.org/abs/2106.09685) · [Reading-list entry](https://programasweights.com/reading-list/papers/2106.09685/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2106.09685/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2106.09685/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2106.09685/record.json)

**Contribution:** Reduces the parameters trained and stored for each task. Evaluates adaptation across several language-model families.

**Program form:** Low-rank matrices that update selected model weights.

**Conditioning:** Training examples for the target task.

**Execution:** The language model applies the low-rank updates during inference.

**Relationship to PAW:** PAW uses LoRA as a program representation and generates the adapter with a trained compiler.

**Limitations:** A separate adapter is trained for each task.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: LoRA, low-rank adaptation, parameter-efficient fine-tuning, adapters.

## Method comparisons

**Program form and compiler**

LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.

Sources: [LoRA paper](https://arxiv.org/abs/2106.09685) · [Prefix-tuning paper](https://aclanthology.org/2021.acl-long.353/) · [PAW implementations](https://arxiv.org/html/2607.02512v1#S3)

Paper details: [LoRA: Low-Rank Adaptation of Large Language Models](https://programasweights.com/reading-list/papers/2106.09685/) · [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://programasweights.com/reading-list/papers/2101.00190/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

LoRA: Low-Rank Adaptation of Large Language Models. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. International Conference on Learning Representations, 2022.
[Publication](https://openreview.net/forum?id=nZeVKeeFYf9) · [BibTeX](https://programasweights.com/reading-list/papers/2106.09685/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2106.09685/publication.csl.json)
[Publication record](https://openreview.net/pdf?id=nZeVKeeFYf9).

## Approaches

- [Parameter-efficient building blocks](https://programasweights.com/reading-list/guides/compilation/#peft): Adapt a frozen model with a small set of learned parameters.

Reading-list version 1.2.0; updated 2026-10-19.