Paper
Prefix-Tuning: Optimizing Continuous Prompts for Generation
First arXiv submission: · ACL-IJCNLP 2021
Adapts a frozen language model by learning a short sequence of continuous prefix vectors.
What it does
Trains prefixes for table-to-text generation and summarization while updating far fewer parameters than full fine-tuning.
- Conditioning
- Training examples for the target task.
- Program form
- Continuous prefix activations.
- Execution
- The frozen language model attends to the learned prefix.
Related work
PAW evaluates compiler-generated prefixes alongside LoRA adapters.
Limitations
A separate prefix is optimized for each task.
Related terms: prefix-tuning, continuous prompts, virtual tokens, parameter-efficient fine-tuning.
Related methods
Program form and compiler
LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.
Cite this paper
Published version
Xiang Lisa Li, Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2101.00190, initial submission 2021-01-01.