Paper
HyperTuning: Toward Adapting Large Language Models without Back-propagation
First arXiv submission: · ICML 2023
Produces task-specific parameters for a frozen language model from a few demonstrations.
What it does
Introduces HyperT5 models that generate prefixes or LoRA parameters. Hyperpretraining and multitask training prepare the generator for new tasks.
- Conditioning
- A few examples of the task.
- Program form
- Generated prefix or LoRA parameters.
- Execution
- A frozen T5 model runs with the generated parameters.
Related work
PAW cites HyperTuning as prior work on generating adaptation parameters from examples.
Limitations
The hypermodel accepts a limited number of demonstrations. The paper also studies further task-specific tuning.
Related terms: HyperTuning, HyperT5, few-shot adaptation, generated LoRA.
Related methods
From generated adapters to reusable functions
Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.
Cite this paper
Published version
Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen. HyperTuning: Toward Adapting Large Language Models without Back-propagation. Proceedings of the 40th International Conference on Machine Learning, 2023.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2211.12485, initial submission 2022-11-22.