# HyperTuning: Toward Adapting Large Language Models without Back-propagation

Authors: Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen.

Produces task-specific parameters for a frozen language model from a few demonstrations.

ArXiv initial submission: 2022-11-22.

[Original paper](https://arxiv.org/abs/2211.12485) · [Reading-list entry](https://programasweights.com/reading-list/papers/2211.12485/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2211.12485/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2211.12485/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2211.12485/record.json)

**Contribution:** Introduces HyperT5 models that generate prefixes or LoRA parameters. Hyperpretraining and multitask training prepare the generator for new tasks.

**Program form:** Generated prefix or LoRA parameters.

**Conditioning:** A few examples of the task.

**Execution:** A frozen T5 model runs with the generated parameters.

**Relationship to PAW:** PAW cites HyperTuning as prior work on generating adaptation parameters from examples.

**Limitations:** The hypermodel accepts a limited number of demonstrations. The paper also studies further task-specific tuning.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: HyperTuning, HyperT5, few-shot adaptation, generated LoRA.

## Method comparisons

**From generated adapters to reusable functions**

Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.

Sources: [PAW related work](https://arxiv.org/html/2607.02512v1#S10) · [Text-to-LoRA paper](https://proceedings.mlr.press/v267/charakorn25a.html)

Paper details: [Learning to Generate Task-Specific Adapters from Task Description](https://programasweights.com/reading-list/papers/2101.00420/) · [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://programasweights.com/reading-list/papers/2212.10315/) · [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://programasweights.com/reading-list/papers/2211.12485/) · [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/papers/2506.06105/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

HyperTuning: Toward Adapting Large Language Models without Back-propagation. Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen. Proceedings of the 40th International Conference on Machine Learning, 2023.
[Publication](https://proceedings.mlr.press/v202/phang23a.html) · [BibTeX](https://programasweights.com/reading-list/papers/2211.12485/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2211.12485/publication.csl.json)
[Publication record](https://proceedings.mlr.press/v202/phang23a.html).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.