Paper
HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation
First arXiv submission: · ACL 2023
Turns instructions and examples into reusable prefixes, adapters, and encoded instruction representations.
What it does
Combines a hypernetwork with instruction fusion so repeated calls can reuse the instruction computation. Evaluates zero-shot and few-shot generalization.
- Conditioning
- Task instructions and optional examples.
- Program form
- Prefixes, adapters, and encoded instruction representations.
- Execution
- An encoder-decoder model uses the generated modules and attends to cached instruction representations.
Related work
HINT reuses generated modules and an encoded instruction. PAW reuses a program containing text and adaptation parameters.
Limitations
The generated modules and cached instruction representations are tied to the trained encoder-decoder model.
Related terms: HINT, instruction tuning, hypernetwork, instruction fusion.
Related methods
From generated adapters to reusable functions
Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.
Cite this paper
Published version
Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew E. Peters. HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2212.10315, initial submission 2022-12-20.