Paper
Learning to Compress Prompts with Gist Tokens
First arXiv submission: · NeurIPS 2023
Compresses an instruction into a few reusable token activations.
What it does
Trains with an attention mask that forces later tokens to use gist activations in place of the original prompt.
- Conditioning
- The instruction or prompt.
- Program form
- Cached activations at a few gist-token positions.
- Execution
- The language model generates outputs while attending to the cached gist.
Related work
PAW cites gisting as another way to reuse instruction computation. Gisting stores activations; PAW stores a program with text and adaptation parameters.
Limitations
Compression can reduce output quality. The cached activations are tied to the trained model.
Related terms: gist tokens, prompt compression, activation caching, amortized inference.
Related methods
What gets reused?
Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.
Gist paper · Generative Adapter paper · PAW program definition
Cite this paper
Published version
Jesse Mu, Xiang Li, Noah Goodman. Learning to Compress Prompts with Gist Tokens. Advances in Neural Information Processing Systems, 2023.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2304.08467, initial submission 2023-04-17.