# Learning to Compress Prompts with Gist Tokens

Authors: Jesse Mu, Xiang Lisa Li, Noah Goodman.

Compresses an instruction into a few reusable token activations.

ArXiv initial submission: 2023-04-17.

[Original paper](https://arxiv.org/abs/2304.08467) · [Reading-list entry](https://programasweights.com/reading-list/papers/2304.08467/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2304.08467/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2304.08467/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2304.08467/record.json)

**Contribution:** Trains with an attention mask that forces later tokens to use gist activations in place of the original prompt.

**Program form:** Cached activations at a few gist-token positions.

**Conditioning:** The instruction or prompt.

**Execution:** The language model generates outputs while attending to the cached gist.

**Relationship to PAW:** PAW cites gisting as another way to reuse instruction computation. Gisting stores activations; PAW stores a program with text and adaptation parameters.

**Limitations:** Compression can reduce output quality. The cached activations are tied to the trained model.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: gist tokens, prompt compression, activation caching, amortized inference.

## Method comparisons

**What gets reused?**

Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.

Sources: [Gist paper](https://arxiv.org/abs/2304.08467) · [Generative Adapter paper](https://arxiv.org/abs/2411.05877) · [PAW program definition](https://arxiv.org/html/2607.02512v1#S2)

Paper details: [Learning to Compress Prompts with Gist Tokens](https://programasweights.com/reading-list/papers/2304.08467/) · [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://programasweights.com/reading-list/papers/2411.05877/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

Learning to Compress Prompts with Gist Tokens. Jesse Mu, Xiang Li, Noah Goodman. Advances in Neural Information Processing Systems, 2023.
[Publication](https://proceedings.neurips.cc/paper_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html) · [BibTeX](https://programasweights.com/reading-list/papers/2304.08467/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2304.08467/publication.csl.json)
[Publication record](https://proceedings.neurips.cc/paper_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html).

## Approaches

- [Context compression](https://programasweights.com/reading-list/guides/compilation/#context-compression): Compress an instruction or context once and reuse the result across calls.

Reading-list version 1.2.0; updated 2026-10-19.