# Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Authors: Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng.

Converts new context into low-rank model updates in one forward pass.

ArXiv initial submission: 2024-11-08.

[Original paper](https://arxiv.org/abs/2411.05877) · [Reading-list entry](https://programasweights.com/reading-list/papers/2411.05877/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2411.05877/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2411.05877/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2411.05877/record.json)

**Contribution:** Trains an adapter generator with self-supervision. Tests learning from documents, demonstrations, and conversation history.

**Program form:** Additive low-rank adapters.

**Conditioning:** Documents, demonstrations, or conversation history, including successive chunks of context.

**Execution:** The adapted model answers new inputs using context stored in its parameter updates.

**Relationship to PAW:** PAW cites this context-to-adapter method. Generative Adapter stores information from context; PAW compiles the behavior of a function.

**Limitations:** The adapter generator is tied to its frozen base model.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: Generative Adapter, contextualization, fast weights, streaming context.

## Method comparisons

**What gets reused?**

Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.

Sources: [Gist paper](https://arxiv.org/abs/2304.08467) · [Generative Adapter paper](https://arxiv.org/abs/2411.05877) · [PAW program definition](https://arxiv.org/html/2607.02512v1#S2)

Paper details: [Learning to Compress Prompts with Gist Tokens](https://programasweights.com/reading-list/papers/2304.08467/) · [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://programasweights.com/reading-list/papers/2411.05877/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


An AFM 2024 OpenReview version appeared before the first arXiv submission. [Source](https://arxiv.org/abs/2411.05877).

## Published version

Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass. Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng. International Conference on Learning Representations, 2025.
[Publication](https://openreview.net/forum?id=bc3sUsS6ck) · [BibTeX](https://programasweights.com/reading-list/papers/2411.05877/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2411.05877/publication.csl.json)
[Publication record](https://openreview.net/pdf?id=bc3sUsS6ck).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.
- [Context compression](https://programasweights.com/reading-list/guides/compilation/#context-compression): Compress an instruction or context once and reuse the result across calls.

Reading-list version 1.2.0; updated 2026-10-19.