Paper
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass
First arXiv submission: · ICLR 2025
An AFM 2024 OpenReview version appeared before the first arXiv submission. Submission history ↗
Converts new context into low-rank model updates in one forward pass.
What it does
Trains an adapter generator with self-supervision. Tests learning from documents, demonstrations, and conversation history.
- Conditioning
- Documents, demonstrations, or conversation history, including successive chunks of context.
- Program form
- Additive low-rank adapters.
- Execution
- The adapted model answers new inputs using context stored in its parameter updates.
Related work
PAW cites this context-to-adapter method. Generative Adapter stores information from context; PAW compiles the behavior of a function.
Limitations
The adapter generator is tied to its frozen base model.
Related terms: Generative Adapter, contextualization, fast weights, streaming context.
Related methods
What gets reused?
Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.
Gist paper · Generative Adapter paper · PAW program definition
Cite this paper
Published version
Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng. Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass. International Conference on Learning Representations, 2025.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2411.05877, initial submission 2024-11-08.