Generating LoRA Weights from Task Descriptions with Program-as-Weights
Yuntian Deng ·
Program-as-Weights (PAW) generates LoRA weights from natural-language task descriptions and packages them as reusable local functions. A neural compiler reads the function definition and builds a program; a small language model loads that program and executes it on new inputs. The large model builds the tool once, and the small model runs it repeatedly.
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. NeurIPS 2026.
First released · PAW BibTeX
Generate a program once, run it on new inputs
Suppose the task description is “Classify whether a message needs immediate attention.” PAW compiles that definition into a program containing a text pseudo-program and a neural module. Load it into the interpreter, pass in a message, and read the classification. Reuse the same program for the next message.
This is PAW’s fuzzy-function programming model: the function definition is separate from its calls. Programs can be saved, versioned, distributed, and composed with ordinary code. The paper implements neural modules as LoRA adapters and prefix vectors; the compiler–interpreter interface supports both. PAW, §2.
Hypernetworks for language model adaptation
A hypernetwork generates the parameters of another network. PAW uses this mechanism to specialize a frozen language model for a function described in English. Its LoRA compiler and mapper are trained together to produce adapters that make the interpreter execute the requested function.
The LoRA-generation path has three components:
- Pseudo compiler: an off-the-shelf language model rewrites the specification as a text pseudo-program with examples.
- LoRA compiler and mapper: a trained 4B model reads the specification and pseudo-program. A learned mapper converts its hidden states into the interpreter’s LoRA weights in a forward pass.
- Interpreter: a small, frozen language model loads the generated adapter and uses the pseudo-program to answer each input.
After compiler training, generating a new function’s LoRA does not require a new optimization run. At execution time, the compiler is absent: the saved program and interpreter are enough. A single interpreter serves many programs by loading their adapters.
How PAW relates to other adapter generators
PAW combines generated weights with a programming abstraction and a local runtime. Earlier work establishes several ways to condition a weight generator:
- HyperNetworks, by Ha, Dai, and Le, uses one network to produce another network’s weights.
- Hypter, by Ye and Ren, generates task-specific adapters from natural-language descriptions. It tests generalization to unseen tasks with BART.
- HyperTuning, by Phang and colleagues, generates prefixes or LoRA parameters for a frozen T5 model from a few demonstrations. The generated parameters also serve as an initialization for further tuning.
- HINT, by Ivison and colleagues, converts instructions and examples into reusable modules and an encoded instruction, avoiding repeated instruction encoding.
- Text-to-LoRA, by Charakorn and colleagues, uses a hypernetwork to generate LoRA adapters from task descriptions. It studies reconstructing existing adapters and generating adaptations for unseen tasks.
Read PAW’s related-work discussion for these connections. The teacher-generated training data guide explains how the compiler learns, and PAW’s neural-compilation benchmark tests whether it builds functions it has not seen during training.
The broader goal is AI as a tool builder: train models to produce reusable neural programs that people can run locally. Decision models show this build-once, run-many architecture in classification, routing, and triage.
Papers and citations
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. The Fortieth Annual Conference on Neural Information Processing Systems, 2026.
BibTeX · Paper summary
HyperNetworks
David Ha, Andrew Dai, Quoc V. Le. arXiv, 2016.
BibTeX · Paper summary
Learning to Generate Task-Specific Adapters from Task Description
Qinyuan Ye, Xiang Ren. arXiv, 2021.
BibTeX · Paper summary
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen. arXiv, 2022.
BibTeX · Paper summary
HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation
Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew Peters. arXiv, 2022.
BibTeX · Paper summary
Text-to-LoRA: Instant Transformer Adaption
Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange. arXiv, 2025.
BibTeX · Paper summary