Weight-Space Programming and Program-as-Weights
Yuntian Deng ·
Program-as-Weights (PAW) introduced fuzzy-function programming on July 2, 2026: a large model compiles a natural-language function specification into a reusable neural program, and a small model executes it locally.
The original PAW paper makes model weights a program representation: define a function, compile it, save the program, and call it on new inputs. Its fast compiler generates a small model’s LoRA weights in a single forward pass. PAW established this programming model four months before the November 2, 2026 post Sparks of Weight-Space Programming.
What PAW contributes
PAW separates function definition from function execution. A neural compiler turns a description into a program for a shared, lightweight interpreter. The saved program runs on new inputs without calling the compiler again. Fuzzy functions, such as repairing malformed JSON or identifying urgent messages, become software components that developers can store, distribute, and compose.
The paper’s implementation uses a 4B language model and a mapper to generate LoRA adapter weights for a frozen 0.6B interpreter. It also implements programs with prefix vectors. The compiler–interpreter interface defines the programming model; adapters and prefixes are its program formats.
The compiler is trained beforehand. Compiling a new function with PAW’s fast compiler generates its weights without per-function gradient descent. The small interpreter then loads the program and executes it locally.
Publication history
- — Program-as-Weights. Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. Program-as-Weights: A Programming Paradigm for Fuzzy Functions. NeurIPS 2026. Introduces fuzzy-function programming, the compiler–interpreter interface, and generated neural programs for local execution. First public paper and submission date.
- — Compile by Training. Yuntian Deng, Pengyu Nie, Stuart Shieber. Compile by Training: Turning Natural-Language Specifications into Local Neural Functions. EMNLP 2026 System Demonstrations. Adds a compilation method that trains a program on teacher-generated examples, retaining PAW’s program format and local runtime. First public paper and submission date.
- — Sparks of Weight-Space Programming. Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab. Demonstrates a general-purpose frontier LLM reading neural-network weights, writing networks, and editing an existing language model’s weights through reasoning and tools, without gradient descent. Original post.
How the methods relate
These works connect natural-language goals to neural-network parameters. They differ in how they produce the program or weight update.
- PAW’s fast compiler generates weights. A trained compiler and mapper produce a reusable adapter from the function specification in one forward pass. PAW, §3.2.
- Compile by Training optimizes a program. Teacher models generate examples, and training refines the adapter initialized by the fast compiler. The resulting function runs without the teachers. Compile by Training, §3.
- The November 2 work reasons over numerical weights. A general-purpose LLM inspects weights and writes numerical changes directly, using a reasoning and tool interface. Sparks of Weight-Space Programming.
PAW’s related-work section connects its implementation to earlier hypernetworks and generated-adapter methods.
Cite the research
For fuzzy-function programming, the compiler–interpreter model, and PAW’s fast compiler, cite Program-as-Weights. For compilation through teacher-generated examples and adapter training, cite Compile by Training.
PAW BibTeXCompile by Training BibTeXFull citations
Compile a function → · Compare neural compilation methods · PAW research