Teacher-Generated Examples for Small Models: Program-as-Weights
Yuntian Deng ·
Program-as-Weights (PAW) learns from teacher-generated examples to build reusable functions for a small language model. The examples train a neural compiler to turn a task description into program weights. The interpreter’s base weights stay frozen; a generated adapter supplies the behavior for each function.
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. NeurIPS 2026.
First released · PAW BibTeX
From teacher-generated data to a small local function
Training a small model on teacher-generated examples transfers task behavior from a large model. PAW uses that supervision to learn how to build functions. Its compiler learns across many specifications, then generates programs for new specifications without a fresh training run for each one.
For example, a teacher supplies an urgency-classification specification, messages, and the expected labels. The compiler generates a program from the specification. The small interpreter runs that program on each message, and its prediction loss trains the compiler. At deployment, a user supplies a new function definition; the trained compiler produces the program, which runs locally on future inputs.
This separates three operations:
- Train the compiler: learn from specifications and teacher-generated input–output examples.
- Compile a function: generate its program from a natural-language definition.
- Call the function: execute the saved program on a new input with the small interpreter.
The teacher and compiler are absent from those repeated function calls. PAW treats the result as software that can be saved, loaded, and composed, following its compiler–interpreter programming model.
Teacher examples describe the behavior to learn
FuzzyBench contains 10 million triples: a function specification, an input, and the expected output. Its construction uses GPT-5.2 in two stages. First, the teacher generates function descriptions under task-category constraints. Then it generates eight input–output pairs for each description.
Functions in the dataset parse text, repair formats, match strings by meaning, and translate natural-language commands. The dataset spans more than 800 task categories. Appendix B provides the generation prompts.
What learns from the teacher examples?
The training objective connects the generated program to the behavior it must implement. For each training triple:
- The compiler reads the specification and a pre-generated text pseudo-program.
- A learned mapper converts compiler hidden states into LoRA weights.
- The frozen interpreter loads those weights and predicts the target output from the input.
- The prediction loss updates the compiler and mapper through the interpreter.
The interpreter’s base weights and the off-the-shelf model that writes the text pseudo-program stay frozen. The teacher examples therefore supervise the program generator, rather than a separate full student model for every task.
One runtime, many programs
A PAW program combines a text pseudo-program with a parameter-efficient neural module. The paper implements both LoRA and prefix-tuning programs. Changing the program changes the function while retaining the same interpreter.
That makes the output of compilation a software artifact: it can be saved, versioned, distributed, and called repeatedly. PAW’s contribution includes this compiler–interpreter interface and local execution, alongside the method for producing adapters. The paper’s developer interface shows the path from a function definition to calls in an application.
FuzzyBench separates training and test specifications, so evaluation asks whether the compiler builds functions it did not see during training. Read the neural-compilation benchmark guide for that evaluation and generating LoRA weights from task descriptions for the mechanism. The PAW paper and compiler source provide the full system.
This training objective implements the AI-as-tool-builder vision: learn to produce a program that handles new inputs, then reuse that program. Two-stage decision models apply the same separation of building and running to classification, routing, and triage.
Papers and citations
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. The Fortieth Annual Conference on Neural Information Processing Systems, 2026.
BibTeX · Paper summary