Weight-Space Programming Reading List
How do we turn a specification into neural-network weights?
Program-as-Weights is weight-space programming through a trained neural compiler. A large model builds a program for a small interpreter. PAW introduced fuzzy-function programming on July 2, 2026: define a function in natural language, compile it, save the neural program, and call it locally.
A LoRA adapter is a weight update. PAW’s compiler writes that update for the small interpreter from a function specification. The compiler learns this ability during training; compiling a new function generates its weights in one forward pass, without per-function gradient descent.
Read and cite Program-as-Weights
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng
NeurIPS 2026 · First released July 2, 2026
The vision is to train large models to become powerful compilers that build reusable neural programs. The November 2026 Sparks of Weight-Space Programming work investigates weight programming in existing pretrained LLMs. Both program through weights. PAW trains the large model to be a compiler; Sparks uses an existing LLM’s reasoning and tools to read and write numerical weights.
Read the field guide →Start with these fiveBrowse 12 works
Start with these five
- Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Start with the programming paradigm: a large compiler builds a saved program for a small interpreter. - HyperNetworks
Read the foundation for using one neural network to generate another network’s weights. - Text-to-LoRA: Instant Transformer Adaption
See how a trained hypernetwork generates adapters from task descriptions. - Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights
Compare trained compilation with a general-purpose LLM reading and writing numerical weights. - Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations
Follow reasoning over weights into a study of repeated searches and cross-task transfer.
Four ways to build with weights
Reusable neural programs
Build a function once, save its program, and execute it on new inputs. PAW defines the compiler–interpreter interface; Compile by Training adds a compilation method.
Program-as-WeightsCompile by Training
Trained weight generators
Train one network to generate another network’s parameters from task descriptions, examples, or repository contents.
HyperNetworksLearning to Generate Task-Specific Adapters from Task DescriptionHyperTuningHINTText-to-LoRACode2LoRA
Formal program compilation
Translate a program written in a formal language into a network with a known computation.
Reasoning over numerical weights
Use a pretrained LLM to inspect parameters, write networks, or edit existing weights through reasoning and tools.
Sparks of Weight-Space ProgrammingWeight-Space Programming from scratch on simple tasksExploring Weight-Space Programming
Papers and research articles
Ordered by first public release. Each entry links to the original work and its citation.
Trained weight generators
HyperNetworks
First arXiv submission · · ICLR 2017
Uses one neural network to generate the weights of another.
- Input
- Layer embeddings or the current recurrent state.
- Building the program
- A trained network generates the parameters of another network.
- Training and feedback
- The weight generator is trained jointly with the target network.
- Saved artifact
- Weights for the target network.
- Execution
- The target network runs with the generated weights.
HyperNetworks supplies a mechanism for generating weights. PAW trains a compiler to use generated parameters as reusable programs.
Trained weight generators
Learning to Generate Task-Specific Adapters from Task Description
First arXiv submission · · ACL-IJCNLP 2021
Introduces Hypter, which generates adapters from natural-language task descriptions.
- Input
- A natural-language task description.
- Building the program
- A description encoder and hypernetwork generate adapters for a text-to-text model.
- Training and feedback
- The generator learns from tasks with descriptions and labeled examples.
- Saved artifact
- Bottleneck adapters inserted into a text-to-text model.
- Execution
- BART performs the task with the generated adapters.
An early description-to-adapter method cited by PAW. PAW develops the programming interface for saving and calling the resulting neural functions.
Trained weight generators
HyperTuning: Toward Adapting Large Language Models without Back-propagation
First arXiv submission · · ICML 2023
Produces task-specific parameters for a frozen language model from a few demonstrations.
- Input
- A few examples of the task.
- Building the program
- HyperT5 generates prefix or LoRA parameters from task demonstrations.
- Training and feedback
- Hyperpretraining and multitask training prepare the generator; the paper also studies further task-specific tuning.
- Saved artifact
- Generated prefix or LoRA parameters.
- Execution
- A frozen T5 model runs with the generated parameters.
PAW cites HyperTuning as prior work on generating adaptation parameters from examples.
Trained weight generators
HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation
First arXiv submission · · ACL 2023
Turns instructions and examples into reusable prefixes, adapters, and encoded instruction representations.
- Input
- Task instructions and optional examples.
- Building the program
- An instruction-conditioned hypernetwork generates modules, and instruction fusion caches the instruction representation.
- Training and feedback
- Instruction tuning trains the generator and model to generalize across tasks.
- Saved artifact
- Prefixes, adapters, and encoded instruction representations.
- Execution
- An encoder-decoder model uses the generated modules and attends to cached instruction representations.
HINT reuses generated modules and an encoded instruction. PAW reuses a program containing text and adaptation parameters.
Formal program compilation
Tracr: Compiled Transformers as a Laboratory for Interpretability
First arXiv submission · · NeurIPS 2023
Compiles RASP programs into transformer weights with known computational structure.
- Input
- A RASP program, input vocabulary, and maximum sequence length.
- Building the program
- A symbolic compiler translates each RASP operation into network components and assembles their weights.
- Training and feedback
- Compilation constructs the weights without model training. A separate experiment trains a projection to compress the compiled model.
- Saved artifact
- Transformer weights assembled from attention and MLP components.
- Execution
- The compiled transformer executes the specified sequence computation.
Tracr compiles a formal program. PAW learns to compile natural-language specifications of fuzzy functions into programs for a shared interpreter.
Trained weight generators
Text-to-LoRA: Instant Transformer Adaption
First arXiv submission · · ICML 2025
Uses a hypernetwork to generate LoRA adapters from task descriptions.
- Input
- A natural-language task description.
- Building the program
- A hypernetwork maps the task description to LoRA matrices.
- Training and feedback
- The paper trains generators by reconstructing adapters and through supervised task learning.
- Saved artifact
- Generated LoRA matrices.
- Execution
- A base transformer performs the task with the generated LoRA.
PAW cites Text-to-LoRA as prior work on generating adapters. PAW develops fuzzy-function programming, FuzzyBench, and an interface for reusable neural programs.
Trained weight generators
Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution
First arXiv submission · · arXiv
Trains a hypernetwork to generate repository-specific LoRA adapters from source code and update them as commits arrive.
- Input
- A repository snapshot; the evolution variant also receives successive commit diffs.
- Building the program
- Static maps an encoded repository snapshot to LoRA weights. Evo updates a GRU state with each commit diff and generates the weights from that state.
- Training and feedback
- The hypernetwork learns across repositories through the frozen code model’s assertion-completion loss. New repositories receive generated adapters without per-repository fine-tuning.
- Saved artifact
- Generated LoRA matrices for a frozen code language model.
- Execution
- The code model uses the generated adapter without adding repository context tokens to each request.
Code2LoRA and PAW train models to generate reusable weights. Code2LoRA puts repository knowledge into adapters; PAW compiles function specifications into neural programs with a compiler–interpreter interface.
Reusable neural programs
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
First arXiv submission · · NeurIPS 2026
Proposes fuzzy-function programming: compile a natural-language function specification into a neural program, then reuse it locally.
- Input
- A function specification, optionally with examples. The image extension also accepts visual information.
- Building the program
- A large trained compiler produces a neural program from the function specification. Its fast compiler generates LoRA weights in one forward pass.
- Training and feedback
- The compiler learns to generate programs before deployment. New functions compile without per-function gradient descent.
- Saved artifact
- A discrete pseudo-program and a compiler-generated parameter-efficient module.
- Execution
- A frozen lightweight interpreter loads the program and runs new inputs locally.
PAW makes weight-space programming a software interface: define a fuzzy function, compile it, save the neural program, and call it locally. The large model is the compiler; the small model is the interpreter.
Reusable neural programs
Compile by Training: Turning Natural-Language Specifications into Local Neural Functions
First arXiv submission · · EMNLP 2026 System Demonstrations
Compiles a function description into a reusable local program by generating training examples and optimizing an adapter.
- Input
- The specification guides teacher synthesis. PAW’s fast compiler supplies the initial adapter and runtime prompt.
- Building the program
- Teacher models generate examples from the specification. Gradient descent refines the adapter initialized by PAW’s fast compiler.
- Training and feedback
- Training happens at function compilation; the saved program runs without the teachers.
- Saved artifact
- A LoRA adapter, runtime prompt, specification, and interpreter metadata in a PAW program.
- Execution
- A frozen Qwen3-0.6B interpreter runs the saved function locally without further teacher calls.
Extends PAW with training at compile time while keeping its program format and runtime interface.
Reasoning over numerical weights
Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights
Article publication date · · Research article
Uses a general-purpose LLM to read neural-network weights, construct networks, and edit an existing model through numerical reasoning and tools.
- Input
- A natural-language objective, network weights or an empty network, and task-dependent tool access.
- Building the program
- The LLM reasons about the computation and writes the numerical parameters. Larger experiments use a Python environment and forward-pass feedback.
- Training and feedback
- The reported weight-construction and editing procedures prohibit gradient descent and training loops.
- Saved artifact
- New network weights or an edit to an existing checkpoint.
- Execution
- The constructed or edited network runs on new inputs.
PAW generates reusable neural programs with a trained compiler. Sparks uses a general-purpose LLM to inspect and write numerical weights directly.
Reasoning over numerical weights
Weight-Space Programming from scratch on simple tasks
Article publication date · · Research article
An LLM coding agent writes transformer weights for tasks such as addition, sorting, and parity, then revises the programs using evaluation feedback.
- Input
- A task definition and a transformer implementation with an empty weight-writing function.
- Building the program
- The agent edits the architecture and parameter-setting code, runs the resulting model, and simplifies successful programs.
- Training and feedback
- The algorithmic-task procedure uses closed-form parameter assignment. The report identifies fitting and evaluation leakage in its language-task runs.
- Saved artifact
- A transformer architecture and its explicitly assigned weights.
- Execution
- The written transformer generates its answer token by token.
Both this work and PAW produce neural programs that run after construction. Here an LLM writes the implementation through iterative code edits; PAW uses a trained compiler and a shared interpreter.
Reasoning over numerical weights
Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations
First arXiv submission · · arXiv
Studies direct weight editing across 16 searches, seven benchmark families, and two model scales, measuring held-out gains and transfer between tasks.
- Input
- A pretrained Qwen3.5 checkpoint, task data, and generation or teacher-forcing feedback.
- Building the program
- A reasoning agent inspects parameters, writes weight updates, and selects a checkpoint using development feedback.
- Training and feedback
- The editing search uses no gradient-based optimization. Each search has a budget of 50 successful forward-tool requests.
- Saved artifact
- A modified pretrained checkpoint.
- Execution
- The selected checkpoint answers held-out questions without the editing agent.
This study evaluates reasoning-driven edits to an existing model. PAW compiles a function definition into a separate reusable program for its interpreter.
Download the reading list
Weight-space programming guide → · PAW research · Neural compilation reading list