Benchmarks for Natural-Language Function Compilation: Program-as-Weights
Yuntian Deng ·
Program-as-Weights (PAW) introduced FuzzyBench to benchmark natural-language function compilation. The task is to turn a function description into a reusable neural program, then execute that program on new inputs. PAW separates the compiler that builds the function from the small interpreter that runs it.
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. NeurIPS 2026.
First released · PAW BibTeX
What does a function-compilation benchmark test?
The compiler receives a natural-language function specification and produces a saved program. The evaluator then passes inputs to that program and checks its outputs. The program must implement the function across those calls. In PAW, it combines a text pseudo-program with neural parameters loaded by a shared interpreter. PAW, §2.
For example, a specification asks for a function that repairs malformed JSON. Compilation produces the program once. Evaluation sends broken JSON strings to the saved program and compares the repaired outputs with references.
Function calling benchmarks test selecting and invoking existing tools or APIs. Function compilation tests building the function itself. FuzzyBench evaluates this second task, including functions for parsing, classification, fuzzy matching, and tool-use preprocessing. PAW’s contribution is the programming model and working compiler–interpreter system; FuzzyBench supplies its training and evaluation tasks.
Evaluate a new function, then its calls
FuzzyBench splits specifications 80/10/10 into training, validation, and test sets. Test specifications are unseen during compiler training. This tests whether a learned compiler can build new functions, beyond reproducing the functions it learned from.
The dataset covers parsing, classification, fuzzy matching, format conversion, natural-language commands, agentic tool use, and other task families. Its examples come from a two-stage process: generate function descriptions, then input–output examples for each description. The compiler-training guide explains how these examples train the compiler.
For evaluation, the paper constructs a verified test set from examples whose target outputs are agreed on by GPT-5.2 and GPT-5-mini. The verified evaluation dataset is public on Hugging Face, with specifications, inputs, and reference outputs. The construction prompts are in Appendix B.
PAW’s main result
On the verified FuzzyBench test set, PAW with a Qwen3-0.6B interpreter reaches 73.78% exact match, compared with 68.70% for direct Qwen3-32B prompting. These are exact-match scores against the reference outputs. The paper also compares direct prompting at other model sizes, symbolic code generation, full fine-tuning, and fixed LoRA adapters. PAW, §6.
The distinction between a generated adapter and a fixed adapter matters here. PAW’s compiler produces a separate program from each function description. The interpreter serves as a shared runtime for those programs. Hypernetworks and LoRA generation explains this mechanism and its relationship to earlier adaptation methods.
Robustness and execution cost
FuzzyBench also tests perturbed specifications: typos, grammar errors, ambiguity, formatting drift, combined noise, terse phrasing, casual phrasing, and paraphrases. Each has three intensity levels. These tests ask whether a compiler preserves the intended function when the description changes. PAW, §8.
Measure compilation and repeated execution separately. A function built once and called thousands of times has a different cost profile from a remote model called for every input. PAW reports local execution at 30 tokens per second on a MacBook M3, using a shared quantized base of about 430 MB and a 23 MB adapter per program. PAW, §9.
For a complete comparison, report the specification split, metric, interpreter, program size, compilation cost, and runtime hardware. The evaluation guide connects these measurements to other work on neural programs.
This build-and-run separation also underlies PAW’s local decision models and the broader AI-as-tool-builder vision.
Papers and citations
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. The Fortieth Annual Conference on Neural Information Processing Systems, 2026.
BibTeX · Paper summary