How should neural functions be evaluated?
Evaluation has two parts: does the compiled function perform its task, and what does it cost to build and run?
What is held out?
New inputs for a familiar specification test the function. New specifications, task families, or domains test the compiler’s ability to generalize. Report how examples and task templates are split, including which training examples were generated synthetically.
PAW introduces FuzzyBench, a collection of 10 million specification–input–output triples. Its main experiments compare compiler–interpreter configurations against prompting baselines.
Compile by Training evaluates on FuzzyBench-Hard. This subset contains specifications for which the fast PAW compiler produced no exact matches. It tests improvement on those difficult cases. For each specification, separate the synthesized training examples from the evaluation inputs.
Compilation and execution costs
Measure compilation latency and resources, program size, shared interpreter size, loading time, and per-call latency. Include the expected number of calls: compilation cost is paid once and spread across later uses.
For local execution, report hardware, quantization, input/output lengths, and runtime. PAW evaluates quantization and execution on a MacBook M3.
On a representative specification and a B300 GPU, Compile by Training takes 50.9 seconds to compile, compared with 3.5 seconds for the fast compiler. The 50.9-second measurement includes cold compilation. The paper also measures compilation on other hardware. Calls to the saved program incur execution cost, without repeating compilation.
Test the specification
Paraphrase the specification, omit details, introduce typos, or add conflicting examples, then test the resulting function. Test corrupted runtime inputs separately. PAW’s robustness experiments evaluate noisy specifications and the role of the discrete pseudo-program.
Choose the metric
Exact match compares output strings. It works for fixed labels and canonical formats. Functions that accept several valid answers need a measure of semantic correctness. Other tasks call for format validation, rejecting inputs the function cannot handle, or success in the application using it.
Compile by Training uses LLM Exact Match (LEM). An LLM judge assesses semantic correctness from the specification, input, reference answer, and prediction. On FuzzyBench-Hard, mean LEM rises from 22.4% for the fast compiler to 83.6% for Compile by Training. The subset’s zero exact-match selection measures string equality; LEM measures semantic correctness. The authors compare the judge with 128 author labels and provide its prompt. Metric and validation, comparison.
Synthetic training examples can contain teacher errors. Test the compiled function on held-out inputs and test the application that uses it. The application experiments in Compile by Training are demonstrations without systematic user studies. Limitations.
Keep held-out tests while iterating on specifications. Rerun them after changing the interpreter or runtime, and record the program and software versions.
See the research overview for PAW and Compile by Training results, and the documentation for current software usage.