# Neural Compilation and Fuzzy-Function Programming

Reading list by Yuntian Deng. Version 1.2.0.

[Introduction](https://programasweights.com/reading-list/guides/fuzzy-functions/) · [Downloads](https://programasweights.com/reading-list/data/)

Papers are ordered by first arXiv submission date.

# Neural Programmer-Interpreters

Authors: Scott Reed, Nando de Freitas.

Learns neural programs that share a recurrent interpreter and a program memory.

ArXiv initial submission: 2015-11-19.

[Original paper](https://arxiv.org/abs/1511.06279) · [Reading-list entry](https://programasweights.com/reading-list/papers/1511.06279/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/1511.06279/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/1511.06279/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/1511.06279/record.json)

**Contribution:** Learns from execution traces and composes subprograms for addition, sorting, and 3D-object canonicalization.

**Program form:** Program embeddings stored in a key-value memory.

**Conditioning:** A program identifier, arguments, and observations of the environment.

**Execution:** A recurrent interpreter calls subprograms and takes actions in the environment.

**Relationship to PAW:** PAW cites this work on learned program representations and shared interpreters.

**Limitations:** Training requires execution traces and task-specific interfaces.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: neural programmer-interpreter, neural programs, program memory, compositional execution.

## Published version

Neural Programmer-Interpreters. Scott Reed, Nando de Freitas. International Conference on Learning Representations, 2016.
[Publication](https://arxiv.org/abs/1511.06279) · [BibTeX](https://programasweights.com/reading-list/papers/1511.06279/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/1511.06279/publication.csl.json)
[Publication record](https://arxiv.org/pdf/1511.06279).

## Approaches

- [Programs and reusable artifacts](https://programasweights.com/reading-list/guides/fuzzy-functions/): Learn programs that can be saved and run on new inputs.

Reading-list version 1.2.0; updated 2026-10-19.


# HyperNetworks

Authors: David Ha, Andrew Dai, Quoc V. Le.

Uses one neural network to generate the weights of another.

ArXiv initial submission: 2016-09-27.

[Original paper](https://arxiv.org/abs/1609.09106) · [Reading-list entry](https://programasweights.com/reading-list/papers/1609.09106/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/1609.09106/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/1609.09106/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/1609.09106/record.json)

**Contribution:** Applies weight generation to convolutional and recurrent networks, relaxing weight sharing across layers and generating dynamic weights for sequence models.

**Program form:** Weights for the target network.

**Conditioning:** Layer embeddings or the current recurrent state.

**Execution:** The target network runs with the generated weights.

**Relationship to PAW:** Provides the weight-generation mechanism used by later description-to-adapter methods and PAW.

**Limitations:** The hypernetwork is trained end to end for the target architecture.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: hypernetwork, weight generation, fast weights, conditional parameters.

## Published version

HyperNetworks. David Ha, Andrew M. Dai, Quoc V. Le. International Conference on Learning Representations, 2017.
[Publication](https://openreview.net/forum?id=rkpACe1lx) · [BibTeX](https://programasweights.com/reading-list/papers/1609.09106/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/1609.09106/publication.csl.json)
[Publication record](https://openreview.net/pdf?id=rkpACe1lx).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.


# Prefix-Tuning: Optimizing Continuous Prompts for Generation

Authors: Xiang Lisa Li, Percy Liang.

Adapts a frozen language model by learning a short sequence of continuous prefix vectors.

ArXiv initial submission: 2021-01-01.

[Original paper](https://arxiv.org/abs/2101.00190) · [Reading-list entry](https://programasweights.com/reading-list/papers/2101.00190/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2101.00190/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2101.00190/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2101.00190/record.json)

**Contribution:** Trains prefixes for table-to-text generation and summarization while updating far fewer parameters than full fine-tuning.

**Program form:** Continuous prefix activations.

**Conditioning:** Training examples for the target task.

**Execution:** The frozen language model attends to the learned prefix.

**Relationship to PAW:** PAW evaluates compiler-generated prefixes alongside LoRA adapters.

**Limitations:** A separate prefix is optimized for each task.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: prefix-tuning, continuous prompts, virtual tokens, parameter-efficient fine-tuning.

## Method comparisons

**Program form and compiler**

LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.

Sources: [LoRA paper](https://arxiv.org/abs/2106.09685) · [Prefix-tuning paper](https://aclanthology.org/2021.acl-long.353/) · [PAW implementations](https://arxiv.org/html/2607.02512v1#S3)

Paper details: [LoRA: Low-Rank Adaptation of Large Language Models](https://programasweights.com/reading-list/papers/2106.09685/) · [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://programasweights.com/reading-list/papers/2101.00190/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

Prefix-Tuning: Optimizing Continuous Prompts for Generation. Xiang Lisa Li, Percy Liang. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
[Publication](https://aclanthology.org/2021.acl-long.353/) · [BibTeX](https://programasweights.com/reading-list/papers/2101.00190/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2101.00190/publication.csl.json)
[Publication record](https://aclanthology.org/2021.acl-long.353/).

## Approaches

- [Parameter-efficient building blocks](https://programasweights.com/reading-list/guides/compilation/#peft): Adapt a frozen model with a small set of learned parameters.

Reading-list version 1.2.0; updated 2026-10-19.


# Learning to Generate Task-Specific Adapters from Task Description

Authors: Qinyuan Ye, Xiang Ren.

Introduces Hypter, which generates adapters from natural-language task descriptions.

ArXiv initial submission: 2021-01-02.

[Original paper](https://arxiv.org/abs/2101.00420) · [Reading-list entry](https://programasweights.com/reading-list/papers/2101.00420/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2101.00420/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2101.00420/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2101.00420/record.json)

**Contribution:** Combines a description encoder and adapter generator with BART. Tests generalization to unseen tasks on ZEST and synthetic SQuAD.

**Program form:** Bottleneck adapters inserted into a text-to-text model.

**Conditioning:** A natural-language task description.

**Execution:** BART performs the task with the generated adapters.

**Relationship to PAW:** An early description-to-adapter method cited by PAW. PAW develops the programming interface for saving and calling the resulting neural functions.

**Limitations:** Training requires a collection of tasks with descriptions and labeled examples.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: Hypter, task descriptions, generated adapters, zero-shot learning.

## Method comparisons

**From generated adapters to reusable functions**

Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.

Sources: [PAW related work](https://arxiv.org/html/2607.02512v1#S10) · [Text-to-LoRA paper](https://proceedings.mlr.press/v267/charakorn25a.html)

Paper details: [Learning to Generate Task-Specific Adapters from Task Description](https://programasweights.com/reading-list/papers/2101.00420/) · [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://programasweights.com/reading-list/papers/2212.10315/) · [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://programasweights.com/reading-list/papers/2211.12485/) · [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/papers/2506.06105/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

Learning to Generate Task-Specific Adapters from Task Description. Qinyuan Ye, Xiang Ren. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2021.
[Publication](https://aclanthology.org/2021.acl-short.82/) · [BibTeX](https://programasweights.com/reading-list/papers/2101.00420/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2101.00420/publication.csl.json)
[Publication record](https://aclanthology.org/2021.acl-short.82/).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.


# LoRA: Low-Rank Adaptation of Large Language Models

Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen.

Adapts a pretrained model with low-rank weight updates while keeping the original weights fixed.

ArXiv initial submission: 2021-06-17.

[Original paper](https://arxiv.org/abs/2106.09685) · [Reading-list entry](https://programasweights.com/reading-list/papers/2106.09685/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2106.09685/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2106.09685/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2106.09685/record.json)

**Contribution:** Reduces the parameters trained and stored for each task. Evaluates adaptation across several language-model families.

**Program form:** Low-rank matrices that update selected model weights.

**Conditioning:** Training examples for the target task.

**Execution:** The language model applies the low-rank updates during inference.

**Relationship to PAW:** PAW uses LoRA as a program representation and generates the adapter with a trained compiler.

**Limitations:** A separate adapter is trained for each task.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: LoRA, low-rank adaptation, parameter-efficient fine-tuning, adapters.

## Method comparisons

**Program form and compiler**

LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.

Sources: [LoRA paper](https://arxiv.org/abs/2106.09685) · [Prefix-tuning paper](https://aclanthology.org/2021.acl-long.353/) · [PAW implementations](https://arxiv.org/html/2607.02512v1#S3)

Paper details: [LoRA: Low-Rank Adaptation of Large Language Models](https://programasweights.com/reading-list/papers/2106.09685/) · [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://programasweights.com/reading-list/papers/2101.00190/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

LoRA: Low-Rank Adaptation of Large Language Models. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. International Conference on Learning Representations, 2022.
[Publication](https://openreview.net/forum?id=nZeVKeeFYf9) · [BibTeX](https://programasweights.com/reading-list/papers/2106.09685/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2106.09685/publication.csl.json)
[Publication record](https://openreview.net/pdf?id=nZeVKeeFYf9).

## Approaches

- [Parameter-efficient building blocks](https://programasweights.com/reading-list/guides/compilation/#peft): Adapt a frozen model with a small set of learned parameters.

Reading-list version 1.2.0; updated 2026-10-19.


# HyperTuning: Toward Adapting Large Language Models without Back-propagation

Authors: Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen.

Produces task-specific parameters for a frozen language model from a few demonstrations.

ArXiv initial submission: 2022-11-22.

[Original paper](https://arxiv.org/abs/2211.12485) · [Reading-list entry](https://programasweights.com/reading-list/papers/2211.12485/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2211.12485/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2211.12485/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2211.12485/record.json)

**Contribution:** Introduces HyperT5 models that generate prefixes or LoRA parameters. Hyperpretraining and multitask training prepare the generator for new tasks.

**Program form:** Generated prefix or LoRA parameters.

**Conditioning:** A few examples of the task.

**Execution:** A frozen T5 model runs with the generated parameters.

**Relationship to PAW:** PAW cites HyperTuning as prior work on generating adaptation parameters from examples.

**Limitations:** The hypermodel accepts a limited number of demonstrations. The paper also studies further task-specific tuning.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: HyperTuning, HyperT5, few-shot adaptation, generated LoRA.

## Method comparisons

**From generated adapters to reusable functions**

Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.

Sources: [PAW related work](https://arxiv.org/html/2607.02512v1#S10) · [Text-to-LoRA paper](https://proceedings.mlr.press/v267/charakorn25a.html)

Paper details: [Learning to Generate Task-Specific Adapters from Task Description](https://programasweights.com/reading-list/papers/2101.00420/) · [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://programasweights.com/reading-list/papers/2212.10315/) · [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://programasweights.com/reading-list/papers/2211.12485/) · [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/papers/2506.06105/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

HyperTuning: Toward Adapting Large Language Models without Back-propagation. Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen. Proceedings of the 40th International Conference on Machine Learning, 2023.
[Publication](https://proceedings.mlr.press/v202/phang23a.html) · [BibTeX](https://programasweights.com/reading-list/papers/2211.12485/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2211.12485/publication.csl.json)
[Publication record](https://proceedings.mlr.press/v202/phang23a.html).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.


# HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation

Authors: Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew Peters.

Turns instructions and examples into reusable prefixes, adapters, and encoded instruction representations.

ArXiv initial submission: 2022-12-20.

[Original paper](https://arxiv.org/abs/2212.10315) · [Reading-list entry](https://programasweights.com/reading-list/papers/2212.10315/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2212.10315/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2212.10315/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2212.10315/record.json)

**Contribution:** Combines a hypernetwork with instruction fusion so repeated calls can reuse the instruction computation. Evaluates zero-shot and few-shot generalization.

**Program form:** Prefixes, adapters, and encoded instruction representations.

**Conditioning:** Task instructions and optional examples.

**Execution:** An encoder-decoder model uses the generated modules and attends to cached instruction representations.

**Relationship to PAW:** HINT reuses generated modules and an encoded instruction. PAW reuses a program containing text and adaptation parameters.

**Limitations:** The generated modules and cached instruction representations are tied to the trained encoder-decoder model.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: HINT, instruction tuning, hypernetwork, instruction fusion.

## Method comparisons

**From generated adapters to reusable functions**

Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.

Sources: [PAW related work](https://arxiv.org/html/2607.02512v1#S10) · [Text-to-LoRA paper](https://proceedings.mlr.press/v267/charakorn25a.html)

Paper details: [Learning to Generate Task-Specific Adapters from Task Description](https://programasweights.com/reading-list/papers/2101.00420/) · [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://programasweights.com/reading-list/papers/2212.10315/) · [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://programasweights.com/reading-list/papers/2211.12485/) · [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/papers/2506.06105/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation. Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew E. Peters. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
[Publication](https://aclanthology.org/2023.acl-long.631/) · [BibTeX](https://programasweights.com/reading-list/papers/2212.10315/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2212.10315/publication.csl.json)
[Publication record](https://aclanthology.org/2023.acl-long.631/).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.


# Learning to Compress Prompts with Gist Tokens

Authors: Jesse Mu, Xiang Lisa Li, Noah Goodman.

Compresses an instruction into a few reusable token activations.

ArXiv initial submission: 2023-04-17.

[Original paper](https://arxiv.org/abs/2304.08467) · [Reading-list entry](https://programasweights.com/reading-list/papers/2304.08467/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2304.08467/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2304.08467/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2304.08467/record.json)

**Contribution:** Trains with an attention mask that forces later tokens to use gist activations in place of the original prompt.

**Program form:** Cached activations at a few gist-token positions.

**Conditioning:** The instruction or prompt.

**Execution:** The language model generates outputs while attending to the cached gist.

**Relationship to PAW:** PAW cites gisting as another way to reuse instruction computation. Gisting stores activations; PAW stores a program with text and adaptation parameters.

**Limitations:** Compression can reduce output quality. The cached activations are tied to the trained model.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: gist tokens, prompt compression, activation caching, amortized inference.

## Method comparisons

**What gets reused?**

Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.

Sources: [Gist paper](https://arxiv.org/abs/2304.08467) · [Generative Adapter paper](https://arxiv.org/abs/2411.05877) · [PAW program definition](https://arxiv.org/html/2607.02512v1#S2)

Paper details: [Learning to Compress Prompts with Gist Tokens](https://programasweights.com/reading-list/papers/2304.08467/) · [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://programasweights.com/reading-list/papers/2411.05877/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

Learning to Compress Prompts with Gist Tokens. Jesse Mu, Xiang Li, Noah Goodman. Advances in Neural Information Processing Systems, 2023.
[Publication](https://proceedings.neurips.cc/paper_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html) · [BibTeX](https://programasweights.com/reading-list/papers/2304.08467/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2304.08467/publication.csl.json)
[Publication record](https://proceedings.neurips.cc/paper_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html).

## Approaches

- [Context compression](https://programasweights.com/reading-list/guides/compilation/#context-compression): Compress an instruction or context once and reuse the result across calls.

Reading-list version 1.2.0; updated 2026-10-19.


# The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators

Authors: Tzu-Heng Huang, Catherine Cao, Vaishnavi Bhargava, Frederic Sala.

Asks a language model to write labeling programs that can be saved and run locally.

ArXiv initial submission: 2024-06-25.

[Original paper](https://arxiv.org/abs/2407.11004) · [Reading-list entry](https://programasweights.com/reading-list/papers/2407.11004/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2407.11004/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2407.11004/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2407.11004/record.json)

**Contribution:** Reuses generated code across many labeling decisions and combines noisy program outputs with weak supervision.

**Program form:** Symbolic labeling programs.

**Conditioning:** A task description, labeling instructions, and a function signature.

**Execution:** Generated programs label examples locally; their outputs can be combined to train a downstream model.

**Relationship to PAW:** PAW evaluates an ALCHEmist-style approach. Both build reusable tools: ALCHEmist produces symbolic code, while PAW produces programs for a neural interpreter.

**Limitations:** Label quality depends on the model’s ability to write useful programs. Complex tasks can produce code that is harder to inspect.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: ALCHEmist, program synthesis, automated labeling, reusable tools.

## Method comparisons

**Code and neural programs**

ALCHEmist generates symbolic labeling programs. PAW compiles fuzzy functions into text and learned parameters. Both build an artifact once and reuse it; one runs as code, the other on a neural interpreter.

Sources: [ALCHEmist paper](https://proceedings.neurips.cc/paper_files/paper/2024/hash/72802bef5cf1a3449e909b20c2ae18d5-Abstract-Conference.html) · [PAW evaluation](https://arxiv.org/html/2607.02512v1#S6)

Paper details: [The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators](https://programasweights.com/reading-list/papers/2407.11004/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. Tzu-Heng Huang, Catherine Cao, Vaishnavi Bhargava, Frederic Sala. Advances in Neural Information Processing Systems, 2024.
[Publication](https://proceedings.neurips.cc/paper_files/paper/2024/hash/72802bef5cf1a3449e909b20c2ae18d5-Abstract-Conference.html) · [BibTeX](https://programasweights.com/reading-list/papers/2407.11004/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2407.11004/publication.csl.json)
[Publication record](https://proceedings.neurips.cc/paper_files/paper/2024/hash/72802bef5cf1a3449e909b20c2ae18d5-Abstract-Conference.html).

## Approaches

- [Programs and reusable artifacts](https://programasweights.com/reading-list/guides/fuzzy-functions/): Learn programs that can be saved and run on new inputs.

Reading-list version 1.2.0; updated 2026-10-19.


# Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Authors: Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng.

Converts new context into low-rank model updates in one forward pass.

ArXiv initial submission: 2024-11-08.

[Original paper](https://arxiv.org/abs/2411.05877) · [Reading-list entry](https://programasweights.com/reading-list/papers/2411.05877/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2411.05877/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2411.05877/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2411.05877/record.json)

**Contribution:** Trains an adapter generator with self-supervision. Tests learning from documents, demonstrations, and conversation history.

**Program form:** Additive low-rank adapters.

**Conditioning:** Documents, demonstrations, or conversation history, including successive chunks of context.

**Execution:** The adapted model answers new inputs using context stored in its parameter updates.

**Relationship to PAW:** PAW cites this context-to-adapter method. Generative Adapter stores information from context; PAW compiles the behavior of a function.

**Limitations:** The adapter generator is tied to its frozen base model.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: Generative Adapter, contextualization, fast weights, streaming context.

## Method comparisons

**What gets reused?**

Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.

Sources: [Gist paper](https://arxiv.org/abs/2304.08467) · [Generative Adapter paper](https://arxiv.org/abs/2411.05877) · [PAW program definition](https://arxiv.org/html/2607.02512v1#S2)

Paper details: [Learning to Compress Prompts with Gist Tokens](https://programasweights.com/reading-list/papers/2304.08467/) · [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://programasweights.com/reading-list/papers/2411.05877/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


An AFM 2024 OpenReview version appeared before the first arXiv submission. [Source](https://arxiv.org/abs/2411.05877).

## Published version

Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass. Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng. International Conference on Learning Representations, 2025.
[Publication](https://openreview.net/forum?id=bc3sUsS6ck) · [BibTeX](https://programasweights.com/reading-list/papers/2411.05877/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2411.05877/publication.csl.json)
[Publication record](https://openreview.net/pdf?id=bc3sUsS6ck).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.
- [Context compression](https://programasweights.com/reading-list/guides/compilation/#context-compression): Compress an instruction or context once and reuse the result across calls.

Reading-list version 1.2.0; updated 2026-10-19.


# Text-to-LoRA: Instant Transformer Adaption

Authors: Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange.

Uses a hypernetwork to generate LoRA adapters from task descriptions.

ArXiv initial submission: 2025-06-06.

[Original paper](https://arxiv.org/abs/2506.06105) · [Reading-list entry](https://programasweights.com/reading-list/papers/2506.06105/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2506.06105/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2506.06105/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2506.06105/record.json)

**Contribution:** Trains hypernetworks to reconstruct existing adapters or learn from supervised task data. Tests adapter compression and generation for unseen tasks.

**Program form:** Generated LoRA matrices.

**Conditioning:** A natural-language task description.

**Execution:** A base transformer performs the task with the generated LoRA.

**Relationship to PAW:** PAW cites Text-to-LoRA as prior work on generating adapters. PAW develops fuzzy-function programming, FuzzyBench, and an interface for reusable neural programs.

**Limitations:** Reconstruction-trained T2L underperforms the supervised variant on unseen tasks.

[Discussion in the paper](https://arxiv.org/html/2607.02512v1#S10)

Related terms: Text-to-LoRA, T2L, task description, adapter generation.

## Method comparisons

**From generated adapters to reusable functions**

Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.

Sources: [PAW related work](https://arxiv.org/html/2607.02512v1#S10) · [Text-to-LoRA paper](https://proceedings.mlr.press/v267/charakorn25a.html)

Paper details: [Learning to Generate Task-Specific Adapters from Task Description](https://programasweights.com/reading-list/papers/2101.00420/) · [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://programasweights.com/reading-list/papers/2212.10315/) · [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://programasweights.com/reading-list/papers/2211.12485/) · [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/papers/2506.06105/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)


## Published version

Text-to-LoRA: Instant Transformer Adaption. Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange. Proceedings of the 42nd International Conference on Machine Learning, 2025.
[Publication](https://proceedings.mlr.press/v267/charakorn25a.html) · [BibTeX](https://programasweights.com/reading-list/papers/2506.06105/publication.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2506.06105/publication.csl.json)
[Publication record](https://proceedings.mlr.press/v267/charakorn25a.html).

## Approaches

- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.


# Program-as-Weights: A Programming Paradigm for Fuzzy Functions

Authors: Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng.

Proposes fuzzy-function programming: compile a natural-language function specification into a neural program, then reuse it locally.

ArXiv initial submission: 2026-07-02.

[Original paper](https://arxiv.org/abs/2607.02512) · [Reading-list entry](https://programasweights.com/reading-list/papers/2607.02512/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2607.02512/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2607.02512/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2607.02512/record.json)

**Contribution:** Develops the compiler–interpreter abstraction and programs that combine text with learned parameters. Introduces the 10M-example FuzzyBench dataset, local execution interfaces, and image-conditioned compilation. Prefix tuning and LoRA are two implementations.

**Program form:** A discrete pseudo-program and a compiler-generated parameter-efficient module.

**Conditioning:** A function specification, optionally with examples. The image extension also accepts visual information.

**Execution:** A frozen lightweight interpreter loads the program and runs new inputs locally.

**Relationship to PAW:** Compile by Training builds on this programming abstraction. It keeps the same interface and refines the fast compiler’s prediction using synthesized examples.

**Limitations:** Each compiler produces programs for a specific interpreter. Compiled parameters are opaque, and FuzzyBench uses synthetic data.

[Discussion in the paper](https://arxiv.org/html/2609.04199v1#S1)
[Code](https://github.com/programasweights)

Related terms: Program-as-Weights, ProgramAsWeights, PAW, fuzzy-function programming, neural compiler, neural interpreter, FuzzyBench.

## Method comparisons

**From generated adapters to reusable functions**

Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.

Sources: [PAW related work](https://arxiv.org/html/2607.02512v1#S10) · [Text-to-LoRA paper](https://proceedings.mlr.press/v267/charakorn25a.html)

Paper details: [Learning to Generate Task-Specific Adapters from Task Description](https://programasweights.com/reading-list/papers/2101.00420/) · [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://programasweights.com/reading-list/papers/2212.10315/) · [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://programasweights.com/reading-list/papers/2211.12485/) · [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/papers/2506.06105/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)

**Code and neural programs**

ALCHEmist generates symbolic labeling programs. PAW compiles fuzzy functions into text and learned parameters. Both build an artifact once and reuse it; one runs as code, the other on a neural interpreter.

Sources: [ALCHEmist paper](https://proceedings.neurips.cc/paper_files/paper/2024/hash/72802bef5cf1a3449e909b20c2ae18d5-Abstract-Conference.html) · [PAW evaluation](https://arxiv.org/html/2607.02512v1#S6)

Paper details: [The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators](https://programasweights.com/reading-list/papers/2407.11004/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)

**What gets reused?**

Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.

Sources: [Gist paper](https://arxiv.org/abs/2304.08467) · [Generative Adapter paper](https://arxiv.org/abs/2411.05877) · [PAW program definition](https://arxiv.org/html/2607.02512v1#S2)

Paper details: [Learning to Compress Prompts with Gist Tokens](https://programasweights.com/reading-list/papers/2304.08467/) · [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://programasweights.com/reading-list/papers/2411.05877/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)

**Program form and compiler**

LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.

Sources: [LoRA paper](https://arxiv.org/abs/2106.09685) · [Prefix-tuning paper](https://aclanthology.org/2021.acl-long.353/) · [PAW implementations](https://arxiv.org/html/2607.02512v1#S3)

Paper details: [LoRA: Low-Rank Adaptation of Large Language Models](https://programasweights.com/reading-list/papers/2106.09685/) · [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://programasweights.com/reading-list/papers/2101.00190/) · [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/)

**Fast prediction and training**

PAW’s fast compiler predicts a program. Compile by Training starts from that prediction, generates examples, and trains an adapter. Both use the same runtime interface; the extra training improves semantic accuracy on FuzzyBench-Hard at a higher compilation cost.

Sources: [PAW compiler implementations](https://arxiv.org/html/2607.02512v1#S3) · [Compile by Training method](https://arxiv.org/html/2609.04199v1#S3) · [Compile by Training evaluation](https://arxiv.org/html/2609.04199v1#S5)

Paper details: [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/) · [Compile by Training: Turning Natural-Language Specifications into Local Neural Functions](https://programasweights.com/reading-list/papers/2609.04199/)


## Approaches

- [Programs and reusable artifacts](https://programasweights.com/reading-list/guides/fuzzy-functions/): Learn programs that can be saved and run on new inputs.
- [Generated weights and adapters](https://programasweights.com/reading-list/guides/compilation/#generated-adapters): Generate model weights from a task description, examples, or context.

Reading-list version 1.2.0; updated 2026-10-19.


# Compile by Training: Turning Natural-Language Specifications into Local Neural Functions

Authors: Yuntian Deng, Pengyu Nie, Stuart Shieber.

Compiles a function description into a reusable local program by generating training examples and optimizing an adapter.

ArXiv initial submission: 2026-09-03.

[Original paper](https://arxiv.org/abs/2609.04199) · [Reading-list entry](https://programasweights.com/reading-list/papers/2609.04199/)
[Preprint BibTeX](https://programasweights.com/reading-list/papers/2609.04199/citation.bib) · [Preprint CSL-JSON](https://programasweights.com/reading-list/papers/2609.04199/citation.csl.json) · [JSON record](https://programasweights.com/reading-list/papers/2609.04199/record.json)

**Contribution:** Makes task-specific training a software build step. Raises FuzzyBench-Hard semantic accuracy (LEM) from 22.4% to 83.6% and demonstrates composition in a website helper, avatar control, and English–Claudish translation.

**Program form:** A LoRA adapter, runtime prompt, specification, and interpreter metadata in a PAW program.

**Conditioning:** The specification guides teacher synthesis. PAW’s fast compiler supplies the initial adapter and runtime prompt.

**Execution:** A frozen Qwen3-0.6B interpreter runs the saved function locally without further teacher calls.

**Relationship to PAW:** Extends PAW with training at compile time while keeping its program format and runtime interface.

**Limitations:** Synthetic examples can inherit teacher errors. Cold compilation takes 50.9 seconds for one specification on a B300. FuzzyBench-Hard selects fast-compiler failures under exact match; the reported accuracy uses an LLM judge.

[Discussion in the paper](https://arxiv.org/html/2609.04199v1#S1)

Versions: [Original version](https://arxiv.org/html/2609.04199v1)
[Code](https://github.com/programasweights/compile-by-training)

Related terms: compile by training, training-based compilation, specification-specific optimization, synthetic supervision, local neural function, FuzzyBench-Hard, ProgramAsWeights.

## Method comparisons

**Fast prediction and training**

PAW’s fast compiler predicts a program. Compile by Training starts from that prediction, generates examples, and trains an adapter. Both use the same runtime interface; the extra training improves semantic accuracy on FuzzyBench-Hard at a higher compilation cost.

Sources: [PAW compiler implementations](https://arxiv.org/html/2607.02512v1#S3) · [Compile by Training method](https://arxiv.org/html/2609.04199v1#S3) · [Compile by Training evaluation](https://arxiv.org/html/2609.04199v1#S5)

Paper details: [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/papers/2607.02512/) · [Compile by Training: Turning Natural-Language Specifications into Local Neural Functions](https://programasweights.com/reading-list/papers/2609.04199/)


The paper reports public-service measurements from May 2026. [Source](https://arxiv.org/abs/2609.04199).

## Approaches

- [Programs and reusable artifacts](https://programasweights.com/reading-list/guides/fuzzy-functions/): Learn programs that can be saved and run on new inputs.
- [Compilation through training](https://programasweights.com/reading-list/guides/compilation/#training-compilation): Generate examples from a function specification and train a reusable neural program.

Reading-list version 1.2.0; updated 2026-10-19.