Paper
The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators
First arXiv submission: · NeurIPS 2024
Asks a language model to write labeling programs that can be saved and run locally.
What it does
Reuses generated code across many labeling decisions and combines noisy program outputs with weak supervision.
- Conditioning
- A task description, labeling instructions, and a function signature.
- Program form
- Symbolic labeling programs.
- Execution
- Generated programs label examples locally; their outputs can be combined to train a downstream model.
Related work
PAW evaluates an ALCHEmist-style approach. Both build reusable tools: ALCHEmist produces symbolic code, while PAW produces programs for a neural interpreter.
Limitations
Label quality depends on the model’s ability to write useful programs. Complex tasks can produce code that is harder to inspect.
Related terms: ALCHEmist, program synthesis, automated labeling, reusable tools.
Related methods
Code and neural programs
ALCHEmist generates symbolic labeling programs. PAW compiles fuzzy functions into text and learned parameters. Both build an artifact once and reuse it; one runs as code, the other on a neural interpreter.
Cite this paper
Published version
Tzu-Heng Huang, Catherine Cao, Vaishnavi Bhargava, Frederic Sala. The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. Advances in Neural Information Processing Systems, 2024.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2407.11004, initial submission 2024-06-25.