FormulaCode Benchmark Dataset

Explore the FormulaCode dataset — 1,075 benchmark workloads

This is a preview slice of the FormulaCode workload dataset, while we work on api.formulacode.org. Every workload is indexed by repository, optimization level, and task. Click a card to see the benchmark code, per-agent speedups, and a link to the agent's terminal session recording.

1,075

Workloads

3

Repositories

3

Tasks

46%

Agent > Oracle

1,075 matches Showing 1–24

Citation

@misc{sehgal2025formulacode,
    title={Evaluating Agentic Optimization on Large Codebases}, 
    author={Atharva Sehgal and James Hou and Akanksha Sarkar and Ishaan Mantripragada and Swarat Chaudhuri and Jennifer J. Sun and Yisong Yue},
    year={2026},
    eprint={2603.16011},
    archivePrefix={arXiv},
    primaryClass={cs.SE},
    url={https://arxiv.org/abs/2603.16011}, 
}

Acknowledgements

This work was supported in part by a Slingshot Award from the Laude Institute, NSF awards III-#2505097, PPoSS-#2316161, NSF #2505096, NSF #2505098, and gifts from Point72 and OpenAI.