DocumentationThree open-source repositories power the benchmark. Pick the docs you need — run agents against the benchmark, see how tasks are mined, or query the live data behind the leaderboard.
Run frontier LLM agents against the FormulaCode benchmark. Spins up reproducible Docker environments, replays the unit-test suite, and computes per-workload speedup, advantage, and stratified scores. Bring your own Terminus or OpenHands agent — `fceval run -d formulacode -a <your-agent>` is all it takes.
The four-stage pipeline that mines FormulaCode's tasks from real GitHub repositories: scraping high-quality performance PRs, attribute-filtering them with LLM judges, synthesising reproducible build environments, and running the statistical-significance tests that admit a candidate into the benchmark.
Two subdomains expose the live task and run database. Uptime is not guaranteed — these are research endpoints, sometimes rebuilt mid-week. For reproducible evaluation, prefer the static CSV that ships with this site.
Read-only Supabase REST. Tables: repositories, pull_requests, candidate_containers, harbor_runs. Anonymous key required (see fc-eval docs).
Browseable Supabase Studio with the live task and run tables. Useful for ad-hoc inspection and SQL.