SpreadsheetBench · README.md source

Report pages

Raw results data, recipes, artifacts and presentations are not rendered on this site; links to them open the files in the source repo on GitHub. Internal cluster paths and host names are masked on this site.

spreadsheetbench

Results, recipes, and artifacts from running cap-evolve and nine competitor optimizers on SpreadsheetBench verified_400, per the experiment plan in cap-evolve issue #895.

This repo was migrated on 2026-11-10 from the spreadsheet-history orphan branch of skillberry-ai/cap-evolve, with full commit history preserved. It is one of the per-benchmark result repos in the skillberry-runs org (alongside skillsbench and parsec). Clone it like any repo: git clone git@github.com:skillberry-runs/spreadsheetbench.git.

Why this repo exists

cap-evolve produces library code (adapters, scoring, CI), which belongs on main as reviewed PRs, and a result narrative: what a run found, which candidates it tried, what a number does and does not mean. The narrative has no home on main and none on cap-evolve's benchmark-history (machine-written and pruned). This repo is where it lives.

The experiment

benchmark SpreadsheetBench verified_400: 80 train, 40 val, 280 official test tasks (recipes/split_ids.json)
agent / optimizer Gemma-4-31B (instruction-tuned) / GLM-5.3
systems CapEvolve v1, CapEvolve v2, GEPA, GEPA-full, MIPROv2, SkillOpt, EvoSkill, Trace2Skill, Meta-Harness, WikiSkill, ACE
search budget R = 2,160 counted rollouts per search, k = 3 trials per task, 3 seeds per system (33 searches)
test one window per search on the 280 test tasks: 1,680 episodes, the frozen champion (840) and the seed (840) scored in the same window
headline paired per-task gain (champion − same-window seed), with a CR1 standard error clustered by task

Deviations from the plan in #895 (each recorded in docs/decisions.md). The main ones: - The seed is re-scored in every test window and the gain is paired within the window, rather than comparing against a single seed run. - 14 of the 33 windows ran the champion's 840 episodes first and then the seed's (ACE, GEPA s0/s2, GEPA-full, MIPROv2, WikiSkill). The other 19 interleaved the two in one shuffled order. Both orders use the same registered task list and the same pairing. - CapEvolve v2 is CapEvolve at a later main commit (optimizer instructions "Improvement 2", gate k_se 0.2); see docs/HANDOFF_capevolve_v2.md.

Result

Full tables (train, val, test, sorted by gain and by absolute score): reports/results_tables.md. Per-seed figures: results/full_table.json.

system mean test gain per seed (s0 / s1 / s2) SD across seeds
GEPA-full +0.129 +0.145 / +0.090 / +0.152 0.034
SkillOpt +0.119 +0.098 / +0.123 / +0.134 0.018
GEPA +0.118 +0.179 / +0.100 / +0.075 0.054
WikiSkill +0.097 +0.089 / +0.086 / +0.115 0.016
MIPROv2 +0.096 +0.093 / +0.135 / +0.062 0.036
CapEvolve v2 +0.095 +0.075 / +0.068 / +0.143 0.041
CapEvolve v1 +0.090 +0.142 / +0.066 / +0.061 0.045
EvoSkill +0.087 −0.006* / +0.143 / +0.123 0.081
Trace2Skill +0.054 0.000* / +0.069 / +0.093 0.048
Meta-Harness +0.052 +0.041 / +0.060 / +0.055 0.010
ACE +0.040 +0.015 / +0.105 / 0.000* 0.057

* Inert seed: the search ended with the seed itself as its champion, so its window compares the seed with itself (Trace2Skill s0, EvoSkill s0, ACE s2). They are counted as about 0. In ACE s2 every candidate playbook carried a helper script that the common pre-rollout helper gate rejects, so all three of its validation evaluations scored 0.0 and the seed stayed champion.

How to read it. - Use the paired gain. The seed's own test score ranged 0.592–0.657 across the 33 windows (SD 0.018), so absolute champion scores partly reflect which window they ran in. - Only the ends are separated. Ranks 4–8 lie within 0.010 of each other in mean gain, while a single system's seeds differ by an SD of 0.016–0.081. With 3 seeds that middle order is not a ranking. - Validation figures are optimistic: each champion was selected on val.

Layout

directory what's in it
results/<system>/s<seed>/search/ per search: run_summary.json (stop reason, champion, budget used), run_meta.txt (host, code sha, settings), budget_probe.json (the R-counter check), isolation_scan.json, val_evals.jsonl, train_ids.txt, val_ids.txt. CapEvolve keeps its own run state, so its searches have run_main/ instead (champion, baseline, history, eval index, journal, insights).
results/<system>/s<seed>/test/ per test window: summary.json (paired gain), summary_champion_only.json (champion-first windows), registry.json (champion and seed file hashes, order hash, registered before the first episode), per_task_test_scores.json (per-task mean for champion and seed)
results/full_table.json the per-seed train/val/test figures behind the tables
artifacts/seed/, artifacts/<system>/s<seed>/champion/ the seed capability and each frozen champion exactly as scored; the file hashes match registry.json
recipes/<system>/ each system's arm code and LSF launcher, BUILD_RECORD.json and deviations_<system>.json (where our port differs from the published method); recipes/common/ is the shared rollout and budget layer; CapEvolve's are its project templates
scripts/ the test-window runner, the metering gateway, the automatic test launcher, the LSF wrappers, the table builder, and the script that staged this content
docs/decisions.md the append-only decision log of the whole experiment, including every incident and relaunch
reports/ the final tables
ui/results.html a self-contained, sortable results page: one row per system, every seed, and a cost and time table (tokens and wall time from the gateway ledgers in results/usage.json); built by scripts/build_results_page.py

Every headline number can be re-derived from this repo alone: each summary.json gain equals the mean over tasks of champion − seed in the matching per_task_test_scores.json.

Not in this repo

The raw per-rollout and per-episode ledgers, the gateway call ledgers, the LSF logs and the full run folders are kept off-repo in cap-evolve-archive/spreadsheetbench-exp895/. The benchmark data itself is not stored anywhere here, because it contains the gold answers.

Caveats