SpreadsheetBench verified_400: cap-evolve vs 9 optimizers

Issue #895 · Gemma-4-31B agent · GLM-5.3 optimizer · R = 2,160 rollouts per search · k = 3 trials · 3 seeds per system · 280 sealed test tasks

How to read the gain. Each test window scores the frozen champion and the seed on the same 280 tasks in the same window; the gain is the mean of champion − seed over tasks. Use it rather than the absolute score: the seed's own score ranged 0.592–0.657 across the 33 windows. Ranks 4–8 lie within 0.010 of each other while one system's seeds differ by an SD of 0.016–0.081, so with 3 seeds only the top and bottom groups are separated. Validation is optimistic: champions were chosen on it.

One row per system

Means over the system's 3 seeds. Click a column to sort; click a row for its seeds.

Sort:

Every seed

33 searches. * = inert seed: the search returned the seed itself as champion.

Cost and time

Tokens and times come from the experiment's own metering gateway. No dollar figures: there is no price list for these endpoints, so a cost in $ would have to be invented. Optimizer tokens are GLM-5.3 tokens; agent tokens are Gemma tokens. Search time is elapsed wall time, not effort: it includes periods when searches were frozen to relieve the model endpoint, and retry waits after endpoint failures, so it mostly reflects how the run was scheduled. GEPA s2 has no search time (its resumed job left no end marker). Test time is the window's elapsed time from first to last episode (1,680 episodes), so a window that was paused and resumed counts its pause; CapEvolve v1 s0's window, for example, ran in several resumed stretches.

Means over each system's 3 seeds. Click a column to sort.