CapEvolve · Parsec v6 Benchmark

G2 Hill-Climb — Held-Out Test Results

Six optimizer iterations over a shared, 8-file multi-agent prompt bundle (orchestrator + 6 domain agents + shared context), gated on a hidden confirm split, then scored once on a sealed 66-task test set never shown to the optimizer.
66 test tasks · 10 trials each claude-sonnet-4-6 agent claude-opus-5 optimizer
⚠
No competitor frameworks yet. This run only compares CapEvolve's optimized bundle (cand_0005) against the unmodified seed prompt — there is no GEPA/AutoPrompt/other-framework arm here. Add that comparison before citing this as framework-vs-framework evidence.
Test reward (champion)
84.96%
cand_0005, 66 held-out tasks
Improvement vs. seed
+3.69 pp
81.26% → 84.96%, sealed test split
Accepted candidates
2 of 6
iterations 2 & 5 passed the C-split gate
Total run cost
$1,164
$809 eval + $355 optimizer, 6 iterations
The gain generalizes

Both candidates the C-split gate accepted (iter 2, iter 5) show a positive test-split delta — the first time in this benchmark's v4→v5→v6 history that an accepted candidate's F-split gain shows up on sealed, never-touched test tasks.

F and C disagree, as designed

F-split screens (12 random tasks) ranged −0.03 to +0.17 across the 6 candidates with no consistent relationship to the C-split outcome — exactly the overfitting signal this split was built to catch (see v4/v5).

Two iterations died cheap

Iterations 4 and 6 were killed at the F-split rung before ever reaching the confirm split — together they cost $131 of optimizer time against $0 of confirm-split eval, the gate's early-stop working as intended.

F-split screen vs. C-split decision, by iteration

Paired delta on the 12-task feedback screen (bars) against what happened on the hidden confirm split (color)

Optimizer spend per iteration

Cost of drafting each candidate, independent of its outcome

Reward trajectory: seed → champion

The three numbers that matter for the paper — not iteration-by-iteration noise, but where the final champion lands relative to the untouched seed, on each split

Every iteration, F screen and C decision

F = paired delta on the 12-task feedback rung vs. the then-incumbent, measured before any confirm-split spend. C = what the group-sequential test on the hidden confirm split decided. broke/fixed count confirm-split tasks that moved by more than 2 standard errors (from the run's LEDGER).
IterCandidateParent F ΔF z C decisionC val BrokeFixed Optimizer $

Reading this table

  • Iteration 2 (cand_0002) — the strongest F signal (+0.168, z 1.53) and the first acceptance. Became champion, held it through iteration 4.
  • Iteration 5 (cand_0005) — a weaker F signal (+0.078, z 0.65) but still cleared the confirm-split gate, and became the final champion carried into the test-split evaluation.
  • Iterations 4 and 6 — negative F deltas, killed at rung 1 before any confirm-split spend. Iteration 4's optimizer actually found the right diagnosis (a confound in the run's own internal estimate — see "What changed") but the candidate it shipped still didn't clear the F screen.
  • Iterations 1 and 3 — positive but weak F signals that did not survive the confirm split's group-sequential test.
Broad, not concentrated

27 of 66 test tasks improved by more than 0.05, against 6 regressions of the same size — a net gain spread across three of four categories, not one lucky task carrying the mean.

Platform carries the gain

Platform (34 of 66 tasks) moved +5.25pp, more than twice cost's +3.38pp and four times icinga's +2.0pp. Cloud is flat (+0.6pp) — see the per-task table for why.

One real regression

platform-131 dropped 39.5 points (0.985 → 0.590) — the single largest move in either direction. Worth a follow-up read of that task's transcript before trusting this champion further.

Test reward by category

Seed vs. champion (cand_0005), mean reward per category

Task outcome split

Of 66 test tasks, by whether the champion moved the reward by more than 0.05

Every test task: seed vs. champion

Points above the diagonal improved; below it regressed. Marker color is category.

Per-task test results

TaskCategorySeedChampionΔ

How to read this tab

Pulled from each candidate's own PROCESS.md — the optimizer's required, per-iteration explainability record. Quotes are verbatim. "Helped" means the edit is in the final champion (cand_0005, built on cand_0002); "didn't help" means the edit was proposed, and in two cases independently rediscovered, but the candidate carrying it was rejected or killed before reaching test.

✓ cand_0002 — iteration 2, ACCEPTED, first champion

parent: seed
+0.168F-split delta
0.950confirm-split val
6 / 0fixed / broke
$49optimizer cost

What it changed

  • A conclusion checklist in shared_context.md — the single highest-leverage edit of the run. Diagnosis across 284 near-miss trials found 227 of the losses were "the right fact is in a table cell, a heading, or a label — but never stated as a sentence." A six-shape checklist (absence, how-many, number-vs-limit, not-a-measurement, remedy, per-entity) targets exactly that. "The loss is overwhelmingly unstated conclusions, not missing data."
  • Rewrote the epoch/elapsed-time procedure in icinga_agent.md to calendar subtraction instead of building an epoch for "now" by eye — verified against 36/36 anchors by executing the arithmetic in Python, not by reasoning about it.
  • Error-result repair in cost_agent.md — a result shaped like {"error": ...} was being charted as $0.00; the fix reclassifies it as a failed call and forces a partial label instead of a false total.

Caught and reverted mid-iteration

  • A carve-out on a job-log file-path shortcut — a pre-ship check showed it would force an unnecessary extra tool call into a currently-passing task 9 of 10 times. Dropped before shipping.

✓ cand_0005 — iteration 5, ACCEPTED, final champion

parent: cand_0002
+0.078F-split delta
0.873confirm-split val
+3.69pptest-split delta
$58optimizer cost

What it changed

  • Fixed a measurement confound before touching the prompt. The run's own internal estimate said the champion was worse than seed (Δ −0.018). Re-deriving per-task from raw rollouts found the opposite sign (+0.024, rising to +0.050 once timeout-corrupted rollouts are excluded) — the simulator's tool timeout fires at 10× different rates between the seed's and the champion's measurement rounds (0.44% vs 4.66%), and a timeout costs −0.27 mean reward. Two prior iterations had reasoned from the confounded number and drawn the opposite conclusion.
  • Rewrote the retry rule in shared_context.md. The seed's blanket "never call the same tool with the same parameters twice" forbids re-issuing a call that merely timed out — and a re-issued call returns real data 10 or 11 times out of 11. One transcript closed by telling the user to retry a query the agent itself could have retried, on a task that solves 10/10 when it doesn't time out.
  • Deleted two "MANDATORY" instructions that routed every call through query_provisions_db and db_describe_table — tools the harness serves at a measured 0% success rate (302/302 and more, across the whole benchmark, not just the visible tasks). Six further standing contradictions elsewhere in the prompt told the agent to fall back to the same dead tools; all six had to go.
  • Fixed a fictional directory layout in icinga_agent.md (groups/<group>/...) that appears in zero of 200 tasks' real seed data, replacing it with the path shape every graded fetch actually uses.

Caught and reverted mid-iteration

  • Deleted a documented path-fallback convention as "guessing" — restored after a reviewer showed it is a real convention used in 21 successful fetches with no prior search returning anything.
  • Discouraged a search tool for being useful only 22/339 times on average — restored after a reviewer showed it is the graded, expected call on several individual tasks and the edit would have cost ~0.15 reward per rollout there.
  • Wrote "whatever you've already stated still counts" when a round-limit cuts an agent off — a reviewer found this is false under a wall-clock cutoff specifically (verified: 2 rollouts scored exactly 0.0 regardless of partial progress). Rewrote to name both limits correctly.

✗ cand_0004 — iteration 4, killed at F-split rung 1

parent: cand_0002
−0.004F-split delta
killednever reached confirm split
$22optimizer cost

What it found (correctly) but didn't help

  • This iteration first discovered the timeout-rate confound that cand_0005 later used to correct course — same measurement, same two root causes (unanswered-tool-call handling, routing into dead tools). The diagnosis was right.
  • Despite the correct diagnosis, the specific candidate this iteration shipped scored −0.004 on the 12-task F screen against the then-incumbent and was killed before any confirm-split spend — the edit's execution didn't translate the diagnosis into a measurable gain on that screen, whatever the underlying cause.

Lesson carried forward

cand_0005's own PROCESS.md explicitly credits this iteration's diagnosis and rebuilds on it rather than re-deriving from scratch — the diagnosis survived even though the candidate didn't.

✗ cand_0006 — iteration 6, killed at F-split rung 1

parent: cand_0005
−0.032F-split delta
killednever reached confirm split
$109optimizer cost, highest of the run

What didn't help

  • The most expensive optimizer call of the run (~3h of drafting) produced the single worst F-split delta of any iteration (−0.032, z −0.33) — built on top of the final champion, with no actionable failure cluster left in focus per the optimizer's own brief, which it later described as "no actionable failures in focus."
  • Killed at the F-split rung before touching the confirm split, so it never got a chance to break anything the gate would record — the cheapest possible failure mode, despite being the most expensive candidate to produce.

Lesson

Diminishing returns: once the two highest-leverage harness-level fixes (timeout handling, dead-tool routing) were already shipped in cand_0005, the remaining search space produced noise rather than signal — the gate caught it cheaply.

✗ cand_0001 & cand_0003 — rejected on the confirm split

parents: seed, cand_0002
+0.061 / +0.019F-split delta
rejectboth, group-sequential test
$83 / $34optimizer cost

Both showed small positive F-split signals that did not survive the confirm split's group-sequential test — the classic F/C disagreement this gate exists to catch. cand_0001's three fixed tasks (icinga-204, platform-126, platform-158) were preserved by copying its files forward into cand_0002 rather than reverting to seed, per the run's own JOURNAL guidance to treat a near-miss as a power problem, not a design problem.

Run v6_g2_e1 · hill-climb algorithm, promotion_rule: P2, split_mode: pooled · confirm-split group-sequential looks at n = 16/33/67/133/200 · stall: 0 (iterations never auto-stopped on consecutive rejects) · test split sealed and scored exactly once, after the champion (cand_0005) was fixed by iteration 5's acceptance · test-split final.json reconstructed from completed rollouts after the finalize driver was interrupted by a host-level LSF fault partway through the (optional) train/val bookend — the sealed test scoring itself completed and was logged before that fault, and was not re-run. Full writeup: results/v6/v6_g2_e1/summary.md · generated ledger: results.json · per-candidate process records: source/process/. This page's data block is generated by scripts/build_v6_g2_dashboard.py from results.json — never hand-typed.