G2 Hill-Climb — Held-Out Test Results
Both candidates the C-split gate accepted (iter 2, iter 5) show a positive test-split delta — the first time in this benchmark's v4→v5→v6 history that an accepted candidate's F-split gain shows up on sealed, never-touched test tasks.
F-split screens (12 random tasks) ranged −0.03 to +0.17 across the 6 candidates with no consistent relationship to the C-split outcome — exactly the overfitting signal this split was built to catch (see v4/v5).
Iterations 4 and 6 were killed at the F-split rung before ever reaching the confirm split — together they cost $131 of optimizer time against $0 of confirm-split eval, the gate's early-stop working as intended.
F-split screen vs. C-split decision, by iteration
Optimizer spend per iteration
Reward trajectory: seed → champion
Every iteration, F screen and C decision
broke/fixed count confirm-split tasks that moved by more than 2 standard errors (from the run's LEDGER).| Iter | Candidate | Parent | F Δ | F z | C decision | C val | Broke | Fixed | Optimizer $ |
|---|
Reading this table
- Iteration 2 (cand_0002) — the strongest F signal (+0.168, z 1.53) and the first acceptance. Became champion, held it through iteration 4.
- Iteration 5 (cand_0005) — a weaker F signal (+0.078, z 0.65) but still cleared the confirm-split gate, and became the final champion carried into the test-split evaluation.
- Iterations 4 and 6 — negative F deltas, killed at rung 1 before any confirm-split spend. Iteration 4's optimizer actually found the right diagnosis (a confound in the run's own internal estimate — see "What changed") but the candidate it shipped still didn't clear the F screen.
- Iterations 1 and 3 — positive but weak F signals that did not survive the confirm split's group-sequential test.
27 of 66 test tasks improved by more than 0.05, against 6 regressions of the same size — a net gain spread across three of four categories, not one lucky task carrying the mean.
Platform (34 of 66 tasks) moved +5.25pp, more than twice cost's +3.38pp and four times icinga's +2.0pp. Cloud is flat (+0.6pp) — see the per-task table for why.
platform-131 dropped 39.5 points (0.985 → 0.590) — the single largest move in either direction. Worth a follow-up read of that task's transcript before trusting this champion further.
Test reward by category
Task outcome split
Every test task: seed vs. champion
Per-task test results
| Task | Category | Seed | Champion | Δ |
|---|
How to read this tab
PROCESS.md — the optimizer's required, per-iteration explainability record. Quotes are verbatim. "Helped" means the edit is in the final champion (cand_0005, built on cand_0002); "didn't help" means the edit was proposed, and in two cases independently rediscovered, but the candidate carrying it was rejected or killed before reaching test.✓ cand_0002 — iteration 2, ACCEPTED, first champion
parent: seedWhat it changed
- A conclusion checklist in shared_context.md — the single highest-leverage edit of the run. Diagnosis across 284 near-miss trials found 227 of the losses were "the right fact is in a table cell, a heading, or a label — but never stated as a sentence." A six-shape checklist (absence, how-many, number-vs-limit, not-a-measurement, remedy, per-entity) targets exactly that. "The loss is overwhelmingly unstated conclusions, not missing data."
- Rewrote the epoch/elapsed-time procedure in icinga_agent.md to calendar subtraction instead of building an epoch for "now" by eye — verified against 36/36 anchors by executing the arithmetic in Python, not by reasoning about it.
- Error-result repair in cost_agent.md — a result shaped like {"error": ...} was being charted as $0.00; the fix reclassifies it as a failed call and forces a partial label instead of a false total.
Caught and reverted mid-iteration
- A carve-out on a job-log file-path shortcut — a pre-ship check showed it would force an unnecessary extra tool call into a currently-passing task 9 of 10 times. Dropped before shipping.
✓ cand_0005 — iteration 5, ACCEPTED, final champion
parent: cand_0002What it changed
- Fixed a measurement confound before touching the prompt. The run's own internal estimate said the champion was worse than seed (Δ −0.018). Re-deriving per-task from raw rollouts found the opposite sign (+0.024, rising to +0.050 once timeout-corrupted rollouts are excluded) — the simulator's tool timeout fires at 10× different rates between the seed's and the champion's measurement rounds (0.44% vs 4.66%), and a timeout costs −0.27 mean reward. Two prior iterations had reasoned from the confounded number and drawn the opposite conclusion.
- Rewrote the retry rule in shared_context.md. The seed's blanket "never call the same tool with the same parameters twice" forbids re-issuing a call that merely timed out — and a re-issued call returns real data 10 or 11 times out of 11. One transcript closed by telling the user to retry a query the agent itself could have retried, on a task that solves 10/10 when it doesn't time out.
- Deleted two "MANDATORY" instructions that routed every call through query_provisions_db and db_describe_table — tools the harness serves at a measured 0% success rate (302/302 and more, across the whole benchmark, not just the visible tasks). Six further standing contradictions elsewhere in the prompt told the agent to fall back to the same dead tools; all six had to go.
- Fixed a fictional directory layout in icinga_agent.md (groups/<group>/...) that appears in zero of 200 tasks' real seed data, replacing it with the path shape every graded fetch actually uses.
Caught and reverted mid-iteration
- Deleted a documented path-fallback convention as "guessing" — restored after a reviewer showed it is a real convention used in 21 successful fetches with no prior search returning anything.
- Discouraged a search tool for being useful only 22/339 times on average — restored after a reviewer showed it is the graded, expected call on several individual tasks and the edit would have cost ~0.15 reward per rollout there.
- Wrote "whatever you've already stated still counts" when a round-limit cuts an agent off — a reviewer found this is false under a wall-clock cutoff specifically (verified: 2 rollouts scored exactly 0.0 regardless of partial progress). Rewrote to name both limits correctly.
✗ cand_0004 — iteration 4, killed at F-split rung 1
parent: cand_0002What it found (correctly) but didn't help
- This iteration first discovered the timeout-rate confound that cand_0005 later used to correct course — same measurement, same two root causes (unanswered-tool-call handling, routing into dead tools). The diagnosis was right.
- Despite the correct diagnosis, the specific candidate this iteration shipped scored −0.004 on the 12-task F screen against the then-incumbent and was killed before any confirm-split spend — the edit's execution didn't translate the diagnosis into a measurable gain on that screen, whatever the underlying cause.
Lesson carried forward
cand_0005's own PROCESS.md explicitly credits this iteration's diagnosis and rebuilds on it rather than re-deriving from scratch — the diagnosis survived even though the candidate didn't.
✗ cand_0006 — iteration 6, killed at F-split rung 1
parent: cand_0005What didn't help
- The most expensive optimizer call of the run (~3h of drafting) produced the single worst F-split delta of any iteration (−0.032, z −0.33) — built on top of the final champion, with no actionable failure cluster left in focus per the optimizer's own brief, which it later described as "no actionable failures in focus."
- Killed at the F-split rung before touching the confirm split, so it never got a chance to break anything the gate would record — the cheapest possible failure mode, despite being the most expensive candidate to produce.
Lesson
Diminishing returns: once the two highest-leverage harness-level fixes (timeout handling, dead-tool routing) were already shipped in cand_0005, the remaining search space produced noise rather than signal — the gate caught it cheaply.
✗ cand_0001 & cand_0003 — rejected on the confirm split
parents: seed, cand_0002Both showed small positive F-split signals that did not survive the confirm split's group-sequential test — the classic F/C disagreement this gate exists to catch. cand_0001's three fixed tasks (icinga-204, platform-126, platform-158) were preserved by copying its files forward into cand_0002 rather than reverting to seed, per the run's own JOURNAL guidance to treat a near-miss as a power problem, not a design problem.
v6_g2_e1 · hill-climb algorithm, promotion_rule: P2, split_mode: pooled ·
confirm-split group-sequential looks at n = 16/33/67/133/200 · stall: 0 (iterations never auto-stopped on
consecutive rejects) · test split sealed and scored exactly once, after the champion (cand_0005) was fixed
by iteration 5's acceptance · test-split final.json reconstructed from completed rollouts after the
finalize driver was interrupted by a host-level LSF fault partway through the (optional) train/val bookend —
the sealed test scoring itself completed and was logged before that fault, and was not re-run.
Full writeup: results/v6/v6_g2_e1/summary.md ·
generated ledger: results.json ·
per-candidate process records: source/process/.
This page's data block is generated by
scripts/build_v6_g2_dashboard.py from
results.json — never hand-typed.