Parsec · results/v6/v6_g2_e1/summary.md source

parsec v6 — G2 arm: global-scope, shared-bundle hill-climb with a held-out test split

Agent model: claude-sonnet-4-6 Optimizer model: claude-opus-5 (optimizer_skill: claude-code, algorithm_skill: hill-climb) Budget (per source/state.json): max_iterations: 6 (reached), stall: 0 (never auto-stopped on consecutive rejects — overridden mid-run; see "Why 6 iterations" below), max_usd: 2500.0, max_optimizer_usd: 600.0, neither exhausted. Split discipline: split_mode: pooled, promotion_rule: P2 — a genuine 3-way split, unlike every prior parsec arm's train == train == val convention. 200 tasks total, split into feedback (F, 67 tasks, visible to the optimizer), confirm (C, 67 tasks, hidden — the only split the accept/reject gate reads), and test (66 tasks, sealed, scored exactly once at the end). Ledger: results.json (built by ../../../scripts/build_v6_g2_results_json.py from the vendored source/ files) Heatmap: ../../../ui/heatmap_v6_g2_e1.html — test split only (see below for why) Artifacts: ../../../artifacts/v6/v6_g2_e1/ (seed/, accepted/cand_0002/, best/ = cand_0005, rejected/cand_{0001,0003,0004,0006}/) Per-task reports: ../../../reports/v6/v6_g2_e1/ (<task>.md, 66 files, test split only — generated auto+diff blocks, no hand-written narrative yet) Cost/time source: cost_time/ (v6_g2_e1_results_table.md) Per-candidate process records: source/process/ (each candidate's own PROCESS.md — the optimizer's required, per-iteration explainability record) and source/LEDGER.md (the framework's own factual accept/reject record) Run dir: .capevolve/v6_g2_e1/run_g2_e1/ in the parsec-intake_v6 worktree (not committed here — see source/ for the vendored subset this report is built from)

The one-line result

v6_g2_e1 is the first parsec arm to show an accepted candidate's gain survive a genuinely held-out, sealed test split. After 6 iterations (2 accepted, 2 rejected on the confirm split, 2 killed before reaching it), the final champion cand_0005 scores 0.8496 on the sealed 66-task test split against the unmodified seed's 0.8126 — +0.0369 (3.69pp), measured once, after the champion was fixed, with no further tuning possible.

F (feedback, visible) C (confirm, hidden gate) test (sealed, final)
seed 0.8300 (67 tasks, 10 trials) 0.8213 (full re-confirmation, 268 rollouts) 0.8126 (66 tasks, 10 trials)
final champion (cand_0005) 0.8730* 0.8046 (full re-confirmation, 268 rollouts) 0.8496
Δ vs seed +0.0430* −0.0167 +0.0369

*cand_0005's F figure is its 12-task rung-1 screen score, not a full-67-task measurement (the F split is never scored whole per iteration, by design — see below); it is not directly comparable to seed's full-split 0.8300.

The headline caveat, stated plainly: the full-sample re-confirmation of cand_0005 on C (268 rollouts, all 67 confirm tasks) actually comes in below the seed (0.8046 vs 0.8213, −0.0167) — the group-sequential test that accepted it ran on a smaller, early-stopped sample (the C split's own val figure, 0.8728, computed from the sequential test's own paired statistic, not the full re-confirmation). This is the same pattern seen in prior parsec arms (v4/v5): an accept decision made on a partial sample doesn't always hold up under the full re-measurement. What's different here, and the actual news in this arm, is that the test-split number — sealed, never touched by the optimizer, scored exactly once — is unambiguously positive. Whichever of the C-split numbers you trust, the sealed test result stands on its own.

Why there's no per-task F/C ledger

Every prior parsec arm's results.json has one row per task, with a column per candidate, because every candidate was scored on the same split every iteration (v4_g_e1: 34 tasks, every iteration). v6_g2_e1 cannot do this for F or C:

The only split with a stable, full per-task ledger is test — scored exactly once, on all 66 tasks, for exactly two candidates (seed and the final champion). That's what results.json's task_ledger and the heatmap show. See ../../../scripts/build_v6_g2_results_json.py's own docstring for the full account.

Per-iteration record

See cost_time/v6_g2_e1_results_table.md for the full table (F-screen delta/z, C decision, fixed/broke counts, optimizer cost per iteration). Summary:

Iter Candidate C decision What it changed
1 cand_0001 reject A conclusion checklist draft, epoch-conversion repair, cost error-handling — all later folded into cand_0002.
2 cand_0002 accept (false — see caveat 5) The conclusion checklist (shipped), the icinga epoch/elapsed-time rewrite, cost error-result repair. First champion, but its acceptance came from a degraded seed baseline batch.
3 cand_0003 reject Weak F signal (+0.019), didn't survive the confirm-split test.
4 cand_0004 killed at F rung 1 Correctly diagnosed a measurement confound (a 10× different tool-timeout rate between the seed's and the champion's measurement rounds) but the specific candidate shipped still scored below the F-screen threshold.
5 cand_0005 accept Fixed the confound cand_0004 found; rewrote the retry rule for timed-out tool calls; deleted two "MANDATORY" instructions routing to tools served at a measured 0% success rate; fixed a fictional directory layout in icinga_agent.md. Final champion.
6 cand_0006 killed at F rung 1 The most expensive optimizer call of the run (−0.032, the worst F-screen result of any iteration) — diminishing returns once the two highest-leverage fixes were already in cand_0005.

Full narrative for every iteration, including what was tried and reverted mid-iteration, is in each candidate's own source/process/<cand>_PROCESS.md.

Test-split result, by category

category n seed champion Δ
cloud 9 0.7607 0.7666 +0.0059
cost 10 0.8314 0.8652 +0.0338
icinga 13 0.7801 0.8001 +0.0200
platform 34 0.8333 0.8859 +0.0525
overall 66 0.8126 0.8496 +0.0369

27 of 66 tasks improved by more than 0.05, 6 regressed by more than 0.05, 33 moved less than that (not evidence of either direction at this sample size). The single largest move in either direction is platform-131-deployment-first-resource-blind-spot, which regressed 0.3950 (0.985 → 0.590) — worth a transcript read before trusting this champion further; it is not explained by anything in cand_0005's own PROCESS.md, which did not touch that task's cluster.

Why 6 iterations, not a clean budget stop

This run was extended twice beyond its original plan, on explicit instruction, specifically because the first two iterations' results (iteration 1 rejected, iteration 2 accepted) looked too thin to draw a conclusion from: stall was raised from 2 to 0 mid-run (so a second consecutive reject would not auto-stop it), the iteration budget was raised from 5 to 6 after iteration 4 (a cheap rung-1 kill) suggested there was more signal to find, and the run was explicitly continued through iterations 5 and 6 regardless of outcome. cand_0006's rejection (the most expensive iteration, worst result) was the agreed stopping point.

Data-quality caveats

  1. The sealed test-split final.json was reconstructed from completed rollouts, not re-run. The driver that scored test (LSF job 1702040) finished both evaluations — champion as FINAL, seed as FINAL_seed, both logged in source/events.jsonl with final rewards — then crashed (exit 255, apparent host-level LSF fault) while starting the optional post-test train/val bookend. final.json was built by re-reading the 1,320 already-written test rollout files on disk (the same function cap-evolve's own --resume path uses, split_result_from_rollouts()), not by re-scoring test — the test seal was committed only after this reconstruction, and source/final.json's own reconstruction_note field records this. The train/val bookend never ran and is not included in any report here.
  2. No per-task F/C ledger exists, by design — see "Why there's no per-task F/C ledger" above. Do not expect this arm's heatmap to show F or C columns the way v4_g_e1's does.
  3. cand_0005's full re-confirmation on C (0.8046) is below the seed's (0.8213), even though the group-sequential test that accepted it read a positive signal on a smaller sample — see "The one-line result" above. Read the test-split number as the arm's real headline; the confirm-split numbers disagree with each other depending on sample size.
  4. A real bug in the shared report-generator helper was found and fixed while building this arm's reports: v4_report_common.py's build_report() used re.sub(diff_block, text) with diff_block as a plain string replacement — re.sub interprets backslash sequences in a string replacement as regex escapes, so a diffed file containing the literal two-character sequence \n (as text, e.g. a worked example quoting a JSON/YAML string) was silently turned into a real newline on every regeneration. Two of cand_0005's per-task reports (cloud-319-billing-lab-unknown-to-babylon.md, cost-050-partial-account-coverage.md) hit this, because shared_context.md's cache-example worked text happens to contain \n literally. Fixed by passing a lambda instead of a string to re.sub (confirmed as a no-op on every already-committed v4/v5 report — none of them diff on a line containing a literal backslash sequence).

  5. cand_0002's acceptance was false, caused by one degraded seed baseline batch. Before judging a candidate, the gate re-measures the champion with four trials on each of the 67 confirmation tasks, run as four batches. In cand_0002's round (r001) the seed's batches averaged 0.401, 0.748, 0.802, 0.760. In batch 1 the seed scored more than 0.3 below its own score in the other three batches on 47 of 65 tasks. Those trials completed and returned verifier output, so the infrastructure-error rule did not reject them. Averaging the batch in lowered the baseline: the gate's paired statistic was +0.100, z 5.14 (accept; boundary about 2.0). Recomputed on the same 128 cand_0002 trials without batch 1, it is +0.005, z 0.26 — a reject. The full post-run re-measurement agrees: cand_0002 0.804 vs seed 0.821. Recomputation is by an outside script, not a cap-evolve output; the inputs are the sealed rollouts/confirm/*__r001__seed__ctrl_* and *__r001__cand_0002__seq* trials.

What this changes. cand_0005 was built from cand_0002 (93% of the lines cand_0002 added survive verbatim) and accepted for beating it. cand_0002's own baseline batches in that round were consistent (0.763, 0.771, 0.814, 0.763), so cand_0005's acceptance is sound. The sealed test result compares cand_0005 with the seed directly and is unaffected. What cannot be known is whether cand_0005 would have been found had cand_0002 been rejected. The cause of batch 1's degradation is not established. 6. platform-131 regressed 0.985 → 0.590, beyond the 0.20 per-task deployability limit. cand_0005 added a rule to babylon_agent.md that list_anarchy_subjects with a guid "resolves without a cluster". That is true of the simulator, but this task's reward contract requires the call with cluster: merlin, so nine of ten trials scored 0 on tool calls; the agent also stopped before opening the AAP2 job the request asks for. Whether the bundle or the contract should change needs domain review.

Next moves (open)