Parsec · results/v6/README.md source

parsec v6

v6 is a unified task set — v4's original 34 tasks plus v5's held-out 166 (merged from both v5_2026-10-10 and v5_2026-10-12 drops), renamed under a bench-v6- prefix, 200 tasks total, 4 categories (platform, icinga, cloud, cost) — and the first parsec arm family to run with a genuine 3-way hidden split: feedback (F, visible to the optimizer), confirm (C, hidden, the only split the accept/reject gate reads), and test (sealed, scored exactly once).

It exists to answer the question every prior parsec arm (v1/v2/v4/h4/v5) could not: does an accepted candidate's gain survive genuinely held-out data, rather than just the split the optimizer was tuned against. v4's G2 arm never completed its one FINAL test-split attempt; v5's arms ran train == val == test with no holdout at all.

Shared setup

Arms

arm scope result links
g2 (v6_g2_e1) global: one shared 8-file bundle evolved jointly, hill-climb, 6 iterations sealed test-split reward 0.8496 vs. seed's 0.8126 — +0.0369 (3.69pp), the first parsec arm with a positive, completed, held-out test-split result summary.md · results.json · cost_time/ · heatmap · per-candidate process · recipe · reports · artifacts
T-opt (v6_t2_e1) task-wise: 34 independent per-task hill-climbs, each on a private copy of the bundle, train split only 34/34 optimized tasks reach 1.0; train mean 0.830 → 0.937 when each task uses its own champion (fit, 34 bundles, not deployable) summary.md · results.json · heatmap · process · recipe · champions
T-mrg (v6_t4_e1) task-wise: the 34 T-opt edits merged blind into one bundle, three ways; scored on train and the sealed test, 5 trials sealed test: merge by order +0.046, smart merge +0.041, non-conflicting +0.015 over Seed; merge by order and smart merge equal Global (+0.037) within noise, at 42–44% of Global's build cost summary.md · core tables · rendered · per-task · bundles

T vs G: the four core tables

v6_t4_e1/RESULTS_FOR_PAPER.md holds the four tables that compare all arms (Seed, T-opt, the three T-mrg merges, Global): reward on train / validation / test; cost, tokens and compute time; skill edits in words; and the Rule 4 running example (each arm's text, with upstream PARSEC PR #57). Rendered: reward · cost · words · Rule 4.

The headline reverses the hypothesis carried over from the v4 pilot: independently optimized English edits did compose. A score-blind merge of the 34 task-local edits matches the Global bundle on the sealed test split. It cost 42–44% of Global to build, but produces a bundle 1.3–2.4× larger than Global's. Caveat: T-mrg's comparison uses the Seed measured during the Global run, not re-measured alongside it.