Parsec v6 benchmark · 200 tasks · train / validation / test = 67 / 67 / 66

Merged task-wise edits match Global optimization on the sealed test

Each of 34 training tasks got its own optimized bundle (T-opt). We merged those 34 edits into one bundle without looking at any score (T-mrg). The merge by order and the smart merge both score the same as the Global bundle on the 66 sealed test tasks. This reverses our earlier hypothesis, from the v4 pilot, that independently optimized English edits would not compose.

Arm Train67 tasks Validation67 tasks Test (sealed)66 tasks
Seedoriginal skill bundle 0.83010 trials/task 0.8214 trials/task 0.81310 trials/task
Task-wise (T)
T-opt34 private bundles + Seed for the rest; not deployable 0.937+0.107 ± 0.0175 trials/task not measured not measured
T-mrg · merge by orderdiffs applied in gain order, later wins (git -X theirs) 0.902+0.072 ± 0.0185 trials/task not used 0.858+0.046 ± 0.0155 trials/task
T-mrg · smart mergean LLM merged all 34 edits by meaning, without seeing any score 0.908+0.078 ± 0.0215 trials/task not used 0.854+0.041 ± 0.0165 trials/task
T-mrg · non-conflicting hunkskeeps only the 91 of 188 hunks no other task touches 0.880+0.050 ± 0.0155 trials/task not used 0.828+0.015 ± 0.0185 trials/task
Global (G)
Global · cand_0005one shared bundle, gated on validation 0.884+0.053 ± 0.0174 trials/task 0.814−0.005 ± 0.0163 trials/task (gate) 0.850+0.037 ± 0.01310 trials/task
+0.00 gain over Seed, 95% CI above zero +0.00 not distinguishable from zero Δ is paired by task against the Seed; ± is the standard error of the per-task difference.

What the table shows

  • On test, comparing task by task: merge by order − Global = +0.009 ± 0.014, smart merge − Global = +0.004 ± 0.012. Both are within noise of Global.
  • Dropping every contested hunk loses most of the gain. The value is in edits several tasks wanted, mostly shared rules.
  • Global's validation gain over the Seed is not distinguishable from zero, yet its test gain is positive. The ungated merges did as well as Global on test.
  • Cost to build the deployable bundle, before test: $319–334 for T (T-opt plus a $0–14 merge) against $758 for Global (optimizer, train screens and sweeps, validation gates). Shared Seed measurements are excluded from both. Full breakdown in Table 2.

Before we lean on it

  • The Seed was not re-measured for T-mrg. Its Δ compares against Seed trials from the Global run, days earlier. A same-day Seed run on test costs about $40–80.
  • All three merge methods were scored on the same test split, so no method can be chosen because it scored best there.
  • Global's validation figure comes from the 3-trial gate run that accepted cand_0005, compared with a 4-trial Seed remeasurement.
  • One run per arm. The merged bundles are 2–4× the Seed's size and 1.3–2.4× Global's, and move more tasks both up and down.