Parsec · results/v4/v4_t_e1/summary.md source

parsec v4 — T arm: task-by-task hill-climb

Spec: docs/specs/2026-09-21-parsec-v4-experiment-plan-design.md (branch parsec-v4-docs) Agent model: claude-sonnet-4-6 Optimizer model: claude-opus-5 (optimizer_skill: claude-code, algorithm_skill: hill-climb) Budget (per task): num_trials: 5, max_iterations: 3, stall: 2, max_usd: 50.0, max_optimizer_usd: 20.0 Ledger: results.json, generated by ../../../scripts/build_v4_t_results_json.py Heatmap: ../../../ui/heatmap_v4_t.html Recipes: ../../../recipes/v4/v4_t_e1/ Artifacts: ../../../artifacts/v4/v4_t_e1/ Per-task reports: ../../../reports/v4/v4_t_e1/ (<task>.md, 34 files) Cost/time source: cost_time/ (v4_t2_e1_results_table.md/.xlsx, v4_t4_e1_results_table.md) T vs. G: see ../comparison.md

The one-line result

Task-by-task optimization (T2) raises the mean reward over both the JB baseline and our own zero-shot baseline, on both tranches — but it only ran on 21 of the 34 tasks, by design (see "Coverage" below). Statically merging those 21 optimized bundles into one shared bundle (T4) and re-evaluating all 34 tasks zero-shot is a mixed result: it raises the challenge-tranche mean but regresses 8 individual tasks, including 5 of the 13 tasks that were already perfect before the merge (see "T4" below) — the exact failure mode the regression check exists to catch (spec §3). T3 and T5 haven't run yet; how T compares to the G arm's global-scope shared-bundle hill-climb (already run, v4_g2_e1) is in ../comparison.md, not here.

Aggregate: T1 seed vs. T2 final

tranche n JB reward (mean) our T1 seed (mean) our T2 final (mean)* Δ vs JB Δ vs T1
regression 30 0.833 0.845 0.963 +0.131 +0.119
challenge 4 0.730 0.792 0.873 +0.143 +0.081

*T2's final column falls back to the T1 seed score for the 13 tasks T2 never targeted (they were already at ceiling) — this is not a clean "optimized-only" average. Never pool the two tranches above into one number — this convention is carried from the spec's own §0 table and the 3x2_toy pilot; every aggregate in this document keeps regression and challenge separate for that reason.

These numbers are the sections.task.summaries block of results.json, unchanged from parsec-intake_v4's own results/parsec34/results.json (Task 1 vendored it verbatim).

Coverage: T2 ran on 21 of 34 tasks, by design

T2 wasn't cut short by time or resources. It deliberately only targeted tasks where the seed bundle hadn't already scored a perfect 1.0 on our own baseline run (our_baseline):

category tasks already-perfect on seed (skipped) T2-optimized
platform 19 6 13
icinga 8 4 4
cloud 4 2 2
cost 3 1 2
total 34 13 21

All 13 skipped tasks are in the regression tranche (0 challenge) — every challenge task was T2-optimized. Since those 13 tasks were already at ceiling, there is no missing optimized content to merge in for them — but T4 (and, later, C3/C4) must still evaluate every task in their group, including these 13, specifically to catch a regression that merging other tasks' edits into the shared bundle could introduce (spec §3, §4). T4 has now run this check on all 34 tasks — see "T4" below: 5 of these 13 already-perfect tasks did in fact regress.

Cost + wall clock (T2 task-by-task optimization)

Source: cost_time/v4_t2_e1_results_table.md, parsed and cross-validated by build_v4_t_results_json.py into results.json's task_ledger[].t2_* fields (per task, null for the 13 tasks T2 never targeted) and sections.task.cost_time (this table). Only T2 made real optimizer calls per task — T4's cost/time (below) is all zero-shot eval, no optimizer loop, and T3/T5 haven't run at all.

tranche n tasks total cost total tokens total time eval cost optimizer cost
regression 17 $309.48 13,464,546 103,207s (28.7h) $36.56 $272.92
challenge 4 $127.76 9,566,513 37,660s (10.5h) $27.65 $100.11
both, summed* 21 $437.24 23,031,059 140,868s (39.1h) $64.21 $373.03

*The "both, summed" row is an operational total (how much T2 cost, full stop), not a scientific comparison — unlike the reward aggregate above, it isn't a claim that regression and challenge are commensurable. Read the two tranche rows, not the sum, for anything you'd compare across arms.

The optimizer, not evaluation, dominates cost: ~85% of total spend (\$373 of \$437) is the optimizer proposing candidates, not the runner scoring them — evaluation itself is comparatively cheap. Per-task cost ranges from \$1.39 (cost-030-threshold-not-an-anomaly — seed already scored 1.0, so T2 made no optimizer calls at all, only the required seed/FINAL evals) up to \$43.57 (platform-033-schema-change-not-the-oom, 2 accepted iterations on a task with a large context). See cost_time/v4_t2_e1_results_table.md's Summary table for all 21 tasks, and its Per-iteration detail table for the eval-vs-optimizer split at every individual seed/candidate/test eval — each per-task report also carries its own line (e.g. reports/v4/v4_t_e1/cloud-024-guid-to-account.md).

The same 98 detail-table rows, grouped by iteration stage instead of by tranche (sections.task.cost_time.by_stage in results.json, built by parse_cost_time_by_stage() in build_v4_t_results_json.py):

stage n total cost total tokens total time eval cost optimizer cost
seed 21 $12.91 3,946,752 17,423s (4.8h) $12.91 $0.00
iter1 20 $237.49 6,433,879 53,233s (14.8h) $14.40 $223.09
iter2 13 $137.36 3,650,765 33,083s (9.2h) $8.15 $129.21
iter3 3 $22.22 701,518 4,217s (1.2h) $1.48 $20.74
FINAL 21 $13.32 4,056,544 13,304s (3.7h) $13.32 $0.00
FINAL_seed 20 $13.94 4,241,601 19,608s (5.4h) $13.94 $0.00
TOTAL 98 $437.24 23,031,059 140,868s (39.1h) $64.21 $373.03

n counts how many of the 21 T2-optimized tasks reached that stage, not tasks overall — every row's TOTAL still matches the "both, summed" row above exactly (same 98 source rows, grouped differently). seed and FINAL run for all 21 tasks by construction (the required baseline and held-out-test evals). Only cost-030-threshold-not-an-anomaly skips iter1 entirely — its seed already scored a perfect 1.0, so T2 made zero optimizer calls — which is also why iter2/iter3 shrink to 13 and 3 tasks as fewer candidates kept improving enough to justify another round. FINAL_seed (a test-time re-measurement of the seed bundle, giving a seed-vs-final comparison point) is skipped only when the seed itself is already the winning bundle — true for cost-030 alone — so it appears for 20 of 21 tasks. As with the tranche table, read each stage row on its own; the TOTAL row is an operational sum, not a scientific comparison.

T4: merge (union of T2 bundles), zero-shot on all 34 tasks — the regression check

v4_t4_e1 statically unions all 21 T2-optimized bundles' final skill files into one shared 8-file bundle, then evaluates it zero-shot (no further optimization, n=5 trials) on all 34 tasks — including the 13 tasks T2 never targeted — specifically to check whether merging independently optimized edits regresses a task that was fine before (spec §3). Source: cost_time/v4_t4_e1_results_table.md, parsed into results.json's task_ledger[].t4_* fields (present for all 34 tasks) and sections.task.t4_summary (this table).

tranche n T4 reward (mean) our T1 baseline (mean) Δ vs our baseline
regression 30 0.8305 0.8446 −0.0142
challenge 4 0.8776 0.7918 +0.0858

This compares against our_baseline_mean across the full 34 tasks, not T2's smaller 21-task final_mean from the aggregate table above — that's the comparison the regression check is built to make (t4_delta_vs_our_baseline in the per-task ledger; see the note in build_v4_t_results_json.py).

The tranche means hide two opposite effects at the per-task level: 18 of 34 tasks improved, 8 are unchanged, and 8 regressed:

Cost/time: $42.78 total, 13,601,491 tokens, 295.2 min (4.92h) wall time across all 34 single-eval, no-optimizer runs — far cheaper than T2's $437.24/39.1h, since each T4 task is one zero-shot eval rather than an optimizer loop. See cost_time/v4_t4_e1_results_table.md for the per-task breakdown.

This is the merge strategy described in reports/v4-t4-merge-decisions.md (spec §4) — a naive static union with no conflict resolution beyond what that report documents. It's resolved for T4; whether the same strategy holds up for the still-undesigned T5 (which continues optimizing jointly after the merge, and so has a chance to correct these regressions) is still open.

Data-quality caveats

  1. Three tasks each have two finalized T2 runs: platform-005-wrong-owner-trap, platform-007-directory-path-fetch, platform-008-log-does-not-say, all on 2026-09-20, because an unrelated infrastructure issue (a team LLM API budget cap, since fixed) interrupted the first attempt on each and it was re-run. results.json's task_ledger row for each picks whichever run had the best test_reward, tie-broken by latest — not simply "the latest run." See each task's report (platform-005, platform-007, platform-008) for which run won and why; the cost/time figures above are from that same winning run in every case.
  2. seed, cand_0001-3, and best in the ledger are validation-split measurements (n=5 trials per eval); final is the held-out test-split measurement for whichever candidate won on validation (same task, same split as train/val by design — see ../../../recipes/v4/v4_t_e1/README.md on why there is no independent holdout this phase). jb_reward (n=1) and our_baseline (n=3) are separate, earlier measurements from different runs (baseline.md and v4_t1_e1 respectively) — four different ns can appear in one task's row. Read no cell without checking its n and split, exactly as reports/README.md's existing caution already states for v1/v2.

Next moves (open)

For the cross-arm next move (CCC/LSF parallel migration), see ../README.md.