Parsec · results/v4/v4_t_e1/summary.md source
parsec v4 — T arm: task-by-task hill-climb
Spec: docs/specs/2026-09-21-parsec-v4-experiment-plan-design.md (branch parsec-v4-docs)
Agent model: claude-sonnet-4-6
Optimizer model: claude-opus-5 (optimizer_skill: claude-code, algorithm_skill: hill-climb)
Budget (per task): num_trials: 5, max_iterations: 3, stall: 2, max_usd: 50.0, max_optimizer_usd: 20.0
Ledger: results.json, generated by ../../../scripts/build_v4_t_results_json.py
Heatmap: ../../../ui/heatmap_v4_t.html
Recipes: ../../../recipes/v4/v4_t_e1/
Artifacts: ../../../artifacts/v4/v4_t_e1/
Per-task reports: ../../../reports/v4/v4_t_e1/ (<task>.md, 34 files)
Cost/time source: cost_time/ (v4_t2_e1_results_table.md/.xlsx, v4_t4_e1_results_table.md)
T vs. G: see ../comparison.md
The one-line result
Task-by-task optimization (T2) raises the mean reward over both the JB baseline and our own
zero-shot baseline, on both tranches — but it only ran on 21 of the 34 tasks, by design (see
"Coverage" below). Statically merging those 21 optimized bundles into one shared bundle (T4) and
re-evaluating all 34 tasks zero-shot is a mixed result: it raises the challenge-tranche mean but
regresses 8 individual tasks, including 5 of the 13 tasks that were already perfect before the merge
(see "T4" below) — the exact failure mode the regression check exists to catch (spec §3). T3 and T5
haven't run yet; how T compares to the G arm's global-scope shared-bundle hill-climb (already run,
v4_g2_e1) is in ../comparison.md, not here.
Aggregate: T1 seed vs. T2 final
| tranche | n | JB reward (mean) | our T1 seed (mean) | our T2 final (mean)* | Δ vs JB | Δ vs T1 |
|---|---|---|---|---|---|---|
| regression | 30 | 0.833 | 0.845 | 0.963 | +0.131 | +0.119 |
| challenge | 4 | 0.730 | 0.792 | 0.873 | +0.143 | +0.081 |
*T2's final column falls back to the T1 seed score for the 13 tasks T2 never targeted (they
were already at ceiling) — this is not a clean "optimized-only" average. Never pool the two
tranches above into one number — this convention is carried from the spec's own §0 table and the
3x2_toy pilot; every aggregate in this document keeps regression and challenge separate for that reason.
These numbers are the sections.task.summaries block of results.json, unchanged from
parsec-intake_v4's own results/parsec34/results.json (Task 1 vendored it verbatim).
Coverage: T2 ran on 21 of 34 tasks, by design
T2 wasn't cut short by time or resources. It deliberately only targeted tasks where the seed
bundle hadn't already scored a perfect 1.0 on our own baseline run (our_baseline):
| category | tasks | already-perfect on seed (skipped) | T2-optimized |
|---|---|---|---|
| platform | 19 | 6 | 13 |
| icinga | 8 | 4 | 4 |
| cloud | 4 | 2 | 2 |
| cost | 3 | 1 | 2 |
| total | 34 | 13 | 21 |
All 13 skipped tasks are in the regression tranche (0 challenge) — every challenge task was
T2-optimized. Since those 13 tasks were already at ceiling, there is no missing optimized content
to merge in for them — but T4 (and, later, C3/C4) must still evaluate every task in their group,
including these 13, specifically to catch a regression that merging other tasks' edits into the
shared bundle could introduce (spec §3, §4). T4 has now run this check on all 34 tasks — see "T4"
below: 5 of these 13 already-perfect tasks did in fact regress.
Cost + wall clock (T2 task-by-task optimization)
Source: cost_time/v4_t2_e1_results_table.md, parsed and
cross-validated by build_v4_t_results_json.py into results.json's task_ledger[].t2_* fields
(per task, null for the 13 tasks T2 never targeted) and sections.task.cost_time (this table).
Only T2 made real optimizer calls per task — T4's cost/time (below) is all zero-shot eval, no
optimizer loop, and T3/T5 haven't run at all.
| tranche | n tasks | total cost | total tokens | total time | eval cost | optimizer cost |
|---|---|---|---|---|---|---|
| regression | 17 | $309.48 | 13,464,546 | 103,207s (28.7h) | $36.56 | $272.92 |
| challenge | 4 | $127.76 | 9,566,513 | 37,660s (10.5h) | $27.65 | $100.11 |
| both, summed* | 21 | $437.24 | 23,031,059 | 140,868s (39.1h) | $64.21 | $373.03 |
*The "both, summed" row is an operational total (how much T2 cost, full stop), not a scientific comparison — unlike the reward aggregate above, it isn't a claim that regression and challenge are commensurable. Read the two tranche rows, not the sum, for anything you'd compare across arms.
The optimizer, not evaluation, dominates cost: ~85% of total spend (\$373 of \$437) is the
optimizer proposing candidates, not the runner scoring them — evaluation itself is comparatively
cheap. Per-task cost ranges from \$1.39 (cost-030-threshold-not-an-anomaly — seed already
scored 1.0, so T2 made no optimizer calls at all, only the required seed/FINAL evals) up to
\$43.57 (platform-033-schema-change-not-the-oom, 2 accepted iterations on a task with a large
context). See cost_time/v4_t2_e1_results_table.md's Summary
table for all 21 tasks, and its Per-iteration detail table for the eval-vs-optimizer split at every
individual seed/candidate/test eval — each per-task report also carries its own line (e.g.
reports/v4/v4_t_e1/cloud-024-guid-to-account.md).
The same 98 detail-table rows, grouped by iteration stage instead of by tranche
(sections.task.cost_time.by_stage in results.json, built by
parse_cost_time_by_stage() in build_v4_t_results_json.py):
| stage | n | total cost | total tokens | total time | eval cost | optimizer cost |
|---|---|---|---|---|---|---|
| seed | 21 | $12.91 | 3,946,752 | 17,423s (4.8h) | $12.91 | $0.00 |
| iter1 | 20 | $237.49 | 6,433,879 | 53,233s (14.8h) | $14.40 | $223.09 |
| iter2 | 13 | $137.36 | 3,650,765 | 33,083s (9.2h) | $8.15 | $129.21 |
| iter3 | 3 | $22.22 | 701,518 | 4,217s (1.2h) | $1.48 | $20.74 |
| FINAL | 21 | $13.32 | 4,056,544 | 13,304s (3.7h) | $13.32 | $0.00 |
| FINAL_seed | 20 | $13.94 | 4,241,601 | 19,608s (5.4h) | $13.94 | $0.00 |
| TOTAL | 98 | $437.24 | 23,031,059 | 140,868s (39.1h) | $64.21 | $373.03 |
n counts how many of the 21 T2-optimized tasks reached that stage, not tasks overall — every row's
TOTAL still matches the "both, summed" row above exactly (same 98 source rows, grouped differently).
seed and FINAL run for all 21 tasks by construction (the required baseline and held-out-test
evals). Only cost-030-threshold-not-an-anomaly skips iter1 entirely — its seed already scored a
perfect 1.0, so T2 made zero optimizer calls — which is also why iter2/iter3 shrink to 13 and 3
tasks as fewer candidates kept improving enough to justify another round. FINAL_seed (a test-time
re-measurement of the seed bundle, giving a seed-vs-final comparison point) is skipped only when the
seed itself is already the winning bundle — true for cost-030 alone — so it appears for 20 of 21
tasks. As with the tranche table, read each stage row on its own; the TOTAL row is an operational sum,
not a scientific comparison.
T4: merge (union of T2 bundles), zero-shot on all 34 tasks — the regression check
v4_t4_e1 statically unions all 21 T2-optimized bundles' final skill files into one shared 8-file
bundle, then evaluates it zero-shot (no further optimization, n=5 trials) on all 34 tasks —
including the 13 tasks T2 never targeted — specifically to check whether merging independently
optimized edits regresses a task that was fine before (spec §3). Source:
cost_time/v4_t4_e1_results_table.md, parsed into
results.json's task_ledger[].t4_* fields (present for all 34 tasks) and
sections.task.t4_summary (this table).
| tranche | n | T4 reward (mean) | our T1 baseline (mean) | Δ vs our baseline |
|---|---|---|---|---|
| regression | 30 | 0.8305 | 0.8446 | −0.0142 |
| challenge | 4 | 0.8776 | 0.7918 | +0.0858 |
This compares against our_baseline_mean across the full 34 tasks, not T2's smaller 21-task
final_mean from the aggregate table above — that's the comparison the regression check is built to
make (t4_delta_vs_our_baseline in the per-task ledger; see the note in build_v4_t_results_json.py).
The tranche means hide two opposite effects at the per-task level: 18 of 34 tasks improved, 8 are unchanged, and 8 regressed:
- 5 of the 13 already-perfect seed tasks regressed — merging in other tasks' optimizer edits
broke something these tasks depended on, exactly the failure mode this arm exists to catch:
platform-018-find-jobs-window(1.0 → 0.028, −0.972),platform-019-anarchysubject-state(1.0 → 0.32, −0.68),platform-020-ocpv-vm-inventory(1.0 → 0.34, −0.66),cost-028-highest-spend-provider(1.0 → 0.4, −0.6),platform-009-catalog-item-no-config(1.0 → 0.825, −0.175). - 3 of the 21 T2-optimized tasks also slipped when their own optimized edits were combined with
20 other tasks' edits instead of standing alone:
cloud-026-gpu-abuse-triage(0.886 → 0.613, −0.273),cost-030-threshold-not-an-anomaly(0.953 → 0.72, −0.233),platform-032-shared-secret-not-a-registry-outage(0.65 → 0.645, −0.005, within noise). - The 18 improved tasks are mostly the remaining T2-optimized tasks, whose own edits still apply
unchanged inside the union and score as they did standing alone — e.g.
cloud-024-guid-to-account(0.2 → 1.0, +0.8),platform-003-tojson-dict-literal-rca(0.478 → 1.0, +0.522).
Cost/time: $42.78 total, 13,601,491 tokens, 295.2 min (4.92h) wall time across all 34
single-eval, no-optimizer runs — far cheaper than T2's $437.24/39.1h, since each T4 task is one
zero-shot eval rather than an optimizer loop. See
cost_time/v4_t4_e1_results_table.md for the per-task
breakdown.
This is the merge strategy described in reports/v4-t4-merge-decisions.md (spec §4) — a naive
static union with no conflict resolution beyond what that report documents. It's resolved for T4;
whether the same strategy holds up for the still-undesigned T5 (which continues optimizing jointly
after the merge, and so has a chance to correct these regressions) is still open.
Data-quality caveats
- Three tasks each have two finalized T2 runs:
platform-005-wrong-owner-trap,platform-007-directory-path-fetch,platform-008-log-does-not-say, all on 2026-09-20, because an unrelated infrastructure issue (a team LLM API budget cap, since fixed) interrupted the first attempt on each and it was re-run.results.json'stask_ledgerrow for each picks whichever run had the besttest_reward, tie-broken by latest — not simply "the latest run." See each task's report (platform-005,platform-007,platform-008) for which run won and why; the cost/time figures above are from that same winning run in every case. seed,cand_0001-3, andbestin the ledger are validation-split measurements (n=5trials per eval);finalis the held-out test-split measurement for whichever candidate won on validation (same task, same split astrain/valby design — see../../../recipes/v4/v4_t_e1/README.mdon why there is no independent holdout this phase).jb_reward(n=1) andour_baseline(n=3) are separate, earlier measurements from different runs (baseline.mdandv4_t1_e1respectively) — four differentns can appear in one task's row. Read no cell without checking itsnand split, exactly asreports/README.md's existing caution already states for v1/v2.
Next moves (open)
- A. Design T3 (transfer) — spec §2 describes it but it isn't designed yet: one task's T2 bundle, evaluated zero-shot on a different task.
- B. Run T5. Designed in the spec (§2) but not submitted. C1/G1 need no new job — they're
already derivable from T1 and are computed directly into
results.jsonbybuild_v4_t_results_json.py. - C. Decide whether T4's merge strategy (a naive static union,
reports/v4-t4-merge-decisions.md, spec §4) is good enough for T5, or whether the 8 regressions found above (5 of them on previously-perfect tasks) call for a different conflict-resolution approach before it runs. T5 gets a chance T4 didn't — continued joint optimization after the merge — so it may self-correct some of these; that's untested until T5 runs.
For the cross-arm next move (CCC/LSF parallel migration), see ../README.md.