parsec — per-task heatmap

Optimization target: the parsec aap2 sub-agent's SKILL.md (capabilities: [system-prompt]). Agent: Sonnet-4. Harness: Harbor tasks against a kaegis simulated AAP2/GitHub/Babylon/provisions-db fixture set. Two independent benchmark generations are shown, and they are not comparable to each other — different tasks, different scoring schema, different trial counts:
v1 — 30 tasks extracted from real prod0 AAP2 job traces; reward = gate · (0.5·trajectory + 0.5·assertions); 2 of the 30 went through cap-evolve. v2 — 10 hand-authored tasks with per-task answer contracts (schema bench/v2); reward = gate · (wtc·tool_calls + wans·answer) with weights that vary per task; all 10 went through cap-evolve.
Click any task name for its report — what changed, what worked, what didn't, what we can (or can't) learn. See reports/README.md.

Read the v2 section with this in front of you. The seed SKILL.md is byte-identical across all nine runs on this page, so every seed measurement below is a measurement of the same artifact. On v2's ten tasks it measured 0.910 · 0.876 · 0.749 · 0.662 · 0.643 depending on trial count and run — a spread of 0.267, against a headline accepted improvement of +0.086. v1 reached the same conclusion first and wrote it down ("N=10 is not enough") after seeing its own seed move 0.203 between two n=10 measurements. v2 then ran at n=9 with the significance gate loosened from gate_k_se: 1.0 to 0.2. The v2 improvement is inside its own baseline's noise band. Full derivation: results/v2/summary.md, and the ladder table at the bottom of this page.

v1 — 30 trace-extracted tasks

Badges — traj-match baseline trajectory = 1.0 (the seed already reproduces the golden tool sequence) · multi-tool it does not · gate-failed baseline completion = 0.0 · capped 0.5 perfect trajectory, zero assertions — the 50/50 weighting caps the reward at 0.5 · tool gap the contract demands a tool no simulator served, so trajectory: 1.0 was unreachable and the reward capped at 0.5 by the harness · solved baseline reward = 1.0 · optimized entered a cap-evolve run · baseline-only baselined but never optimized.

0.00
1.00

v2 — 10 hand-authored tasks

All 10 tasks are in train and val; test is empty. The report.md for this run nevertheless prints "Held-out test (optimized skills): 0.0" and "the improvement above is on held-out tasks the optimizer never saw" — that is an evaluation of an empty split, and it cost 2 × 2700 s of wall clock. There is no test column here because there is no test.

0.00
1.00

The seed ladder — every measurement of the same byte-identical skill

seed/SKILL.md is identical in all nine runs (sha256 ed6e8920…7411, 484 lines). Each row below is therefore a measurement of the same artifact. Rows are grouped by experiment and ordered as the runs happened. The spread row is the max−min within each (experiment, split) group — read it against the accepted Δ for that experiment.