Optimization target: the parsec aap2 sub-agent's SKILL.md
(capabilities: [system-prompt]). Agent: Sonnet-4. Harness: Harbor tasks against a
kaegis simulated AAP2/GitHub/Babylon/provisions-db fixture set.
Two independent benchmark generations are shown, and they are not comparable to each other
— different tasks, different scoring schema, different trial counts:
v1 — 30 tasks extracted from real prod0 AAP2 job traces; reward =
gate · (0.5·trajectory + 0.5·assertions); 2 of the 30 went through cap-evolve.
v2 — 10 hand-authored tasks with per-task answer contracts (schema bench/v2);
reward = gate · (wtc·tool_calls + wans·answer) with weights that
vary per task; all 10 went through cap-evolve.
Click any task name for its report — what changed, what worked, what didn't, what we can
(or can't) learn. See reports/README.md.
SKILL.md is
byte-identical across all nine runs on this page, so every seed measurement below is a
measurement of the same artifact. On v2's ten tasks it measured
0.910 · 0.876 · 0.749 · 0.662 · 0.643 depending on trial count and run — a spread of
0.267, against a headline accepted improvement of +0.086. v1 reached the
same conclusion first and wrote it down ("N=10 is not enough") after seeing its own seed move
0.203 between two n=10 measurements. v2 then ran at n=9 with the significance gate
loosened from gate_k_se: 1.0 to 0.2. The v2 improvement is
inside its own baseline's noise band. Full derivation:
results/v2/summary.md, and the ladder table at the
bottom of this page.
Badges — traj-match baseline trajectory = 1.0 (the seed already
reproduces the golden tool sequence) · multi-tool it does not ·
gate-failed baseline completion = 0.0 ·
capped 0.5 perfect trajectory, zero assertions — the 50/50
weighting caps the reward at 0.5 ·
tool gap the contract demands a tool no simulator served, so
trajectory: 1.0 was unreachable and the reward capped at 0.5 by the harness ·
solved baseline reward = 1.0 ·
optimized entered a cap-evolve run ·
baseline-only baselined but never optimized.
All 10 tasks are in train and val; test is empty. The
report.md for this run nevertheless prints "Held-out test (optimized skills): 0.0" and
"the improvement above is on held-out tasks the optimizer never saw" — that is an evaluation of an
empty split, and it cost 2 × 2700 s of wall clock. There is no test column here because there is no test.
seed/SKILL.md is identical in all nine runs
(sha256 ed6e8920…7411, 484 lines). Each row below is therefore a measurement of the
same artifact. Rows are grouped by experiment and ordered as the runs happened. The
spread row is the max−min within each (experiment, split) group — read it against
the accepted Δ for that experiment.