Parsec · reports/v2/bench-aap2-001-single-job-outcome.md source
bench-aap2-001-single-job-outcome
Status: OPTIMIZED · Experiment: v2 (hand-authored) · Category: aap2 / single-job-outcome · Run: run_20260818_161550
Prompt given to the agent:
What was the outcome of AAP2 job 90142 on controller prod0, and which job template ran it?
Contract (tests/expected.json, schema bench/v2): 1 expected tool call · 3 required answer facts · weights tool_calls 0.2 / answer 0.8.
| measurement | split | n | reward | |
|---|---|---|---|---|
| baseline sweep | all |
1 | 1.000 | single trial, no optimizer — v2 reference sweep |
seed |
val |
3 | 0.844 | cap-evolve baseline on val — reused from .capevolve/run_20260816_202942, not re-measured in this run |
seed_train |
train |
9 | 0.500 | the seed re-measured inside the iteration, at the iteration's trial count |
seed_train (errored trials as 0.0) |
train |
9 | 0.444 | same trials, other denominator — this is the convention cap-evolve printed for v2 |
cand_0001 |
train |
9 | 1.000 | accepted champion |
Best: cand_0001 @ 1.000 · Δ vs seed: +0.156 · Iterations spent (whole run): 1 · Harness-errored trials in this task's cells: 1
Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.
Material:
- results/v2/tasks.json — this task's full contract: expected calls, required/derived/forbidden facts, per-task weights
- results/v2/per_task_scores.csv — the hand-maintained per-task analysis, including a note per task per iteration. Its numbers use the errored-as-zero denominator, so they can differ from the table above; compare like with like
- run_20260818_161550/report.md — the run's own write-up
- run_20260818_161550/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words
- run_20260818_161550/events.jsonl — the split aggregates every number above was reconciled against
- results/v2/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null
- artifacts/v2/best/ vs artifacts/v2/seed/ — the accepted champion against the seed
- ui/heatmap.html — this row in context of all 40 tasks
What changed
This is the task the whole v2 iteration was designed around, and it is the one clear, large, mechanistically explained win in this branch.
The champion cand_0001 made six edits to SKILL.md. Every one of them is about which tool to
call, and three of them target this task directly:
- Rule 2 relaxed. The seed demanded that the last text output always be "the full structured analysis (config trace table, failure analysis, root cause, recommendations)". The champion adds: for a simple metadata question — outcome / template / status / timing — "a short report stating the job_id, status, and template_name IS the full report".
- Investigation Flow step 4 and the Tips bullet rewritten. The seed said, twice and in bold,
"Always use
get_job_loginstead ofget_job." The champion replaces both with an action-selection rule:get_jobfor metadata,get_job_logfor log text, and both when a question needs both. - A new Tips bullet naming this task's own shape: "for a 'what was the outcome / which template'
question about a specific job ID, the minimum-viable call is a single
get_job… the answer must state the numericjob_id, thestatusstring and thetemplate_nameverbatim."
cand_0001 was accepted at val 0.962, Δ +0.086, and cap-evolve's RESULT line names this task
in its fixed={} set. Here that verdict is correct.
What worked
Everything, all nine times. cand_0001 scored 1.000 on 9 of 9 train trials. The seed, at
the same n=9, scored 0.500 (errored trial dropped) — a +0.500 gain, the largest single-task
delta anywhere in either experiment.
The reward decomposes exactly, and the decomposition is the finding. With weights
tool_calls 0.2 / answer 0.8, one expected call and three required facts, only a few values are
reachable, and each observed value has a unique solution:
| observed | expected calls matched | answer facts | trials |
|---|---|---|---|
| 0.267 | 0 of 1 | 1 of 3 | 1 (seed) |
| 0.533 | 0 of 1 | 2 of 3 | 7 (seed) + 1 (seed, val) |
| 1.000 | 1 of 1 | 3 of 3 | 9 (cand_0001) + 2 (seed, val) |
The seed's modal trial earned zero tool-call credit. It was following its own instruction —
"always use get_job_log" — and this task's golden call is get_job. Having called the wrong
action it then could not report the template fact, because in this harness get_job_log returns
log text only. The champion's own new text says so outright: "get_job_log in this simulator
returns ONLY log fields (job_id, controller, log, log_original_size, log_trimmed_size) — it does
NOT include metadata."
So the seed's failure on this task was not a reasoning failure. It was a factually wrong sentence in its own documentation, obeyed correctly.
What didn't
The headline number for this task is the wrong number, in both directions. The run reports
Δ +0.156 against a seed of 0.844. That 0.844 is an n=3 measurement — [1.000, 0.533,
1.000] — reused from the previous run rather than re-measured, while cand_0001 was measured
at n=9. At n=9 the seed scores 0.500, and its trial vector shows why the n=3 sample was
flattering: seven of eight valid trials sat at 0.533, so the val sample caught one typical trial
and two lucky ones. The true gain is +0.500, three times the reported one.
And the seed lost a trial the candidate did not. Position 0 of the seed's n=9 vector is an
AgentSetupTimeoutError. That is one of eight such errors in this run, and all eight fall in
seed_train cells; cand_0001 lost none — six of them at trial index 0, two at index 5. Whatever
that is, it is not a property of the skill under test, and it sits entirely on the baseline's side of
the comparison. Counting those trials as zeros would push this task's gain to +0.556; that figure
is biased and is not the one to quote.
What we can (or can't) learn
We can conclude that this edit works, on this harness, reproducibly. Nine of nine at ceiling
against a seed that never once reached ceiling on train is about as clean as a single iteration
gets, the mechanism is legible, and the mechanism predicts the per-task pattern across the rest of
the suite (see results/v2/summary.md): the tasks that improved are the ones whose required facts
live in get_job's metadata, and the tasks that did not move are the ones already served correctly
by get_job_log.
We cannot conclude that the skill got better at troubleshooting. What the optimizer found is a documentation defect — the seed described a tool response shape that this harness does not produce — and what it shipped is a correction to that description. That is a real and useful fix, and it is also the kind of fix that transfers only as far as the description is wrong in the same way elsewhere.
The open risk, which this run cannot settle, is which side the defect is on. The seed is the
production skill, and its claim — get_job_log "returns metadata plus the trimmed log" — is
presumably true of the real AAP2 MCP server. The kaegis simulator splits them. If so, the champion is
now correct for the simulator and wrong for production, and the +0.500 measures the gap between
the two rather than an improvement to either. Nothing in this run's artifacts resolves that: the
v2-baseline trajectories on disk are from the broken-wiring sweep and captured no tool results.
Checking the production get_job_log response shape against the champion's new text is the single
highest-value follow-up in this branch, and it is a five-minute check.