Parsec · reports/v2/bench-aap2-001-single-job-outcome.md source

bench-aap2-001-single-job-outcome

Status: OPTIMIZED · Experiment: v2 (hand-authored) · Category: aap2 / single-job-outcome · Run: run_20260818_161550

Prompt given to the agent:

What was the outcome of AAP2 job 90142 on controller prod0, and which job template ran it?

Contract (tests/expected.json, schema bench/v2): 1 expected tool call · 3 required answer facts · weights tool_calls 0.2 / answer 0.8.

measurement split n reward
baseline sweep all 1 1.000 single trial, no optimizer — v2 reference sweep
seed val 3 0.844 cap-evolve baseline on val — reused from .capevolve/run_20260816_202942, not re-measured in this run
seed_train train 9 0.500 the seed re-measured inside the iteration, at the iteration's trial count
seed_train (errored trials as 0.0) train 9 0.444 same trials, other denominator — this is the convention cap-evolve printed for v2
cand_0001 train 9 1.000 accepted champion

Best: cand_0001 @ 1.000 · Δ vs seed: +0.156 · Iterations spent (whole run): 1 · Harness-errored trials in this task's cells: 1

Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.

Material: - results/v2/tasks.json — this task's full contract: expected calls, required/derived/forbidden facts, per-task weights - results/v2/per_task_scores.csv — the hand-maintained per-task analysis, including a note per task per iteration. Its numbers use the errored-as-zero denominator, so they can differ from the table above; compare like with like - run_20260818_161550/report.md — the run's own write-up - run_20260818_161550/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words - run_20260818_161550/events.jsonl — the split aggregates every number above was reconciled against - results/v2/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null - artifacts/v2/best/ vs artifacts/v2/seed/ — the accepted champion against the seed - ui/heatmap.html — this row in context of all 40 tasks

What changed

This is the task the whole v2 iteration was designed around, and it is the one clear, large, mechanistically explained win in this branch.

The champion cand_0001 made six edits to SKILL.md. Every one of them is about which tool to call, and three of them target this task directly:

cand_0001 was accepted at val 0.962, Δ +0.086, and cap-evolve's RESULT line names this task in its fixed={} set. Here that verdict is correct.

What worked

Everything, all nine times. cand_0001 scored 1.000 on 9 of 9 train trials. The seed, at the same n=9, scored 0.500 (errored trial dropped) — a +0.500 gain, the largest single-task delta anywhere in either experiment.

The reward decomposes exactly, and the decomposition is the finding. With weights tool_calls 0.2 / answer 0.8, one expected call and three required facts, only a few values are reachable, and each observed value has a unique solution:

observed expected calls matched answer facts trials
0.267 0 of 1 1 of 3 1 (seed)
0.533 0 of 1 2 of 3 7 (seed) + 1 (seed, val)
1.000 1 of 1 3 of 3 9 (cand_0001) + 2 (seed, val)

The seed's modal trial earned zero tool-call credit. It was following its own instruction — "always use get_job_log" — and this task's golden call is get_job. Having called the wrong action it then could not report the template fact, because in this harness get_job_log returns log text only. The champion's own new text says so outright: "get_job_log in this simulator returns ONLY log fields (job_id, controller, log, log_original_size, log_trimmed_size) — it does NOT include metadata."

So the seed's failure on this task was not a reasoning failure. It was a factually wrong sentence in its own documentation, obeyed correctly.

What didn't

The headline number for this task is the wrong number, in both directions. The run reports Δ +0.156 against a seed of 0.844. That 0.844 is an n=3 measurement — [1.000, 0.533, 1.000] — reused from the previous run rather than re-measured, while cand_0001 was measured at n=9. At n=9 the seed scores 0.500, and its trial vector shows why the n=3 sample was flattering: seven of eight valid trials sat at 0.533, so the val sample caught one typical trial and two lucky ones. The true gain is +0.500, three times the reported one.

And the seed lost a trial the candidate did not. Position 0 of the seed's n=9 vector is an AgentSetupTimeoutError. That is one of eight such errors in this run, and all eight fall in seed_train cells; cand_0001 lost none — six of them at trial index 0, two at index 5. Whatever that is, it is not a property of the skill under test, and it sits entirely on the baseline's side of the comparison. Counting those trials as zeros would push this task's gain to +0.556; that figure is biased and is not the one to quote.

What we can (or can't) learn

We can conclude that this edit works, on this harness, reproducibly. Nine of nine at ceiling against a seed that never once reached ceiling on train is about as clean as a single iteration gets, the mechanism is legible, and the mechanism predicts the per-task pattern across the rest of the suite (see results/v2/summary.md): the tasks that improved are the ones whose required facts live in get_job's metadata, and the tasks that did not move are the ones already served correctly by get_job_log.

We cannot conclude that the skill got better at troubleshooting. What the optimizer found is a documentation defect — the seed described a tool response shape that this harness does not produce — and what it shipped is a correction to that description. That is a real and useful fix, and it is also the kind of fix that transfers only as far as the description is wrong in the same way elsewhere.

The open risk, which this run cannot settle, is which side the defect is on. The seed is the production skill, and its claim — get_job_log "returns metadata plus the trimmed log" — is presumably true of the real AAP2 MCP server. The kaegis simulator splits them. If so, the champion is now correct for the simulator and wrong for production, and the +0.500 measures the gap between the two rather than an improvement to either. Nothing in this run's artifacts resolves that: the v2-baseline trajectories on disk are from the broken-wiring sweep and captured no tool results. Checking the production get_job_log response shape against the champion's new text is the single highest-value follow-up in this branch, and it is a five-minute check.