Parsec · reports/v2/bench-aap2-008-preceding-task.md source

bench-aap2-008-preceding-task

Status: OPTIMIZED · Experiment: v2 (hand-authored) · Category: aap2 / preceding-task · Run: run_20260818_161550

Prompt given to the agent:

AAP2 job 90810 on prod0 stopped part-way through its playbook. Which task failed, and which task completed immediately before it?

Contract (tests/expected.json, schema bench/v2): 1 expected tool call · 2 required answer facts · weights tool_calls 0.2 / answer 0.8.

measurement split n reward
baseline sweep all 1 1.000 single trial, no optimizer — v2 reference sweep
seed val 3 1.000 cap-evolve baseline on val — reused from .capevolve/run_20260816_202942, not re-measured in this run
seed_train train 9 0.844 the seed re-measured inside the iteration, at the iteration's trial count
cand_0001 train 9 0.978 accepted champion

Best: cand_0001 @ 0.978 · Δ vs seed: -0.022 · Iterations spent (whole run): 1 · Harness-errored trials in this task's cells: 0

Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.

Material: - results/v2/tasks.json — this task's full contract: expected calls, required/derived/forbidden facts, per-task weights - results/v2/per_task_scores.csv — the hand-maintained per-task analysis, including a note per task per iteration. Its numbers use the errored-as-zero denominator, so they can differ from the table above; compare like with like - run_20260818_161550/report.md — the run's own write-up - run_20260818_161550/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words - run_20260818_161550/events.jsonl — the split aggregates every number above was reconciled against - results/v2/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null - artifacts/v2/best/ vs artifacts/v2/seed/ — the accepted champion against the seed - ui/heatmap.html — this row in context of all 40 tasks

What changed

The champion improved this task, and the run recorded it as a regression. cap-evolve's RESULT line names it in the broke={} set:

ACCEPTED (new champion) · val=0.962 Δ=+0.086 · fixed={…} · broke={bench-aap2-006-log-root-cause, bench-aap2-008-preceding-task}

Measured inside the same iteration at the same trial count:

That is +0.133 — an improvement, on a task the run reports as broken with a Δ of −0.022. The −0.022 is measured against a reused n=3 seed value of 1.000 ([1.000, 1.000, 1.000]) that was never re-measured. Three ceiling trials drawn from a distribution whose n=9 mode is 0.800.

This task has no errored trials in either cell, so the sign flip is not a denominator question. It is purely n=3 against n=9. Together with 006, which is exactly flat at n=9, both entries in this run's broke={} list are wrong.

What worked

The ordering discrimination — on every trial, for both artifacts. With weights 0.2 / 0.8, one expected call and two required facts, the values resolve uniquely:

observed expected calls matched answer facts seed @ n=9 cand_0001 @ n=9
0.800 0 of 1 2 of 2 7 1
1.000 1 of 1 2 of 2 2 8

The answer component was 2 of 2 on all 18 trials. The agent always identified both the failing task (Start broker service, counter 5) and the task immediately preceding it (Install broker package 4.2.1, counter 4). The entire spread is the tool call, and the champion moved it from 2-of-9 to 8-of-9.

That is worth stating plainly because this scenario was built to test ordering, and the ordering was never in doubt. The provenance predicted the failure mode in arithmetic:

An agent that correctly finds the failed event but reads the preceding task as the first task in the play now scores 0.6 instead of 0.733… However, 0.6 should not be read as "mostly succeeded."

0.600 never appears in this task's data. Not once, in any run, for any artifact. The four surrounding distractor task names — counter 2 Stop broker service, counter 3 Back up broker data directory, counter 6 Verify broker health endpoint — were never taken. The discrimination axis is sound and the skill clears it.

What didn't

The 0.800 mode is a missing get_job_events call, on a task whose author called that call obvious. The weight rationale reads: "tool_calls: 0.2 because get_job_events is a single, obvious call that a lazy agent makes anyway." On 7 of the seed's 9 trials it did not make it — and still answered both facts correctly, presumably from get_job_log output. The 0.2 weight was set on the assumption that the call was free; measured, it is the only thing this task discriminated on.

Which champion edit fixed it is not recorded. The plausible candidate is the Rule 4 rewrite that stopped treating a second call on the same job as a redundant re-fetch, or the Investigation Flow step-4 rewrite that names get_job_events explicitly for failed jobs. But the iteration-2 optimizer wrote no handover at all — ## Iteration cand_0001 — (no handover written by the optimizer) — so the run contains no statement of intent to check against. Six edits shipped together and the per-task attribution is inference from the reward decomposition, not from the record.

And the sign error propagated into the run's own conclusions. A broke={} entry is the signal a subsequent iteration is instructed to act on: "an entry whose RESULT says rejected with broke={...} tells you which specific edits to drop or redesign." Had iteration 3 run, it would have been told to redesign an edit that produced a +0.133 gain here. (Iteration 3 did start and was aborted — see results/v2/runs/run_run_20260818_161550/, role discarded-aborted-iter3.)

What we can (or can't) learn

The methodological finding is the one that generalises: --reuse-baseline across a change in num_trials invalidates every per-task verdict in the run. This task is the cleanest demonstration because the sign actually flips: −0.022 reported, +0.133 measured. The fix is mechanical — re-measure the seed at the iteration's own n, which this run in fact did (seed_train at n=9 exists and is committed) and then did not gate against. The number was collected and ignored.

We can also learn that this scenario's pre-run design review paid off. Dropping the cause required fact was justified as an encoding fix, not a difficulty change:

failing-task and cause both derive from the same runner_on_failed event at counter 5 — finding that event yields both facts. This made the first half of the question worth 2/3 of the answer component, while the ordering half (the scenario's actual purpose) was worth only 1/3.

With two facts instead of three, each sub-question carries half the answer weight. And the measured result is that both halves were always satisfied — so the rebalance never had to do its work here, but the task's remaining signal (the tool call) is uncontaminated by it.

We cannot say the champion is at ceiling. One trial in nine still loses the call. And this task's +0.133 is the third-largest equal-n gain in the run while being reported as a regression, which is the most concise statement of what is wrong with this run's measurement design.