Parsec · reports/v2/bench-aap2-008-preceding-task.md source
bench-aap2-008-preceding-task
Status: OPTIMIZED · Experiment: v2 (hand-authored) · Category: aap2 / preceding-task · Run: run_20260818_161550
Prompt given to the agent:
AAP2 job 90810 on prod0 stopped part-way through its playbook. Which task failed, and which task completed immediately before it?
Contract (tests/expected.json, schema bench/v2): 1 expected tool call · 2 required answer facts · weights tool_calls 0.2 / answer 0.8.
| measurement | split | n | reward | |
|---|---|---|---|---|
| baseline sweep | all |
1 | 1.000 | single trial, no optimizer — v2 reference sweep |
seed |
val |
3 | 1.000 | cap-evolve baseline on val — reused from .capevolve/run_20260816_202942, not re-measured in this run |
seed_train |
train |
9 | 0.844 | the seed re-measured inside the iteration, at the iteration's trial count |
cand_0001 |
train |
9 | 0.978 | accepted champion |
Best: cand_0001 @ 0.978 · Δ vs seed: -0.022 · Iterations spent (whole run): 1 · Harness-errored trials in this task's cells: 0
Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.
Material:
- results/v2/tasks.json — this task's full contract: expected calls, required/derived/forbidden facts, per-task weights
- results/v2/per_task_scores.csv — the hand-maintained per-task analysis, including a note per task per iteration. Its numbers use the errored-as-zero denominator, so they can differ from the table above; compare like with like
- run_20260818_161550/report.md — the run's own write-up
- run_20260818_161550/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words
- run_20260818_161550/events.jsonl — the split aggregates every number above was reconciled against
- results/v2/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null
- artifacts/v2/best/ vs artifacts/v2/seed/ — the accepted champion against the seed
- ui/heatmap.html — this row in context of all 40 tasks
What changed
The champion improved this task, and the run recorded it as a regression. cap-evolve's RESULT
line names it in the broke={} set:
ACCEPTED (new champion) · val=0.962 Δ=+0.086 · fixed={…} · broke={bench-aap2-006-log-root-cause, bench-aap2-008-preceding-task}
Measured inside the same iteration at the same trial count:
- the seed, re-measured at
n=9, scored 0.844 cand_0001, atn=9, scored 0.978
That is +0.133 — an improvement, on a task the run reports as broken with a Δ of −0.022. The
−0.022 is measured against a reused n=3 seed value of 1.000 ([1.000, 1.000, 1.000]) that
was never re-measured. Three ceiling trials drawn from a distribution whose n=9 mode is 0.800.
This task has no errored trials in either cell, so the sign flip is not a denominator question. It
is purely n=3 against n=9. Together with 006, which is
exactly flat at n=9, both entries in this run's broke={} list are wrong.
What worked
The ordering discrimination — on every trial, for both artifacts. With weights 0.2 / 0.8, one
expected call and two required facts, the values resolve uniquely:
| observed | expected calls matched | answer facts | seed @ n=9 | cand_0001 @ n=9 |
|---|---|---|---|---|
| 0.800 | 0 of 1 | 2 of 2 | 7 | 1 |
| 1.000 | 1 of 1 | 2 of 2 | 2 | 8 |
The answer component was 2 of 2 on all 18 trials. The agent always identified both the failing
task (Start broker service, counter 5) and the task immediately preceding it (Install broker
package 4.2.1, counter 4). The entire spread is the tool call, and the champion moved it from 2-of-9
to 8-of-9.
That is worth stating plainly because this scenario was built to test ordering, and the ordering was never in doubt. The provenance predicted the failure mode in arithmetic:
An agent that correctly finds the failed event but reads the preceding task as the first task in the play now scores 0.6 instead of 0.733… However, 0.6 should not be read as "mostly succeeded."
0.600 never appears in this task's data. Not once, in any run, for any artifact. The four
surrounding distractor task names — counter 2 Stop broker service, counter 3 Back up broker data
directory, counter 6 Verify broker health endpoint — were never taken. The discrimination axis is
sound and the skill clears it.
What didn't
The 0.800 mode is a missing get_job_events call, on a task whose author called that call
obvious. The weight rationale reads: "tool_calls: 0.2 because get_job_events is a single,
obvious call that a lazy agent makes anyway." On 7 of the seed's 9 trials it did not make it — and
still answered both facts correctly, presumably from get_job_log output. The 0.2 weight was set on
the assumption that the call was free; measured, it is the only thing this task discriminated on.
Which champion edit fixed it is not recorded. The plausible candidate is the Rule 4 rewrite that
stopped treating a second call on the same job as a redundant re-fetch, or the Investigation Flow
step-4 rewrite that names get_job_events explicitly for failed jobs. But the iteration-2 optimizer
wrote no handover at all — ## Iteration cand_0001 — (no handover written by the optimizer) — so
the run contains no statement of intent to check against. Six edits shipped together and the per-task
attribution is inference from the reward decomposition, not from the record.
And the sign error propagated into the run's own conclusions. A broke={} entry is the signal a
subsequent iteration is instructed to act on: "an entry whose RESULT says rejected with
broke={...} tells you which specific edits to drop or redesign." Had iteration 3 run, it would have
been told to redesign an edit that produced a +0.133 gain here. (Iteration 3 did start and was
aborted — see results/v2/runs/run_run_20260818_161550/, role discarded-aborted-iter3.)
What we can (or can't) learn
The methodological finding is the one that generalises: --reuse-baseline across a change in
num_trials invalidates every per-task verdict in the run. This task is the cleanest demonstration
because the sign actually flips: −0.022 reported, +0.133 measured. The fix is mechanical — re-measure
the seed at the iteration's own n, which this run in fact did (seed_train at n=9 exists and is
committed) and then did not gate against. The number was collected and ignored.
We can also learn that this scenario's pre-run design review paid off. Dropping the cause
required fact was justified as an encoding fix, not a difficulty change:
failing-taskandcauseboth derive from the samerunner_on_failedevent at counter 5 — finding that event yields both facts. This made the first half of the question worth 2/3 of the answer component, while the ordering half (the scenario's actual purpose) was worth only 1/3.
With two facts instead of three, each sub-question carries half the answer weight. And the measured result is that both halves were always satisfied — so the rebalance never had to do its work here, but the task's remaining signal (the tool call) is uncontaminated by it.
We cannot say the champion is at ceiling. One trial in nine still loses the call. And this task's
+0.133 is the third-largest equal-n gain in the run while being reported as a regression, which is
the most concise statement of what is wrong with this run's measurement design.