Parsec · reports/v2/bench-aap2-006-log-root-cause.md source

bench-aap2-006-log-root-cause

Status: OPTIMIZED · Experiment: v2 (hand-authored) · Category: aap2 / log-root-cause · Run: run_20260818_161550

Prompt given to the agent:

The certificate rotation on AAP2 controller prod1 failed. Read the job's log and tell me the root cause.

Contract (tests/expected.json, schema bench/v2): 2 expected tool calls · 3 required answer facts · weights tool_calls 0.5 / answer 0.5.

measurement split n reward
baseline sweep all 1 1.000 single trial, no optimizer — v2 reference sweep
seed val 3 1.000 cap-evolve baseline on val — reused from .capevolve/run_20260816_202942, not re-measured in this run
seed_train train 9 0.778 the seed re-measured inside the iteration, at the iteration's trial count
cand_0001 train 9 0.778 accepted champion

Best: cand_0001 @ 0.778 · Δ vs seed: -0.222 · Iterations spent (whole run): 1 · Harness-errored trials in this task's cells: 0

Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.

Material: - results/v2/tasks.json — this task's full contract: expected calls, required/derived/forbidden facts, per-task weights - results/v2/per_task_scores.csv — the hand-maintained per-task analysis, including a note per task per iteration. Its numbers use the errored-as-zero denominator, so they can differ from the table above; compare like with like - run_20260818_161550/report.md — the run's own write-up - run_20260818_161550/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words - run_20260818_161550/events.jsonl — the split aggregates every number above was reconciled against - results/v2/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null - artifacts/v2/best/ vs artifacts/v2/seed/ — the accepted champion against the seed - ui/heatmap.html — this row in context of all 40 tasks

What changed

Nothing. Not the skill's behaviour on this task, and not the score. This is the task cap-evolve's RESULT line names as broken by the accepted champion:

ACCEPTED (new champion) · val=0.962 Δ=+0.086 · fixed={…} · broke={bench-aap2-006-log-root-cause, bench-aap2-008-preceding-task}

Measured at equal n, that verdict is false. Inside the same iteration:

Identical to six decimal places. The delta at equal n is 0.000. The −0.222 the run reports is the difference between a reused n=3 seed value of 1.000 and an n=9 candidate — that is, it is entirely a sampling artifact. The n=3 val sample drew three trials from a distribution whose n=9 mode is 0.750, and got three ceilings.

This task also has no errored trials in either cell, so nothing about the comparison is confounded by harness loss. It is simply the same number twice.

What worked

The answer, every single time, for both artifacts. With weights 0.5 / 0.5, two expected tool calls and three required facts, the observed values have unique solutions:

observed expected calls matched answer facts seed @ n=9 cand_0001 @ n=9
0.750 1 of 2 3 of 3 8 8
1.000 2 of 2 3 of 3 1 1

The agent named the right job (90520, the recent one — not the 90521 distractor), the right root cause, and the expiry date 2026-07-18, on 18 of 18 trials. It missed one of the two expected tool calls on 16 of 18.

That matters because the scenario was built to catch exactly the failure it never produced. Its stated purpose is to "discriminate agents that read the most recent failed run vs. the first match in the result set" — job 90521 is a week older, is also a certificate-rotation failure on prod1, and fails for a different reason ("connection refused"). An agent that grabs it "answers about a completely different cause and misses the expiry-date fact entirely." No trial ever did. The distractor was never taken.

What didn't

per_task_scores.csv's note on this task is wrong, and the decomposition shows it. The note reads:

No improvement — persistent assertion mismatch

There is no assertion mismatch. The answer component was 3 of 3 on every trial in both cells. The persistent 0.25 shortfall is a tool-call shortfall — one of the two calls the contract names, missed on 16 of 18 trials by both the seed and the champion alike. The hand-written note attributes a stable, fully explained tool-sequencing gap to the answer half of the reward. It is recorded here rather than silently corrected in the CSV, because the CSV is a verbatim copy of the source analysis; the correction belongs in the reading, not in the artifact.

The contract itself has a suspicious asymmetry worth checking before this task is scored again: n_expected_calls is 2 while n_golden_tool_calls is 3. The golden path makes three calls; the contract scores against two. That is the only task in v2 where those disagree, and it sits on the one task where every trial loses exactly half the call credit. Whether the missing call is one the agent should reasonably make, or one the extraction picked up from a golden path that does more than the contract asks, is answerable from results/v2/tasks.json and expected.json in about a minute — and it is a far better use of time than another skill edit.

The 0.25 from iteration 1 is a different animal. run_20260816_202942's rejected cand_0001 scored [1.000, 1.000, 0.250] here, and the CSV calls that trial a "one-trial flake." It decomposes to 1 of 2 calls and 0 of 3 facts — the only trial in this task's entire history where the answer was wrong. The framework's RESULT for that iteration also named this task under broke={}. With one trial and no rollout read, "flake" is a label, not a diagnosis.

What we can (or can't) learn

The load-bearing lesson is about the verdict, not the task. This is the cleanest counterexample in the branch to reading cap-evolve's fixed/broke lists as facts about candidates. Two tasks appear under broke={} in this run. At equal n, this one is exactly flat and 008 is +0.133 — an improvement. Both entries are wrong, and both are wrong for the same structural reason: the run gated an n=9 candidate against an n=3 baseline it reused instead of re-measuring (--reuse-baseline), so every per-task verdict is a comparison between different trial counts. See results/v2/summary.md.

We can also learn that this scenario's honest design cost is visible and was worth paying. The author wanted a forbidden fact no-wrong-run-cause forbidding "connection refused", but that string lives in job 90521's log in the seed, so the required-present / forbidden-absent gate would have failed on forbidden-absent. The recorded decision — "forbidden fact omitted; distractor job retained" — kept the discriminating distractor and relied on the expiry-date required fact to catch a wrong-run answer instead. Eighteen trials later, expiry-date was never missed, so the substitution held. That is a design trade-off documented before the run and then vindicated by it, which is rarer than it should be.

What we cannot say is anything about whether the champion helped or hurt here. It did neither, at n=9, to six decimal places. The only open question on this task is the contract's 2-vs-3 call count, and that is a question for the benchmark author, not the optimizer.