Parsec · reports/v1/aap2-0048.md source
traces_parsec-aap2-0048
Status: OPTIMIZED · Experiment: v1 (trace-extracted from a production session) · Category: aap2 / multi-tool · Run: run_20260814_214843
Prompt given to the agent:
debug aap logs: 297831, 297926, 297964, 298164. recommend fix for provision failure issue.
Contract (tests/expected.json): subset match over 11 demanded tool calls · 5 answer_contains assertions (0 passed at baseline) · weights trajectory 0.5 / assertions 0.5 · gate requires status == ok and a non-empty answer.
Trajectory reachability at baseline: the contract demands lookup_catalog_item, which no simulator running on 2026-08-12 exposed. Under a subset match one uncallable op fails the whole containment, so trajectory: 1.0 was unreachable by construction on this task and its reward was capped at 0.5 whatever the skill said.
Flags: gate-failed-at-baseline · zero-at-baseline · trajectory-unreachable-tool-gap
| measurement | split | n | reward | |
|---|---|---|---|---|
| baseline sweep | all |
1 | 0.000 | single trial, no optimizer — v1 reference sweep |
seed |
val |
30 | 0.345 | cap-evolve baseline on val |
seed (errored trials as 0.0) |
val |
30 | 0.253 | same trials, other denominator |
seed_train |
train |
30 | 0.373 | the seed re-measured inside the iteration, at the iteration's trial count |
cand_0001 |
val |
30 | 0.217 | rejected |
cand_0002 |
val |
30 | 0.163 | rejected |
FINAL (test) |
test |
— | 0.373 | sealed-test evaluation — NOT held out: this run's test split equals its train/val split, so read it as a fit metric |
Best: seed @ 0.345 · Δ vs seed: +0.000 · Iterations spent (whole run): 2 · Harness-errored trials in this task's cells: 8
Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.
Material:
- results/v1/baseline_all_1786543049.json — the n=1 sweep row this task's baseline comes from, with its completion / trajectory / assertions components
- results/v1/tasks.json — this task's full contract: the demanded tool-call sequence and every answer_contains string
- run_20260814_214843/report.md — the run's own write-up
- run_20260814_214843/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words
- run_20260814_214843/events.jsonl — the split aggregates every number above was reconciled against
- results/v1/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null
- artifacts/v1/seed/SKILL.md and the two rejected candidates — diff them to see what was tried and thrown away
- ui/heatmap.html — this row in context of all 40 tasks
What changed
Nothing shipped: both candidates in the run were rejected and the champion stayed the seed. But
this task, not -0047, is the one the second candidate was actually designed around, and the
design is on the record.
The failure mode the optimizer identified here is specific and mechanical. From the journal:
get_job_logall-not-found → "Which information can you provide?" → trajectory=0, assertions=0.8, no report data.
The agent is handed job IDs, every ID returns not found on every controller it tries, and instead
of pivoting it stops and asks the user a question. That produces a completed, non-empty answer — so
the gate passes — with zero trajectory credit and 4 of 5 assertion strings present. Exactly
0.5 × 0.000 + 0.5 × 0.800 = 0.400, which is the only non-zero value in this task's entire seed,
seed_train and FINAL trial vectors. The journal counted 21 trials hitting this dead-end.
cand_0001 attacked it with an additive "Investigation Contract (non-negotiable)" plus a
separate Fallback section — three new rules layered on top of an already long skill.
cand_0002 deleted the Contract and kept only the pivot, tightly guarded: it fires only when
every user-supplied ID returns not found on every valid controller — "a condition no
currently-passing task can reach". The flow it installs is find_jobs → template_name → catalog
item → lookup_catalog_item → fetch_github_file on common.yaml and the stage YAML → scm_ref.
That sequence was chosen to satisfy the subset trajectory check (it names all four tool families)
and to surface the catalog item and __meta__.deployer.scm_ref strings the assertions want. The
predicted lift was trajectory 0→1 on the 21 partial trials.
The journal also records a deliberate anti-overfitting choice: the examples in the new Fallback
flow are format-shaped (account.catalog_item.stage, <product>-<X.Y>, development) and
avoid the literal assertion strings for -0047, to stop the agent parroting a memorised
answer. That is the right instinct and it is rare to see it written down before the run rather than
excused after it.
What worked
The pivot worked — once. One cand_0002 trial scored 0.900, which decomposes as
0.5 × trajectory 1.000 + 0.5 × assertions 0.800. The trajectory matched. In 30 seed trials, 30
seed_train trials and 30 FINAL trials this task never once scored above 0.400; the rejected
candidate is the only artifact that ever cleared it, on this task or on -0047 (where it produced
the matching 0.833). Two trajectory matches, in 60 candidate trials, both from cand_0002.
cand_0001 produced a different, smaller win worth separating out. One of its trials scored
0.500 = 0.5 × trajectory 0.000 + 0.5 × assertions 1.000 — all five assertion strings
present, with no trajectory credit. The seed never reached 5 of 5. So cand_0001's Rule 3 ("name
the catalog item and scm_ref") did fire and did work, exactly once, on the assertion side
while contributing nothing on the trajectory side it was aimed at.
Two candidates, two different halves of the reward moved, each once out of thirty.
What didn't
Both means. cand_0001 scored 0.217 and cand_0002 0.163 against a seed of 0.345 — the
paired gate rejected both at gate_k_se = 1.0 on val means of 0.225 and 0.157. The per-trial
picture:
| artifact | trials at 0.400+ |
trials at 0.000 |
best single trial |
|---|---|---|---|
seed (22 valid) |
19 | 3 | 0.400 |
cand_0001 |
16 | 14 | 0.500 |
cand_0002 |
11 | 19 | 0.900 |
Both candidates roughly doubled or quadrupled the number of outright zeros. Whatever the pivot does when it fires correctly, it more often pushes the agent into a state that scores nothing at all — and the guard that was supposed to make it safe ("a condition no currently-passing task can reach") did not keep it from firing destructively on this task's own trials.
The Investigation Contract did not survive contact. The journal's own post-hoc reading —
"the reader (sonnet-4-5) does not obey a second set of rules layered on top of an already-detailed
skill — the fix has to be a MODIFICATION of the existing decision points, not an addition" — is
supported by the data: cand_0001 lifted trajectory on zero of 30 trials despite a rule
explicitly demanding all four tool families be called.
And eight trials of the seed measurement never ran. Positions 16–23 of this task's seed
vector are null, all NetworkConnectionError, in the same contiguous window that is missing from
-0047's seed vector in the same run — one outage, not sixteen coincidences. The journal recorded
"6/30 trials fail on NetworkConnectionError … Ceiling on 0048 stays around ~0.80"; the rollouts
say 8. As on -0047, the run under-counted its own infrastructure loss, and the resulting seed
figure differs by convention: 0.345 with errored trials dropped, 0.253 with them counted as
zero.
What we can (or can't) learn
The most defensible finding here is a process one: the gate is measuring the wrong statistic for
the question being asked. cap-evolve accepts or rejects on a paired mean, which is correct if the
goal is "ship a champion that is better on average". It is the wrong statistic for "did this edit
teach the agent a behaviour it did not have", and on this task the two answers diverge completely.
cand_0002 is worse on average and is the only artifact that ever demonstrated the target
behaviour. Both statements are true and only one of them survived into the run's summary.
The second finding is that this task's dead-end is a real, reproducible, single-cause failure —
and that makes it unusually valuable. 21 of 30 seed trials fail the same way for the same reason,
with a legible transcript signature ("Which information can you provide?"). That is a much better
optimizer target than a task that fails diffusely, and it was correctly identified. What failed was
the intervention, twice, in two different ways — additive rules ignored, then a guarded pivot
that destabilised the common case. A third attempt with the pivot expressed as a modification of
the existing get_job_log decision point (which is what the journal's own next-lever note
proposes) has not been tried.
What we cannot conclude is anything about held-out generalisation. This run's test split is
identical to its train and val splits — the same two tasks — and cap-evolve emitted
"test overlaps train/val (no-holdout fit) — the test number is NOT held out; report it as a fit
metric". The 0.373 in the table is a re-measurement, not a generalisation result, and
test_delta = +0.0 simply reflects that the champion is the seed.
And half of this task's reward was not in play when it was baselined — but it was by the time the
optimizer ran, and that discrepancy is itself a finding. The contract demands 11 calls under a
subset match, including lookup_catalog_item. No simulator running on 2026-08-12 served that
tool, so the baseline row's trajectory: 0.000 was fixed before the agent started; 25 of the 30 v1
tasks share that defect and the ledger flags them trajectory-unreachable-tool-gap. Yet on
2026-08-14 cand_0002 scored trajectory: 1.000 here, and again on
-0047 — which it could not have done unless lookup_catalog_item had become
callable in between. The tool surface changed mid-experiment and nothing records it. These two
tasks are the only direct evidence in the branch that it changed at all.
That reframes the 0.900. It is not evidence that a long contract can occasionally be matched by luck; it is evidence that this contract became matchable and was matched once in thirty trials by a candidate the gate then threw away. Both readings are thin — one trial — but they point at different next steps, and only the second one is supported.