Parsec · reports/v1/aap2-0047.md source
traces_parsec-aap2-0047
Status: OPTIMIZED · Experiment: v1 (trace-extracted from a production session) · Category: aap2 / multi-tool · Run: run_20260814_214843
Prompt given to the agent:
debug aap logs: 297831, 297926, 297964, 298164. recommend fix for provision failure issue.
Contract (tests/expected.json): subset match over 13 demanded tool calls · 6 answer_contains assertions (0 passed at baseline) · weights trajectory 0.5 / assertions 0.5 · gate requires status == ok and a non-empty answer.
Trajectory reachability at baseline: the contract demands lookup_catalog_item, which no simulator running on 2026-08-12 exposed. Under a subset match one uncallable op fails the whole containment, so trajectory: 1.0 was unreachable by construction on this task and its reward was capped at 0.5 whatever the skill said.
Flags: gate-failed-at-baseline · zero-at-baseline · trajectory-unreachable-tool-gap
| measurement | split | n | reward | |
|---|---|---|---|---|
| baseline sweep | all |
1 | 0.000 | single trial, no optimizer — v1 reference sweep |
seed |
val |
30 | 0.288 | cap-evolve baseline on val |
seed (errored trials as 0.0) |
val |
30 | 0.211 | same trials, other denominator |
seed_train |
train |
30 | 0.311 | the seed re-measured inside the iteration, at the iteration's trial count |
cand_0001 |
val |
30 | 0.233 | rejected |
cand_0002 |
val |
30 | 0.150 | rejected |
FINAL (test) |
test |
— | 0.278 | sealed-test evaluation — NOT held out: this run's test split equals its train/val split, so read it as a fit metric |
Best: seed @ 0.288 · Δ vs seed: +0.000 · Iterations spent (whole run): 2 · Harness-errored trials in this task's cells: 8
Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.
Material:
- results/v1/baseline_all_1786543049.json — the n=1 sweep row this task's baseline comes from, with its completion / trajectory / assertions components
- results/v1/tasks.json — this task's full contract: the demanded tool-call sequence and every answer_contains string
- run_20260814_214843/report.md — the run's own write-up
- run_20260814_214843/JOURNAL.md — per-iteration hypothesis, edit, and verdict, in the optimizer's own words
- run_20260814_214843/events.jsonl — the split aggregates every number above was reconciled against
- results/v1/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null
- artifacts/v1/seed/SKILL.md and the two rejected candidates — diff them to see what was tried and thrown away
- ui/heatmap.html — this row in context of all 40 tasks
What changed
Two candidates were generated for this task and both were rejected; the champion at the end of the run is still the seed. So the answer to "what changed" is: nothing shipped. But the two attempts are the only real optimizer signal v1 produced, and they are worth reading as experiments rather than as failures.
cand_0001 — "Force config-hierarchy trace even when job logs are missing." Four edits to
SKILL.md, all additive:
- a new "Investigation Contract (non-negotiable)" section with three rules — echo the user's
IDs verbatim; call
query_aap2+lookup_catalog_item+fetch_github_file+search_github_repoat least once each; name the catalog item andscm_refin the report; - a rewritten Tips bullet keeping
get_job_log-first behaviour but adding a mandatoryfind_jobs(status=failed)fallback oneast+westwhen every supplied ID returns not found on every controller; - a new "Fallback: When Specific Job Data Is Unavailable" section spelling out a 4-step flow;
- a tightened "MANDATORY
fetch_github_file" preamble closing the loophole the agent used to skip the config trace on a not found.
Rule 2 was aimed squarely at this task's trajectory: 0.000. The journal's stated expectation was
explicit and quantified: "trajectory 0 → 1 on 26/30 completed trials, +0.5 per trial to reward."
cand_0002 — "Redirect 'not found on all controllers' from an ask-user dead-end to a compact
Fallback pivot." It dropped the Investigation Contract entirely — the optimizer's own reading of
cand_0001's failure was that a second layer of rules is ignored — and kept only the pivot,
guarded to fire exclusively when every user-supplied ID returns not found on every valid
controller.
Both were evaluated on 30 trials on val and both scored below the seed: paired Δ̄ −0.092 and
−0.160, rejected at gate_k_se = 1.0. Neither is recorded as fixing or breaking a named
task — the run's rejected.jsonl shows fixed={—} broke={—} for both, because the val split is
only two tasks and neither crossed a pass/fail boundary.
What worked
One thing worked, exactly twice, and it is the most interesting number in the whole v1 experiment.
Every trial of the seed on this task scored either 0.333 or 0.000 — no other value appears in
30 seed trials, 30 seed_train trials, or 30 FINAL trials. 0.333 decomposes as
0.5 × trajectory 0.000 + 0.5 × assertions 0.667: the seed reliably recovered 4 of this task's
6 answer_contains strings and never once matched the 13-call trajectory. That is a ceiling,
not a distribution.
cand_0002 broke it. One of its 30 trials scored 0.833 — which decomposes as
0.5 × trajectory 1.000 + 0.5 × assertions 0.667. A full trajectory match. It is the only
trajectory match ever recorded on this task by any artifact in any v1 run, and cand_0002 also
produced the only one on -0048 (a 0.900, same run). Two matches in 60 candidate trials.
The journal had predicted the number before it happened: "Achievable upper bound on 0047 stays ~0.83 while these persist." The one trial that got there hit 0.8333.
So the rejected candidate did something the champion cannot do. It just did it about 3% of the time, while making the ordinary case markedly worse.
What didn't
The mean, decisively, for both candidates. Read the per-trial vectors in
results/v1/per_task_scores.json rather than the means:
| artifact | trials at 0.333+ |
trials at 0.000 |
best single trial |
|---|---|---|---|
seed (22 valid) |
19 | 3 | 0.333 |
cand_0001 |
20 | 10 | 0.333 |
cand_0002 |
11 | 19 | 0.833 |
cand_0001 did not lift trajectory on a single trial out of 30 — the Investigation Contract's
Rule 2 produced no trajectory credit at all — and it converted three additional trials to outright
zero. cand_0002 traded nineteen zeros for one 0.833. Both are correctly rejected by a
mean-based paired gate. Neither is a "regression" in the sense of damage shipped: the champion
never moved.
The optimizer's own diagnosis of why is on the record and is the useful part. From the journal's refuted-hypotheses section: "The reader (sonnet-4-5) does not obey a second set of rules layered on top of an already-detailed skill — the fix has to be a MODIFICATION of the existing decision points, not an addition." And separately, that "echo every user-supplied identifier verbatim" is not the missing behaviour. Two hypotheses were tested and both were falsified, which is more than most iterations of anything produce.
And eight of thirty seed trials never ran. The seed vector has null at eight consecutive
positions (trials 16–23), all NetworkConnectionError. That is not eight independent samples of
bad luck — it is one outage window, and the identical eight positions are missing from -0048's
seed vector in the same run. The journal called this out at the time as "Network/harness errors
(t16-t23 subset) — ~5/30 trials"; the rollouts on disk say 8, so the run under-counted its own
infrastructure loss by three trials. The candidates, evaluated later, lost none.
That last point matters for the headline: the seed's val figure is 0.288 with those eight
dropped and 0.211 with them counted as zero. Both appear in the ledger. cap-evolve reported
the first.
What we can (or can't) learn
We can learn that the trajectory half of this task's reward is reachable but not reliably
reachable, and that this was discovered by a candidate the gate threw away. A single 0.833
against a seed that is pinned at 0.333 is weak evidence — one trial — but it is positive
evidence about the ceiling, and it is exactly the kind of finding a mean-based accept/reject
decision is designed to discard. The gate made the right call for the champion and the wrong call
for the knowledge: cand_0002 should have been re-run at higher n on 0047 alone rather than
dropped.
We cannot learn much about the skill from the baseline row, and this task is the reason we know
why. Its contract demands 13 calls under a subset match, one of which is
lookup_catalog_item — a tool the babylon simulator serves, and babylon was not running when
the 30-task sweep scored this task on 2026-08-12. Under a subset match one uncallable op fails the
whole containment, so the trajectory: 0.000 in the baseline row was guaranteed before the agent
started. 25 of the 30 v1 tasks are in that position; see results/v1/summary.md.
But this task is also one of only two places in the branch where that gap is demonstrably
closed. cand_0002 scored a full trajectory: 1.000 here on 2026-08-14, which is only possible if
lookup_catalog_item was callable by then — and the same candidate did it on
-0048, the other task that demands it. So the tool surface changed between
2026-08-12 and 2026-08-14, and no file in this branch or in the source worktree records when or by
whom. The practical consequence: this task's baseline 0.000 and its pilot numbers were measured
against different harnesses, and the 28 baseline-only rows elsewhere in v1 were all measured against
the earlier, narrower one. Do not line them up in a single column.
What is left of the reward-level reading is the assertion half, which was reachable throughout — and
it never moved. Every non-zero trial of every artifact on this task, in every one of the four v1 runs,
scored assertions 0.667 and no other value: 4 of 6 strings, never 3, never 5. That is a hard
ceiling on the half of the reward the harness was not interfering with, and no candidate touched it.
The demanded-length story (13 calls is long) is a secondary and confounded effect — see the
op_count_confound_note in results.json.
We cannot treat the test number as a result at all. The pilot ran with
train == val == test == {0047, 0048} and cap-evolve emitted its own warning:
"test overlaps train/val (no-holdout fit) — the test number is NOT held out; report it as a fit
metric". The 0.278 in the table above is a re-measurement of the same two tasks the optimizer
was tuned on. test_delta is +0.0 because the champion is the seed.
The concrete next step this task argues for is not a better skill edit. It is: fix the
contract (13 mechanically-scraped calls is not a specification of correct behaviour), fix the
infrastructure (an 8-trial outage inside a 30-trial measurement), and only then re-run — with
cand_0002's pivot as the first candidate to retry, since it is the only edit with any
demonstrated upside on this task.