Parsec · reports/README.md source

reports/ — per-task reports (level 3)

One file per task answering: what did we change, what worked, what didn't, and what can we (or can't) learn from this task. These are the bottom level of a three-level drill-down:

level what where
1 headline numbers for the whole benchmark, next to other benchmarks the parsec row on the dashboard (benchmark-history)
2 per-task scores across candidates and runs, as a heatmap ui/heatmap.html (v1/v2), ui/heatmap_v4_t.html (v4 T arm), ui/heatmap_v4_g_e1.html (v4 G arm), ui/heatmap_v5_b1_e1.html + ui/trials_v5_b1_e1.html (v5 baseline arm), ui/heatmap_v5_g2_e1.html + ui/trials_v5_g2_e1.html (v5 G2-arm reuse), ui/heatmap_v6_g2_e1.html (v6 G2 arm, sealed test split only)
3 why a task moved, and what it teaches reports/<exp>/<task>.md

Naming

Five experiments each get their own subfolder — v1/, v2/, v4/, v5/, v6/ — mirroring artifacts/<exp>/, so the task-id namespaces never collide or imply kinship. v4, with two arms, splits one level further; v5 and v6 do the same:

experiment pattern count
v1 (trace-extracted) v1/aap2-<NNNN>.md 30
v2 (hand-authored) v2/<task>.md 10
v4 T arm (task-by-task) v4/v4_t_e1/<task>.md 34
v4 G arm (global, shared bundle) v4/v4_g_e1/<task>.md 34
v5 baseline arm (original skill, zero-shot) v5/v5_b1_e1/<task>.md 136
v5 G2-arm reuse (cand_0004, zero-shot) v5/v5_g2_e1/<task>.md 136
v6 G2 arm (hill-climb, global/shared bundle) v6/v6_g2_e1/<task>.md 66 (sealed test split only — see below)

v4/h4/ is the exception to "one file per task": it holds the h4 held-out suite — 30 new scenarios (bench-v4-035 … -064) authored after the original 34, used to test a bundle on tasks the optimizer never saw. Its material is cross-arm by nature (a per-trial table across three arms, and the run handoff), so it is grouped by suite rather than split per task. Per-task h4 reports, if they are ever written, belong beside them as v4/h4/<task>.md.

Cross-cutting analyses that span arms or experiments stay at this directory's root with an experiment prefix — v4-skillset-variant-comparison.md, v4-t4-merge-decisions.md, v4-cap-evolve-question-answers.md, v5-benchmark-quality-review.md — rather than inside an arm folder, since none of them belongs to a single arm.

v1's task ids are all of the form traces_parsec-aap2-<NNNN>, so the shared prefix is dropped and only the number kept — v1/aap2-0047.md. v2's ids are already readable (bench-aap2-004-failing-task-and-host) and are used whole. The mapping is not a convention to re-derive by hand: every task row in results/results.json carries a report field with the exact path, and scripts/build_task_reports.py reads it.

v4's ids are already readable (platform-005-wrong-owner-trap) and are used whole, like v2's. Unlike v1/v2, v4's report pointer is carried per arm — results/v4/v4_t_e1/results.json and results/v4/v4_g_e1/results.json (separate ledgers from results/results.json and from each other) — and scripts/build_v4_t_task_reports.py / scripts/build_v4_g_task_reports.py read those files instead of build_task_reports.py's.

The auto block

Each report opens with a block delimited by <!-- BEGIN:auto --> / <!-- END:auto -->. That block is generated from results/results.json by scripts/build_task_reports.py — status, experiment, category, run, prompt, contract shape, flags, the measurement table, and links to whatever material exists for that task.

Everything below the block is hand-written and the script never touches it.

python3 scripts/build_task_reports.py           # create missing reports, refresh auto blocks
python3 scripts/build_task_reports.py --check    # exit 1 if any auto block is stale (CI-friendly)
python3 scripts/build_task_reports.py --stats     # coverage: how many written, how many analysed

Do not hand-edit inside the auto block — edits there are overwritten on the next run. Numbers stay derived; narrative stays human.

The diff block (v4 and v6 only — neither v5 arm gets one)

v5_b1_e1 is a zero-shot baseline with no best/ directory anywhere, so build_v5_b1_task_reports.py calls the same scripts/v4_report_common.py::build_diff_block() every v4 generator uses and it returns None for all 136 tasks — the same path v4's never-optimized tasks already take. v5_g2_e1 reuses an already-optimized bundle (artifacts/v4/v4_g_e1/best) as-is rather than optimizing per task, so it has no per-task best/ directory either; build_v5_g2_task_reports.py takes the identical path. No special-casing in either generator.

v6_g2_e1 DOES get a diff block (seed vs. artifacts/v6/v6_g2_e1/best/, the global shared bundle, same as v4's G arm) — but only 66 reports exist, not v6's full 200-task set. Its results.json only ever has per-task rows for the sealed test split (the only split with a stable, full per-task ledger — see results/v6/v6_g2_e1/summary.md's "Why there's no per-task F/C ledger"), so build_v6_g2_task_reports.py only has 66 rows to generate from. There is no reports/v6/v6_g2_e1/<feedback-or-confirm-task>.md for any of the other 134 tasks, and there never will be for this split design — not a gap to fill in later.

The v4 arms' diff block

Both v4 arms get a second generated section, <!-- BEGIN:diff --> / <!-- END:diff -->, appended after the hand-written narrative (or, for the G arm, after the not-yet-written-narrative placeholder — see below): real, file-by-file unified diffs of artifacts/v4/seed/*.md against the arm's own candidate.

For the T arm, the 20 tasks where T2 accepted a candidate over the seed get the diff against that task's artifacts/v4/v4_t_e1/<task>/best/*.md, one collapsible <details> per changed file so the report stays scannable with the full edit one click away (e.g. platform-004-events-then-config). build_v4_t_task_reports.py regenerates it the same way it regenerates the auto block above; the 14 T-arm tasks with no best/ directory (13 T2 never ran on, plus cost-030-threshold-not-an-anomaly, which only has a NOTE.md) get no diff block at all rather than an empty one.

For the G arm, every one of the 34 tasks gets the same diff — artifacts/v4/seed/*.md against the single shared artifacts/v4/v4_g_e1/best/*.md — since G evolves one bundle jointly across all tasks rather than one per task. build_v4_g_task_reports.py generates it from scripts/v4_report_common.py's shared diff logic, the same module the T generator uses, so the splice markers and diff rendering stay identical between arms.

This answers Item 2 of this branch's cleanup — a prose summary of "what the optimizer tried" is not the same as seeing the edit — without needing a separate diff viewer: GitHub and most local Markdown renderers syntax-highlight a fenced ```diff block on their own.

Where this departs from skillsbench-history

The scheme is copied from skillsbench-history's reports/ and three things are deliberately different, all for the same reason: parsec's measurements are not comparable across cells the way skillsbench's were.

1. The measurement table is vertical, not horizontal. skillsbench-history renders one row per task with a column per candidate, which is readable because every candidate there was measured at the same trial count. Parsec's were not. v1's baseline sweep is n=1 and its pilot is n=30; v2's seed is a reused n=3 measurement while everything else in that run is n=9. A horizontal row would silently invite comparing an n=3 number with an n=9 number — which is exactly the mistake that produced v2's headline. So each report lists one row per distinct measurement, with its n and its split, and omits rows for measurements that do not exist rather than printing a null. The length of the table is itself information: a 1-row table means the optimizer never ran on that task.

2. Narrative is pre-filled for more classes. skillsbench-history leaves most reports at _Not yet analysed._ and says so honestly, because for a task the optimizer improved, the story needs a human reading of the diff. Here, 28 of the 30 v1 tasks were never seen by the optimizer at all, and for those "what happened" genuinely is the contract plus one trial — which is derivable. The generator pre-fills seven v1 classes (vacuous-assertion contract, solved-at-baseline, trajectory-matched, capped-at-0.5, zero-at-baseline, partial-assertions, and gate-failed, the last of which stays a placeholder because a gate failure needs a rollout read). This is not padding: each pre-filled report states a task-specific, derived fact, and none of them claim an optimizer result that does not exist.

3. The "a rejected candidate is not a regression" caution is conditional. skillsbench-history prints it on every report. Here it appears only when a candidate was actually evaluated on that task (status == OPTIMIZED and a cand_0001 measurement exists) — otherwise it would be filler on 28 of 30 v1 reports and would train the reader to skip the italics. Note that once a report's hand section is written, the author's own narrative supersedes it; the in-block caution "Read no cell without the n beside it" is inside the auto block and therefore survives on all 40 reports regardless.

Status

40 reports, 40 fully analysed, 0 placeholders. Run --stats for the live count.

That is unusual for a first pass and it is worth saying why it was achievable: parsec's optimizer ran on only 12 of the 40 tasks, so most reports are an account of a contract and a single trial rather than of an optimization. The contracts are all committed here (results/v1/tasks.json, results/v2/tasks.json), the per-trial vectors are committed (results/v{1,2}/per_task_scores.json), and v2's ten scenarios each ship a hand-written provenance.md in the source tree recording why the scenario exists and every defect fixed in it. There was more written material per task than there were tasks to write about.

Where the material comes from

source covers what it gives
results/results.json 40/40 every number in every auto block: rewards per tag per split, n, errored-trial counts, contract shape, flags
results/v1/tasks.json 30/30 v1 contracts: the demanded tool-call sequence and every answer_contains string
results/v2/tasks.json 10/10 v2 contracts: expected calls, required/derived/forbidden facts, per-task weights
results/v{1,2}/per_task_scores.json 40/40 per-trial reward vectors for every task in every run, errored trials as null
results/v2/per_task_scores.csv 10/40 the hand-maintained per-task analysis, one note per task per iteration
recipes/v1/PROJECT.md 30/40 the dated decision log for v1 — including the simulator tool-coverage note that explains 25 of the 30 trajectory: 0.000 rows
results/v1/runs/run_20260814_214843/JOURNAL.md 2/40 the optimizer's own hypotheses, refutations and next-lever notes — the single richest source for v1
results/v2/runs/*/JOURNAL.md 10/40 same for v2, but see the caveat below
<source>/harbor-tasks-v2.1/<task>/provenance.md 10/40 why the scenario exists, its discrimination axis, and a dated edit log for every defect fixed
artifacts/ 40/40 the skill packages themselves — diff seed/ against best/ and the rejected candidates

The v2 journal caveat: the headline run's optimizer wrote no handover at all — its entry reads ## Iteration cand_0001 — (no handover written by the optimizer), and events.jsonl records an optimizer_error for that candidate. Six edits shipped together with no statement of intent, so every per-task attribution in the v2 reports is inference from the reward decomposition, not from the record. The iter-1 run (run_20260816_202942) does have a full journal entry and is the better read for how the optimizer was reasoning.

Four cautions when writing one

Say which run and which n a number came from. This branch's central finding is that the same byte-identical seed skill measures anywhere from 0.000 to 0.367 in v1 and 0.643 to 1.000 in v2 depending only on the trial count and the run. A number without its n is not a result. The ui/heatmap.html ladder table lists all 24 seed measurements side by side.

Check mean_valid against mean_all before quoting. Errored trials are written with reward: 0.0 and are distinguishable from genuine zeros only via rollout.error. Both denominators are carried everywhere in the ledger. They differ materially: v2's headline seed_train is 0.822 with errored trials dropped and 0.749 with them counted as zero. results/v2/per_task_scores.csv uses the errored-as-zero convention, so its numbers legitimately disagree with the auto block above them.

On a v1 report, check the reachability line before reading trajectory as a skill measurement. When the 30-task sweep ran, two simulators were up and 25 of the 30 contracts demand a tool neither served. Under a subset match one uncallable op fails the whole containment, so those 25 rows have trajectory: 0.000 by construction and a reward capped at 0.5 — a fact about the harness, not the skill. Every affected report says so in its auto block and carries the trajectory-unreachable-tool-gap flag; the ledger carries trajectory_tools_missing_at_baseline per task. The two optimized tasks (-0047, -0048) are the exception worth knowing: their baseline rows are flagged, but by the 2026-08-14 pilot the tool had become callable, and their reports are the only record of that change.

A rejected candidate is not a regression, and an accepted one is not necessarily an improvement. Both halves matter here. v1's cand_0002 was rejected and is the only artifact that ever produced a trajectory match on either of the two tasks it ran on. v2's accepted cand_0001 is recorded as having broken two tasks, and at equal n one of them is flat to six decimal places and the other improved by +0.133. Re-derive per-task verdicts from seed_train rather than reading them off report.md.

Worked examples

Four reports carry the most, and are the best entry points: