Parsec · reports/README.md source
reports/ — per-task reports (level 3)
One file per task answering: what did we change, what worked, what didn't, and what can we (or can't) learn from this task. These are the bottom level of a three-level drill-down:
| level | what | where |
|---|---|---|
| 1 | headline numbers for the whole benchmark, next to other benchmarks | the parsec row on the dashboard (benchmark-history) |
| 2 | per-task scores across candidates and runs, as a heatmap | ui/heatmap.html (v1/v2), ui/heatmap_v4_t.html (v4 T arm), ui/heatmap_v4_g_e1.html (v4 G arm), ui/heatmap_v5_b1_e1.html + ui/trials_v5_b1_e1.html (v5 baseline arm), ui/heatmap_v5_g2_e1.html + ui/trials_v5_g2_e1.html (v5 G2-arm reuse), ui/heatmap_v6_g2_e1.html (v6 G2 arm, sealed test split only) |
| 3 | why a task moved, and what it teaches | reports/<exp>/<task>.md |
Naming
Five experiments each get their own subfolder — v1/, v2/, v4/, v5/, v6/ — mirroring
artifacts/<exp>/, so the task-id namespaces never collide or imply kinship. v4, with two
arms, splits one level further; v5 and v6 do the same:
| experiment | pattern | count |
|---|---|---|
| v1 (trace-extracted) | v1/aap2-<NNNN>.md |
30 |
| v2 (hand-authored) | v2/<task>.md |
10 |
| v4 T arm (task-by-task) | v4/v4_t_e1/<task>.md |
34 |
| v4 G arm (global, shared bundle) | v4/v4_g_e1/<task>.md |
34 |
| v5 baseline arm (original skill, zero-shot) | v5/v5_b1_e1/<task>.md |
136 |
v5 G2-arm reuse (cand_0004, zero-shot) |
v5/v5_g2_e1/<task>.md |
136 |
| v6 G2 arm (hill-climb, global/shared bundle) | v6/v6_g2_e1/<task>.md |
66 (sealed test split only — see below) |
v4/h4/ is the exception to "one file per task": it holds the h4 held-out
suite — 30 new scenarios (bench-v4-035 … -064) authored after the
original 34, used to test a bundle on tasks the optimizer never saw. Its
material is cross-arm by nature (a per-trial table across three arms, and the
run handoff), so it is grouped by suite rather than split per task. Per-task h4
reports, if they are ever written, belong beside them as
v4/h4/<task>.md.
Cross-cutting analyses that span arms or experiments stay at this directory's
root with an experiment prefix — v4-skillset-variant-comparison.md,
v4-t4-merge-decisions.md, v4-cap-evolve-question-answers.md,
v5-benchmark-quality-review.md — rather than
inside an arm folder, since none of them belongs to a single arm.
v1's task ids are all of the form traces_parsec-aap2-<NNNN>, so the shared prefix is dropped
and only the number kept — v1/aap2-0047.md. v2's ids are already readable
(bench-aap2-004-failing-task-and-host) and are used whole. The mapping is not a convention to
re-derive by hand: every task row in results/results.json carries a report field with the exact
path, and scripts/build_task_reports.py reads it.
v4's ids are already readable (platform-005-wrong-owner-trap) and are used whole, like v2's.
Unlike v1/v2, v4's report pointer is carried per arm — results/v4/v4_t_e1/results.json and
results/v4/v4_g_e1/results.json (separate ledgers from results/results.json and from each
other) — and scripts/build_v4_t_task_reports.py / scripts/build_v4_g_task_reports.py read
those files instead of build_task_reports.py's.
The auto block
Each report opens with a block delimited by <!-- BEGIN:auto --> / <!-- END:auto -->. That block
is generated from results/results.json by
scripts/build_task_reports.py — status, experiment, category,
run, prompt, contract shape, flags, the measurement table, and links to whatever material exists for
that task.
Everything below the block is hand-written and the script never touches it.
python3 scripts/build_task_reports.py # create missing reports, refresh auto blocks
python3 scripts/build_task_reports.py --check # exit 1 if any auto block is stale (CI-friendly)
python3 scripts/build_task_reports.py --stats # coverage: how many written, how many analysed
Do not hand-edit inside the auto block — edits there are overwritten on the next run. Numbers stay derived; narrative stays human.
The diff block (v4 and v6 only — neither v5 arm gets one)
v5_b1_e1 is a zero-shot baseline with no best/ directory anywhere, so
build_v5_b1_task_reports.py calls the same scripts/v4_report_common.py::build_diff_block()
every v4 generator uses and it returns None for all 136 tasks — the same path v4's
never-optimized tasks already take. v5_g2_e1 reuses an already-optimized bundle
(artifacts/v4/v4_g_e1/best) as-is rather than optimizing per task, so it has no per-task
best/ directory either; build_v5_g2_task_reports.py takes the identical path. No
special-casing in either generator.
v6_g2_e1 DOES get a diff block (seed vs. artifacts/v6/v6_g2_e1/best/, the global shared
bundle, same as v4's G arm) — but only 66 reports exist, not v6's full 200-task set. Its
results.json only ever has per-task rows for the sealed test split (the only split with a
stable, full per-task ledger — see results/v6/v6_g2_e1/summary.md's "Why there's no per-task
F/C ledger"), so build_v6_g2_task_reports.py only has 66 rows to generate from. There is no
reports/v6/v6_g2_e1/<feedback-or-confirm-task>.md for any of the other 134 tasks, and there
never will be for this split design — not a gap to fill in later.
The v4 arms' diff block
Both v4 arms get a second generated section, <!-- BEGIN:diff --> / <!-- END:diff -->,
appended after the hand-written narrative (or, for the G arm, after the not-yet-written-narrative
placeholder — see below): real, file-by-file unified diffs of artifacts/v4/seed/*.md against the
arm's own candidate.
For the T arm, the 20 tasks where T2 accepted a candidate over the seed get the diff against
that task's artifacts/v4/v4_t_e1/<task>/best/*.md, one collapsible <details> per changed file
so the report stays scannable with the full edit one click away (e.g.
platform-004-events-then-config).
build_v4_t_task_reports.py regenerates it the same way it regenerates the auto block above; the
14 T-arm tasks with no best/ directory (13 T2 never ran on, plus
cost-030-threshold-not-an-anomaly, which only has a NOTE.md) get no diff block at all rather
than an empty one.
For the G arm, every one of the 34 tasks gets the same diff — artifacts/v4/seed/*.md against
the single shared artifacts/v4/v4_g_e1/best/*.md — since G evolves one bundle jointly across all
tasks rather than one per task. build_v4_g_task_reports.py generates it from
scripts/v4_report_common.py's shared diff logic, the same module the T generator uses, so the
splice markers and diff rendering stay identical between arms.
This answers Item 2 of this branch's cleanup — a prose summary of "what the optimizer tried" is
not the same as seeing the edit — without needing a separate diff viewer: GitHub and most local
Markdown renderers syntax-highlight a fenced ```diff block on their own.
Where this departs from skillsbench-history
The scheme is copied from skillsbench-history's reports/ and three things are deliberately
different, all for the same reason: parsec's measurements are not comparable across cells the way
skillsbench's were.
1. The measurement table is vertical, not horizontal. skillsbench-history renders one row per
task with a column per candidate, which is readable because every candidate there was measured at
the same trial count. Parsec's were not. v1's baseline sweep is n=1 and its pilot is n=30; v2's
seed is a reused n=3 measurement while everything else in that run is n=9. A horizontal row
would silently invite comparing an n=3 number with an n=9 number — which is exactly the mistake
that produced v2's headline. So each report lists one row per distinct measurement, with its n and
its split, and omits rows for measurements that do not exist rather than printing a null. The
length of the table is itself information: a 1-row table means the optimizer never ran on that task.
2. Narrative is pre-filled for more classes. skillsbench-history leaves most reports at
_Not yet analysed._ and says so honestly, because for a task the optimizer improved, the story
needs a human reading of the diff. Here, 28 of the 30 v1 tasks were never seen by the optimizer at
all, and for those "what happened" genuinely is the contract plus one trial — which is derivable.
The generator pre-fills seven v1 classes (vacuous-assertion contract, solved-at-baseline,
trajectory-matched, capped-at-0.5, zero-at-baseline, partial-assertions, and gate-failed, the last
of which stays a placeholder because a gate failure needs a rollout read). This is not padding: each
pre-filled report states a task-specific, derived fact, and none of them claim an optimizer result
that does not exist.
3. The "a rejected candidate is not a regression" caution is conditional. skillsbench-history
prints it on every report. Here it appears only when a candidate was actually evaluated on that task
(status == OPTIMIZED and a cand_0001 measurement exists) — otherwise it would be filler on 28 of
30 v1 reports and would train the reader to skip the italics. Note that once a report's hand section
is written, the author's own narrative supersedes it; the in-block caution "Read no cell without the
n beside it" is inside the auto block and therefore survives on all 40 reports regardless.
Status
40 reports, 40 fully analysed, 0 placeholders. Run --stats for the live count.
That is unusual for a first pass and it is worth saying why it was achievable: parsec's optimizer ran
on only 12 of the 40 tasks, so most reports are an account of a contract and a single trial rather
than of an optimization. The contracts are all committed here (results/v1/tasks.json,
results/v2/tasks.json), the per-trial vectors are committed
(results/v{1,2}/per_task_scores.json), and v2's ten scenarios each ship a hand-written
provenance.md in the source tree recording why the scenario exists and every defect fixed in it.
There was more written material per task than there were tasks to write about.
Where the material comes from
| source | covers | what it gives |
|---|---|---|
results/results.json |
40/40 | every number in every auto block: rewards per tag per split, n, errored-trial counts, contract shape, flags |
results/v1/tasks.json |
30/30 | v1 contracts: the demanded tool-call sequence and every answer_contains string |
results/v2/tasks.json |
10/10 | v2 contracts: expected calls, required/derived/forbidden facts, per-task weights |
results/v{1,2}/per_task_scores.json |
40/40 | per-trial reward vectors for every task in every run, errored trials as null |
results/v2/per_task_scores.csv |
10/40 | the hand-maintained per-task analysis, one note per task per iteration |
recipes/v1/PROJECT.md |
30/40 | the dated decision log for v1 — including the simulator tool-coverage note that explains 25 of the 30 trajectory: 0.000 rows |
results/v1/runs/run_20260814_214843/JOURNAL.md |
2/40 | the optimizer's own hypotheses, refutations and next-lever notes — the single richest source for v1 |
results/v2/runs/*/JOURNAL.md |
10/40 | same for v2, but see the caveat below |
<source>/harbor-tasks-v2.1/<task>/provenance.md |
10/40 | why the scenario exists, its discrimination axis, and a dated edit log for every defect fixed |
artifacts/ |
40/40 | the skill packages themselves — diff seed/ against best/ and the rejected candidates |
The v2 journal caveat: the headline run's optimizer wrote no handover at all — its entry
reads ## Iteration cand_0001 — (no handover written by the optimizer), and events.jsonl records
an optimizer_error for that candidate. Six edits shipped together with no statement of intent, so
every per-task attribution in the v2 reports is inference from the reward decomposition, not from the
record. The iter-1 run (run_20260816_202942) does have a full journal entry and is the better
read for how the optimizer was reasoning.
Four cautions when writing one
Say which run and which n a number came from. This branch's central finding is that the same
byte-identical seed skill measures anywhere from 0.000 to 0.367 in v1 and 0.643 to 1.000 in v2
depending only on the trial count and the run. A number without its n is not a result. The
ui/heatmap.html ladder table lists all 24 seed measurements side by side.
Check mean_valid against mean_all before quoting. Errored trials are written with
reward: 0.0 and are distinguishable from genuine zeros only via rollout.error. Both denominators
are carried everywhere in the ledger. They differ materially: v2's headline seed_train is 0.822
with errored trials dropped and 0.749 with them counted as zero. results/v2/per_task_scores.csv
uses the errored-as-zero convention, so its numbers legitimately disagree with the auto block
above them.
On a v1 report, check the reachability line before reading trajectory as a skill measurement.
When the 30-task sweep ran, two simulators were up and 25 of the 30 contracts demand a tool neither
served. Under a subset match one uncallable op fails the whole containment, so those 25 rows have
trajectory: 0.000 by construction and a reward capped at 0.5 — a fact about the harness, not the
skill. Every affected report says so in its auto block and carries the
trajectory-unreachable-tool-gap flag; the ledger carries
trajectory_tools_missing_at_baseline per task. The two optimized tasks (-0047, -0048) are the
exception worth knowing: their baseline rows are flagged, but by the 2026-08-14 pilot the tool had
become callable, and their reports are the only record of that change.
A rejected candidate is not a regression, and an accepted one is not necessarily an improvement.
Both halves matter here. v1's cand_0002 was rejected and is the only artifact that ever produced a
trajectory match on either of the two tasks it ran on. v2's accepted cand_0001 is recorded as
having broken two tasks, and at equal n one of them is flat to six decimal places and the other
improved by +0.133. Re-derive per-task verdicts from seed_train rather than reading them off
report.md.
Worked examples
Four reports carry the most, and are the best entry points:
v1-aap2-0047andv1-aap2-0048— the only two tasks v1's optimizer ever ran on. Two candidates, both rejected, and between them the only two trajectory matches ever recorded in the whole experiment. The case that a mean-based gate made the right call for the champion and the wrong call for the knowledge.v2-bench-aap2-001— the one large, clean, mechanistically explained win in the branch (+0.500at equaln), and the argument that what it fixed was a documentation defect in the seed skill rather than a reasoning failure.-
v2-bench-aap2-006— recorded by cap-evolve as broken by the champion; identical to the seed atn=9to six decimal places. The cleanest counterexample to trusting the run's ownfixed/brokelists. -
v1-aap2-0052is the shortest and the most damning: its contract demands zero assertions, an empty list scores a vacuous1.000, and half of its0.500baseline measures nothing at all.