Report pages
- Parsec v6 — G2 Results (ui/dashboard_v6_g2_e1.html)
- parsec — per-task heatmap (v1 traces · v2 authored) (ui/heatmap.html)
- Parsec v4 heatmap — v4_g2_e1 global-scope bundle (ui/heatmap_v4_g_e1.html)
- Parsec v4 T-arm heatmap — JB baseline vs v4_t1_e1 vs v4_t2_e1 vs v4_t4_e1 vs v4_t4_e2 (ui/heatmap_v4_t.html)
- Parsec v5 baseline heatmap — v5_b1_e1 (ui/heatmap_v5_b1_e1.html)
- Parsec v5 G2-arm heatmap — v5_g2_e1 (ui/heatmap_v5_g2_e1.html)
- Parsec v6 heatmap — v6_g2_e1 (sealed test split) (ui/heatmap_v6_g2_e1.html)
- v6_t2 feedback heatmap (ui/heatmap_v6_t2_e1.html)
- Parsec v5 baseline — per-trial view (v5_b1_e1) (ui/trials_v5_b1_e1.html)
- Parsec v5 G2-arm — per-trial view (v5_g2_e1) (ui/trials_v5_g2_e1.html)
- Parsec v6 cost table (ui/v6_tables_cost_table.html)
- Parsec v6 T vs G (ui/v6_tables_main_table.html)
- Parsec v6 Rule 4 (ui/v6_tables_rule4_table.html)
- Parsec v6 skill size (ui/v6_tables_size_table.html)
Raw results data, recipes, artifacts and presentations are not rendered on this site; links to them open the files in the source repo on GitHub. Internal cluster paths and host names are masked on this site.
parsec
Results, recipes, and artifacts from running cap-evolve against parsec (IBM/Red Hat-internal AAP2 job-diagnosis tasks).
This repo was migrated on 2026-11-10 from the parsec-history orphan branch of
skillberry-ai/cap-evolve, with full commit history
preserved. It is one of the per-benchmark result repos in the
skillberry-runs org. Mentions of skillsbench-history and
benchmark-history below refer to the cap-evolve branches as they were at the time; their content now
lives in skillberry-runs/skillsbench and
skillberry-runs/dashboard.
Why this repo exists
cap-evolve produces two kinds of output from a benchmark run. The first is library code:
adapters, scoring logic, CI wiring — that belongs on main, reviewed as a normal PR. The
second is a result narrative: what a specific run found, which candidates it tried, why
one was accepted or rejected, what a number does and doesn't mean. That has no home on
main (it isn't code) and no home on benchmark-history either — that branch is
machine-written (by ci/benchmarks/lib/record.py on main) and periodically pruned; it
carries the numbers, not the story behind them. This repo is where the story lives.
Four independent experiments are recorded here, sharing nothing except the repo — v1 and v2 both target parsec's AAP2 job-diagnosis domain, v4/v5/v6 are wider (see below):
| experiment | tasks | source |
|---|---|---|
| v1 | 30 trace-extracted traces_parsec-aap2-* tasks, real production traces |
parsec-intake_v1 worktree |
| v2 | 10 hand-authored bench-aap2-* scenarios, one isolated simulator per task |
parsec-intake_v2 worktree |
| v4 | 34-task multi-agent AAP2/GCP/ICINGA benchmark (platform-*, cloud-*, cost-*, icinga-*), harbor-native |
parsec-intake_v4 worktree |
| v5 | from-scratch 136-task rebuild of v4's shape, no shared content with v4 | parsec-intake_v5 worktree |
| v6 | unified 200-task set (v4's 34 + v5's 166), first arm with a hidden feedback/confirm/test split | parsec-intake_v6 worktree |
They are not comparable to each other — different task sets, different simulators, different
scoring — see results/v1/summary.md's "v1 is not comparable to
v2" table for exactly why.
v4 is a third, separate experiment against a different, larger task set (34 tasks across four
categories vs. v1/v2's AAP2-only scope) and a different target (one shared 8-file multi-agent bundle
vs. v1/v2's single SKILL.md). It now has two arms — T (task-by-task) and G (global, shared
bundle) — each with its own ledger and heatmap, plus a cross-arm comparison; see
results/v4/README.md.
v6 unifies v4's 34 tasks and v5's held-out 166 into one 200-task set, and is the first arm family
in this repo to run with a genuine hidden split: feedback (visible to the optimizer), confirm
(hidden, gates accept/reject), and test (sealed, scored once). Its Global arm (v6_g2_e1) is
the first in this repo with a positive, completed, held-out test-split result. Its task-wise
arms, T-opt (v6_t2_e1) and T-mrg (v6_t4_e1), add the T-vs-G comparison: a score-blind merge
of 34 task-local edits matches the Global bundle on the sealed test. See
results/v6/README.md.
Layout
| directory | what's in it |
|---|---|
results/ |
the numbers — results.json (the generated ledger v1/v2 read from), plus per-experiment summary.md, tasks.json, per_task_scores.json, and the raw runs/ directories; v4/ holds two arms (v4_t_e1/, v4_g_e1/), each with its own separate results.json + summary.md, plus a v4-level README.md/comparison.md, deliberately not folded into the shared v1/v2 ledger (different task set, different shape); v5/ is a from-scratch 136-task rebuild with its own per-arm ledgers: the baseline arm v5_b1_e1/ and the G2-arm reuse v5_g2_e1/ (v4's accepted G2 candidate, no v5-specific optimization); v6/ is the unified 200-task set with its Global hill-climb arm v6_g2_e1/, the task-wise arms v6_t2_e1/ (T-opt) and v6_t4_e1/ (T-mrg, with the three core T-vs-G tables in RESULTS_FOR_PAPER.md) — its results.json has a different shape from v4/v5's (test-split-only task_ledger, plus a separate per-iteration iterations_detail — see its own summary.md for why there's no per-task F/C ledger) |
recipes/ |
the cap-evolve configs a rerun needs, copied verbatim — and, critically, where each recipe disagrees with what its run actually did; v6's recipe includes split_ids.json, the full published feedback/confirm/test partition (see recipes/v6/v6_g2_e1/README.md) |
artifacts/ |
the skill packages themselves: seed, champion, rejected, and (for v2) discarded candidates |
reports/ |
one file per task — what changed, what worked, what didn't; auto-generated header, hand-written narrative below it; v6's reports are test-split only, for the same reason its results.json is |
scripts/ |
the generators: build_results_json.py, build_task_reports.py, build_heatmap.py, extract_rollout_scores.py (v1/v2), their v4_t_e1/v4_g_e1-named counterparts for v4's two arms, build_v5_b1_* for v5's baseline arm, build_v5_g2_* for v5's G2-arm reuse, and build_v6_g2_* for v6's hill-climb arm — everything numeric in this branch is derived by one of these, never hand-typed twice. All of build_v4_g_task_reports.py/build_v4_t_task_reports.py/build_v5_*_task_reports.py/build_v6_g2_task_reports.py share v4_report_common.py's splice logic. |
docs/ |
the design record — specs and implementation plans for parsec v4, plus its intake handoff; describes decisions, not code, so it lives here rather than on parsec-intake_v4 |
ui/heatmap.html |
a static, generated per-task/per-candidate heatmap for v1/v2 |
ui/heatmap_v4_t.html, ui/heatmap_v4_g_e1.html |
the same, one per v4 arm — separate pages, not a shared one, matching results/v4/'s per-arm ledgers |
ui/heatmap_v5_b1_e1.html, ui/trials_v5_b1_e1.html |
v5's baseline arm: the same sortable heatmap, plus a per-trial view that draws every individual trial — 56% of v5's tasks vary across identical trials, which a page of means would hide |
ui/heatmap_v5_g2_e1.html, ui/trials_v5_g2_e1.html |
v5's G2-arm reuse: cand_0004 vs. the baseline, same column shape with a v5_b1/Δ vs b1 pair in place of the baseline's v4 T1/Δ vs v4 pair |
ui/heatmap_v6_t2_e1.html, ui/v6_tables_main_table.html (+ cost_table, size_table) |
v6's T-opt arm per train task (class, Seed, champion, steps); the three rendered T-vs-G core tables |
ui/heatmap_v6_g2_e1.html |
v6's hill-climb arm, sealed test split only (seed vs. final champion) — no per-candidate columns, since only two capabilities were ever scored on this split |
presentations/management_review_presentation/ |
hand-authored, not generated: the general cap-evolve pitch/roadmap deck (Reveal.js, vendored) — Parsec is one roadmap thread in it, not its subject |
presentations/parsec_optimization_presentation/ |
hand-authored, not generated: the Parsec v1+v2 PoC deck (Reveal.js, vendored) for an IBM/Red Hat audience |
presentations/presentation_v4/ |
fork of the Parsec deck above, being extended with v4 — cover slide updated so far, rest of the deck still v1/v2 only |
No insights/, evidence/, handoffs/, or proposals/ here — those are
skillsbench-history directories for a scope (87 skillsbench tasks, multiple transfer-eval
folds) this repo doesn't have. If a future third parsec experiment needs one of them, add
it then rather than pre-creating an empty directory now.
The lesson this repo is built around
skillsbench-history was built the same way twice, and the second time caught a real bug:
hand-transcribed numbers had drifted from the data they were transcribed from. Every number
that appears more than once in this repo — in a summary.md table, a reports/ auto
block, ui/heatmap.html — is generated by a script in scripts/ from results/results.json,
not retyped. Run a script's --check mode (where it has one) before trusting that a document
is still current; reports/README.md documents the convention in full.
Working in this repo
Clone it like any repo — no worktree needed:
git clone git@github.com:skillberry-runs/parsec.git
Before you commit here
This repo's .gitignore already excludes
__pycache__/, .env, .env.*, .cache/, and .DS_Store — but double-check git status
before every commit anyway. Raw simulator rollouts (~96MB across v1/v2) and v4's raw .capevolve/ run directories are
deliberately not committed here; they remain only in the source parsec-intake_v1/
parsec-intake_v2/parsec-intake_v4 worktrees. If you're adding a new experiment, keep it that
way — commit the derived per-task score vectors and the recipe that produced them, not the raw logs.