Parsec · README.md source

Report pages

Raw results data, recipes, artifacts and presentations are not rendered on this site; links to them open the files in the source repo on GitHub. Internal cluster paths and host names are masked on this site.

parsec

Results, recipes, and artifacts from running cap-evolve against parsec (IBM/Red Hat-internal AAP2 job-diagnosis tasks).

This repo was migrated on 2026-11-10 from the parsec-history orphan branch of skillberry-ai/cap-evolve, with full commit history preserved. It is one of the per-benchmark result repos in the skillberry-runs org. Mentions of skillsbench-history and benchmark-history below refer to the cap-evolve branches as they were at the time; their content now lives in skillberry-runs/skillsbench and skillberry-runs/dashboard.

Why this repo exists

cap-evolve produces two kinds of output from a benchmark run. The first is library code: adapters, scoring logic, CI wiring — that belongs on main, reviewed as a normal PR. The second is a result narrative: what a specific run found, which candidates it tried, why one was accepted or rejected, what a number does and doesn't mean. That has no home on main (it isn't code) and no home on benchmark-history either — that branch is machine-written (by ci/benchmarks/lib/record.py on main) and periodically pruned; it carries the numbers, not the story behind them. This repo is where the story lives.

Four independent experiments are recorded here, sharing nothing except the repo — v1 and v2 both target parsec's AAP2 job-diagnosis domain, v4/v5/v6 are wider (see below):

experiment tasks source
v1 30 trace-extracted traces_parsec-aap2-* tasks, real production traces parsec-intake_v1 worktree
v2 10 hand-authored bench-aap2-* scenarios, one isolated simulator per task parsec-intake_v2 worktree
v4 34-task multi-agent AAP2/GCP/ICINGA benchmark (platform-*, cloud-*, cost-*, icinga-*), harbor-native parsec-intake_v4 worktree
v5 from-scratch 136-task rebuild of v4's shape, no shared content with v4 parsec-intake_v5 worktree
v6 unified 200-task set (v4's 34 + v5's 166), first arm with a hidden feedback/confirm/test split parsec-intake_v6 worktree

They are not comparable to each other — different task sets, different simulators, different scoring — see results/v1/summary.md's "v1 is not comparable to v2" table for exactly why.

v4 is a third, separate experiment against a different, larger task set (34 tasks across four categories vs. v1/v2's AAP2-only scope) and a different target (one shared 8-file multi-agent bundle vs. v1/v2's single SKILL.md). It now has two arms — T (task-by-task) and G (global, shared bundle) — each with its own ledger and heatmap, plus a cross-arm comparison; see results/v4/README.md.

v6 unifies v4's 34 tasks and v5's held-out 166 into one 200-task set, and is the first arm family in this repo to run with a genuine hidden split: feedback (visible to the optimizer), confirm (hidden, gates accept/reject), and test (sealed, scored once). Its Global arm (v6_g2_e1) is the first in this repo with a positive, completed, held-out test-split result. Its task-wise arms, T-opt (v6_t2_e1) and T-mrg (v6_t4_e1), add the T-vs-G comparison: a score-blind merge of 34 task-local edits matches the Global bundle on the sealed test. See results/v6/README.md.

Layout

directory what's in it
results/ the numbers — results.json (the generated ledger v1/v2 read from), plus per-experiment summary.md, tasks.json, per_task_scores.json, and the raw runs/ directories; v4/ holds two arms (v4_t_e1/, v4_g_e1/), each with its own separate results.json + summary.md, plus a v4-level README.md/comparison.md, deliberately not folded into the shared v1/v2 ledger (different task set, different shape); v5/ is a from-scratch 136-task rebuild with its own per-arm ledgers: the baseline arm v5_b1_e1/ and the G2-arm reuse v5_g2_e1/ (v4's accepted G2 candidate, no v5-specific optimization); v6/ is the unified 200-task set with its Global hill-climb arm v6_g2_e1/, the task-wise arms v6_t2_e1/ (T-opt) and v6_t4_e1/ (T-mrg, with the three core T-vs-G tables in RESULTS_FOR_PAPER.md) — its results.json has a different shape from v4/v5's (test-split-only task_ledger, plus a separate per-iteration iterations_detail — see its own summary.md for why there's no per-task F/C ledger)
recipes/ the cap-evolve configs a rerun needs, copied verbatim — and, critically, where each recipe disagrees with what its run actually did; v6's recipe includes split_ids.json, the full published feedback/confirm/test partition (see recipes/v6/v6_g2_e1/README.md)
artifacts/ the skill packages themselves: seed, champion, rejected, and (for v2) discarded candidates
reports/ one file per task — what changed, what worked, what didn't; auto-generated header, hand-written narrative below it; v6's reports are test-split only, for the same reason its results.json is
scripts/ the generators: build_results_json.py, build_task_reports.py, build_heatmap.py, extract_rollout_scores.py (v1/v2), their v4_t_e1/v4_g_e1-named counterparts for v4's two arms, build_v5_b1_* for v5's baseline arm, build_v5_g2_* for v5's G2-arm reuse, and build_v6_g2_* for v6's hill-climb arm — everything numeric in this branch is derived by one of these, never hand-typed twice. All of build_v4_g_task_reports.py/build_v4_t_task_reports.py/build_v5_*_task_reports.py/build_v6_g2_task_reports.py share v4_report_common.py's splice logic.
docs/ the design record — specs and implementation plans for parsec v4, plus its intake handoff; describes decisions, not code, so it lives here rather than on parsec-intake_v4
ui/heatmap.html a static, generated per-task/per-candidate heatmap for v1/v2
ui/heatmap_v4_t.html, ui/heatmap_v4_g_e1.html the same, one per v4 arm — separate pages, not a shared one, matching results/v4/'s per-arm ledgers
ui/heatmap_v5_b1_e1.html, ui/trials_v5_b1_e1.html v5's baseline arm: the same sortable heatmap, plus a per-trial view that draws every individual trial — 56% of v5's tasks vary across identical trials, which a page of means would hide
ui/heatmap_v5_g2_e1.html, ui/trials_v5_g2_e1.html v5's G2-arm reuse: cand_0004 vs. the baseline, same column shape with a v5_b1/Δ vs b1 pair in place of the baseline's v4 T1/Δ vs v4 pair
ui/heatmap_v6_t2_e1.html, ui/v6_tables_main_table.html (+ cost_table, size_table) v6's T-opt arm per train task (class, Seed, champion, steps); the three rendered T-vs-G core tables
ui/heatmap_v6_g2_e1.html v6's hill-climb arm, sealed test split only (seed vs. final champion) — no per-candidate columns, since only two capabilities were ever scored on this split
presentations/management_review_presentation/ hand-authored, not generated: the general cap-evolve pitch/roadmap deck (Reveal.js, vendored) — Parsec is one roadmap thread in it, not its subject
presentations/parsec_optimization_presentation/ hand-authored, not generated: the Parsec v1+v2 PoC deck (Reveal.js, vendored) for an IBM/Red Hat audience
presentations/presentation_v4/ fork of the Parsec deck above, being extended with v4 — cover slide updated so far, rest of the deck still v1/v2 only

No insights/, evidence/, handoffs/, or proposals/ here — those are skillsbench-history directories for a scope (87 skillsbench tasks, multiple transfer-eval folds) this repo doesn't have. If a future third parsec experiment needs one of them, add it then rather than pre-creating an empty directory now.

The lesson this repo is built around

skillsbench-history was built the same way twice, and the second time caught a real bug: hand-transcribed numbers had drifted from the data they were transcribed from. Every number that appears more than once in this repo — in a summary.md table, a reports/ auto block, ui/heatmap.html — is generated by a script in scripts/ from results/results.json, not retyped. Run a script's --check mode (where it has one) before trusting that a document is still current; reports/README.md documents the convention in full.

Working in this repo

Clone it like any repo — no worktree needed:

git clone git@github.com:skillberry-runs/parsec.git

Before you commit here

This repo's .gitignore already excludes __pycache__/, .env, .env.*, .cache/, and .DS_Store — but double-check git status before every commit anyway. Raw simulator rollouts (~96MB across v1/v2) and v4's raw .capevolve/ run directories are deliberately not committed here; they remain only in the source parsec-intake_v1/ parsec-intake_v2/parsec-intake_v4 worktrees. If you're adding a new experiment, keep it that way — commit the derived per-task score vectors and the recipe that produced them, not the raw logs.