Parsec · results/v6/v6_t4_e1/RESULTS_FOR_PAPER.md source

# v6 core results for the paper: Tables 1–4 (Seed, T-opt, T-mrg, Global)

Prepared 2026-10-22 for the agent that edits the MLSys 2027 PARSEC paper. This file is self-contained: every number needed for the four main tables and their prose is here, with its definition and source. The v6 T-mrg experiment is now complete, so the paper's green \missingresult{...} placeholders can be replaced.

Conventions to keep (from the paper's handoff.md): use train / validation / test; name the arms Seed, T-opt, T-mrg and Global; introduce them as Task-wise (T) and Global (G); put new wording in blue (\new{} / newcontent); do not use internal run names (v6_t2, v6_t4, m1/m2/m3) in manuscript prose. Names for the three merge methods are suggested in §1.


0. What changed, in one paragraph

The paper currently treats integration as the open risk: T-opt fits every selected training task, but its 34 private bundles cannot be deployed together. T-mrg now answers that question. Merging the 34 independently optimized English edits into one bundle, without looking at any score, produced a bundle that scores the same as Global on the sealed test split. Merge by order: test +0.046 over Seed; smart merge: +0.041; Global: +0.037. The task-by-task differences from Global (+0.009 ± 0.014 and +0.004 ± 0.012) are within noise. The T path built its bundle for 42–44% of Global's build cost (Table 2), but its merged bundles are 1.3–2.4× larger than Global's (Table 3). This reverses the hypothesis, taken from the historical v4 pilot, that task-local English edits would not compose. The reframing is therefore no longer "does integration work?". It becomes "it works; here is what integration costs and what it changes."


1. T-mrg protocol (needed for Methods §3.2 and Experimental Setup §4.3)

This also answers the open author comment in §3.2 about whether T-mrg uses validation.

Suggested paper name Rule Judgment? Build cost
Merge by order Apply the 34 diffs one after another with git merge -X theirs semantics, so a later diff wins any region it contests. Order: ascending T-opt gain (champion − Seed); ties broken by task name; the highest-gain edit is applied last. none (script) 10 s, no LLM
Smart merge A fresh Claude Code agent (Claude Opus 5.5) plus three sub-agents it spawned merged the 34 edits by meaning: dedupe overlapping rules, reconcile conflicts, keep each task's intent. It read only the Seed, the 34 diffs, each task's request, and each task's optimizer rationale with every score, reward and trial count removed. Every decision is logged. yes, without scores 49.6 min; 10.15M tokens; ≈ $14.16
Non-conflicting hunks Keep every hunk that no other task's hunk overlaps (91 of 188); drop every contested hunk. Hunk-level, not task-level: a task-level rule would keep only 2 of the 34 edits. none (script) 10 s, no LLM

Suggested wording for the §3.2 comment: T-mrg builds each merged bundle without seeing any score, freezes it, and evaluates it once on the train and test splits. It does not use the validation split; no merge method is selected on a held-out split, and all three methods are reported.


2. Table 1 — Reward on train, validation and test (main result)

Mean reward per split. Δ is the change from the Seed, paired by task; ± is the standard error of the per-task paired difference. The 95% CIs are task-level bootstrap intervals (10,000 resamples).

Arm Train (67) Validation (67) Test, sealed (66)
Seed 0.830 (10 trials/task) 0.821 (4) 0.813 (10)
Task-wise (T)
T-opt — 34 private bundles + Seed for the rest; not deployable 0.937, Δ +0.107 ± 0.017 (5) not measured not measured
T-mrg · merge by order 0.902, Δ +0.072 ± 0.018 (5) not used 0.858, Δ +0.046 ± 0.015 (5); CI [+0.016, +0.074]
T-mrg · smart merge 0.908, Δ +0.078 ± 0.021 (5) not used 0.854, Δ +0.041 ± 0.016 (5); CI [+0.011, +0.072]
T-mrg · non-conflicting hunks 0.880, Δ +0.050 ± 0.015 (5) not used 0.828, Δ +0.015 ± 0.018 (5); CI [−0.020, +0.049]
Global (G)
Global (cand_0005) 0.884, Δ +0.053 ± 0.017 (4) 0.814, Δ −0.005 ± 0.016 (3, gate trials) 0.850, Δ +0.037 ± 0.013 (10); CI [+0.011, +0.065]

Head-to-head on test, paired by task: merge by order − Global = +0.009 ± 0.014; smart merge − Global = +0.004 ± 0.012; non-conflicting − Global = −0.022 ± 0.015; merge by order − smart merge = +0.004 ± 0.014. On train: merge by order − Global = +0.019 ± 0.011; smart merge − Global = +0.024 ± 0.015.

Comments the table needs (caption or text): 1. The rows answer different questions. T-opt's train number combines 34 mutually exclusive bundles; it is a fit diagnostic, not a deployable artifact. Every T-mrg and Global row is one deployable bundle. Both T-mrg and Global were optimized using train tasks, so their train column is fit for both. Test is the held-out comparison. 2. Test result: merge by order and smart merge are statistically indistinguishable from Global. Non-conflicting hunks is the weakest: its test gain is not distinguishable from zero. So the value lies in the edits several tasks wanted to make, mostly shared rules. 3. Global's validation figure comes from the trials of the validation-gate run that accepted cand_0005 (195 trials, about 3 per task). It is paired against a 4-trial Seed remeasurement on validation. Do not use −0.018 (0.804 vs 0.821) here: that figure measured cand_0002, the previous accepted candidate, not the final bundle (paper_notes.md already records this). Either way, Global's validation gain is not distinguishable from zero while its test gain is positive. 4. Trial counts differ: 5 per task for T-mrg, 10 for Global on test. The SEs are similar because per-task variation dominates. 5. 3 → 5 trials (robustness check, for text or the supplement):

Train, 3 trials Train, 5 trials Test, 3 trials Test, 5 trials
Merge by order +0.085 ± 0.019 +0.072 ± 0.018 +0.059 ± 0.016 +0.046 ± 0.015
Smart merge +0.086 ± 0.022 +0.078 ± 0.021 +0.040 ± 0.019 +0.041 ± 0.016
Non-conflicting +0.050 ± 0.016 +0.050 ± 0.015 +0.011 ± 0.020 +0.015 ± 0.018

The two extra trials lowered merge by order by 0.013 and left the other two essentially unchanged. The ranking and the "equal to Global on test" conclusion did not change.


3. Table 2 — Cost, tokens and compute time

Table 2 is split in two. 2a is what each arm spent to produce its deployable bundle: no test split, no post-hoc measurement. 2b is what it costs to measure a frozen bundle, at the same 5 trials per task for every arm. Mixing the two, as an earlier draft did, charged T-mrg for a train-and-test evaluation and Global for a 10-trial test plus a post-run remeasurement, so the totals compared different amounts of measurement. The shared Seed measurements appear in neither: every arm reuses them.

Table 2a — cost to build the deployable bundle (main table)

Arm Eval $ Optim. $ Total $ Eval tokens Optim. tokens Total tokens Eval h Optim. h Total h Eval trials
T-opt (34 per-task searches) 38.26 281.09 319.35 11.3M 283.4M 294.7M 13.8 19.4 33.1 310
T-mrg · merge by order 38.26 281.09 (+0) 319.35 11.3M 283.4M (+0) 294.7M 13.8 19.4 (+10 s) 33.1 310
T-mrg · smart merge 38.26 295.25 (+14.16) 333.51 11.3M 293.6M (+10.2M) 304.9M 13.8 20.2 (+0.8) 34.0 310
T-mrg · non-conflicting 38.26 281.09 (+0) 319.35 11.3M 283.4M (+0) 294.7M 13.8 19.4 (+10 s) 33.1 310
Global (cand_0005) 402.87 355.02 757.88 124.9M 155.0M 279.8M 164.2 8.9 173.1 2,223

T-mrg = T-opt + the merge; the figure in brackets is the merge. Global's evaluation is its search: train screens and incumbent sweeps ($108.64, 585 trials) and validation gates ($294.22, 1,638 trials).

Table 2b — cost to measure a frozen bundle, 5 trials per task

Arm Test (66 tasks): $ / tokens / h Train (67 tasks): $ / tokens / h
T-mrg · merge by order $78.03 / 24.4M / 8.7 $70.45 / 22.1M / 7.8
T-mrg · smart merge $107.93 / 34.2M / 9.3 $92.99 / 29.5M / 8.1
T-mrg · non-conflicting $65.63 / 20.4M / 9.6 $55.19 / 17.1M / 8.1
Global (cand_0005) $70.09 / 21.8M / 24.7 — (its train reward comes from the search's own sweeps)

Global's row uses the first 5 of its 10 test trials per task (seeds 0–4). Its full 10-trial test cost $138.02 / 42.9M / 40.0 h. Measurement cost is what Table 3's bundle size predicts: per trial, the largest bundle (smart merge) costs most and non-conflicting least, with Global close to merge by order.

Table 2c — serving usage: one trial on the sealed test split (for a "Test usage" column)

Per-trial resource use of each deployable bundle, measured on the 66 sealed test tasks. For each task its valid trials are averaged, then averaged over the 66 tasks, so every task counts equally. "Per sweep" is one trial of every test task (the sum over tasks). Errored trials are excluded. T-opt has no row: its 34 bundles are not deployable and were never run on test.

Arm Test trials $ / trial Tokens / trial Agent time / trial (mean; median) One sweep of 66 tasks: $ / tokens / h
Seed (reference) 648 0.146 44.2k 211 s; 73 s 9.66 / 2.92M / 3.87
T-mrg · merge by order 330 0.236 73.9k 94 s; 70 s 15.61 / 4.88M / 1.73
T-mrg · smart merge 328 0.328 103.9k 100 s; 77 s 21.62 / 6.86M / 1.84
T-mrg · non-conflicting 327 0.201 62.4k 105 s; 81 s 13.26 / 4.12M / 1.93
T (avg. of the three merges) — 0.255 80.1k 100 s; 76 s —
Global (cand_0005) 651 0.211 65.5k 213 s; 76 s 13.92 / 4.33M / 3.91

Comments: 1. Per-trial cost follows the loaded prompt, not total bundle size. Relative to Global: merge by order costs 1.12× per trial, smart merge 1.55×, non-conflicting 0.95×. The Seed costs 0.69×. PARSEC loads the orchestrator, the shared context and one routed specialist per request. Global put most of its additions into shared_context.md, which every specialist request loads, so it is not the cheapest per trial despite being the smallest bundle. 2. Use dollars and tokens, not time, for this column. Median agent time is 70–81 s for every arm, but Global's and the Seed's means (211–213 s) are twice T-mrg's (94–105 s). The two were measured together in the Global run (2026-10-18), and slow simulator calls stretched their tail. T-mrg was measured on 2026-10-21 with no such tail. The mean time difference is infrastructure, not the bundle. If time is shown, show the median. 3. Trial counts differ (5 per task for T-mrg, 10 for Global and the Seed). Averaging per trial removes that difference, which is the point of this column. 4. How this fits Table 6 (the paper's cost table): Table 6 reports construction resources (Table 2a). This column adds the recurring cost of using the result. Over enough requests the serving difference matters: smart merge spends $0.117 more per trial than Global, so after about 3,600 requests its extra serving cost would exceed the $424 it saved in building. Merge by order ($0.025 more per trial, $439 saved) would take about 17,500. Treat these as illustrative: the benchmark's request mix is not production traffic.

Generator: examples/parsec/v6_t4/test_usage.py → results/v6_t4_e1/test_usage.json.

Definitions: - Eval is the target agent running tasks; Optim. is the optimizer agent proposing edits, plus the smart merge's LLM merger. The other two merges are 10-second scripts with no LLM. - Dollars and tokens are the per-trial records' own cost_usd and tokens, plus each run's recorded optimizer spend. The smart merge's $14.16 is an estimate: its measured tokens priced at Claude Opus 5.5 list price ($4 input, $20 output, $0.20 cache read, $5 cache write per million tokens). 63% of that is output tokens. - Compute time is each trial's own agent run time (recorded start to end), summed, plus each optimizer call's duration, summed. Work ran in parallel, so this is effort, not wall clock. Global's run took 66.1 h of wall clock. T-opt ran 34 searches in parallel over about a day. - Not counted: the shared Seed measurements ($215.29: train 10, validation 4, test 10 trials per task); Global's post-run validation remeasurement of cand_0002 ($53.20, 268 trials), which measured the previous candidate after the run; and a harness step in the T-mrg eval projects that re-scored some tasks a second time ($13.72–28.81 per method, unused in any result).

Comments the table needs: 1. Build cost: T reaches a deployable bundle for $319–334; Global spent $758. T is 42–44% of Global. Every merge method costs the same to build except the smart merge's $14. 2. Where the money goes. T spends 88% on the optimizer (34 small searches; each candidate is evaluated on one task). Global spends 53% on evaluation, almost all of it on validation gates that check each candidate across many tasks. That is the difference in kind between the arms. 3. Tokens are not an effort measure here. Totals look alike (295–305M for T, 280M for Global), but price per token differs by about 3×. Evaluation tokens cost about $3.2 per million in every arm. Optimizer tokens cost $0.99 per million for T and $2.29 for Global, because most optimizer tokens are cache reads: the optimizer re-reading its own context on every turn. Cache reads are billed at a tenth of input price but counted at full weight in the token total. Cache reads are about 95% of T's optimizer tokens (measured on the 13 optimizer transcripts still on disk) and 97% of Global's. If the paper keeps token columns, footnote this, and compare effort in dollars. 4. Compute time: Global used 5.2× T's build compute (173 h vs 33 h). Two factors multiply: Global ran 7.2× as many evaluation trials (2,223 vs 310), and its trials took longer on average (266 s vs 160 s). The second factor is partly the simulator: 9% of trials logged in the Global run took over 10 minutes, accounting for 82 of 295 hours, while T-mrg's evaluation had none. Table 2b shows the same effect on measurement: Global's 5-trial test took 24.7 h of compute against 8.7–9.6 h for each T-mrg bundle, for similar dollars. Present compute time as measured, but note that it partly reflects infrastructure speed during each run. Trial counts and dollars are the cleaner measures. 5. Optimizer time: T-opt's 19.4 h exceeds Global's 8.9 h because it ran 34 separate searches. 6. Not a controlled efficiency claim: the arms used different gates, budgets and trial counts. Keep the paper's existing caveat.

This changes numbers already in the paper: - The paper reports Global at $1,164.39. That is build $757.88 + 10-trial test $138.02 + post-run validation remeasurement $53.20 + shared Seed $215.29. Table 2 separates these; the like-for-like comparison is build cost. - The paper's split of Global's evaluation ($83.09 Seed sweep, $494.06 screens + gates, $232.21 test) mixes Seed and candidate measurements and includes the remeasurement in "gates". Per the trial records, Global's own screens and gates were $402.87, its test $138.02, the Seed's test $94.19, and the remeasurement $53.20. - The paper's 38.0 h (T-opt) and 51.5 h (Global) used the runs' runner_seconds, the wall time of each evaluation batch while its trials ran in parallel. That undercounts compute, by different amounts for T and G. With one definition for every arm (summed per-trial agent time plus optimizer time), build compute is 33.1 h (T-opt) and 173.1 h (Global). T-opt's figure excludes the 10 Seed trials it reused. The paper's cost table and the "summed compute time" sentences need updating. - T-opt's dollar figures are unchanged ($281.09 optimizer + $38.26 eval = $319.35; the paper's $281.12 + $38.25 = $319.37 differs only by rounding across 34 runs).


4. Table 3 — Skill edits in words

Word-level diff of each arm's eight skill files against the Seed (14,753 words), summed over files.

Arm Added Removed Net Files changed Resulting words vs Seed vs Global
Seed — — — — 14,753 — 0.59×
T-opt (34 champions, summed) +47,288 −460 +46,828 7 of 8 — — —
T-mrg · merge by order +26,855 −327 +26,528 7 of 8 41,281 +180% 1.65×
T-mrg · smart merge +45,511 −412 +45,099 7 of 8 59,852 +306% 2.39×
T-mrg · non-conflicting +18,595 −221 +18,374 7 of 8 33,127 +125% 1.32×
Global (cand_0005) +11,125 −818 +10,307 7 of 8 25,060 +70% 1.00×

For T-opt, a single champion (median over 34) adds 1,239 words (range 404–3,424), removes 3 (0–95), changes 2 files (1–4), and totals 15,977 words.

Definitions: words are whitespace-separated tokens in the eight skill files (merge logs and optimizer notes excluded). Added and removed come from a word-level diff per file, summed. A rewritten word counts once as removed and once as added. Net = added − removed. T-opt's row sums 34 separate diffs, each counted once: it is what T-mrg had to integrate, not the size of any deployable bundle. The T-opt and Global numbers match the paper's existing §5.4 figures.

Comments the table needs: 1. Small fixes, large merge: each task's fix is small, yet every integrated T bundle is larger than Global's. 2. How much each merge kept of the words the 34 champions added: smart merge 96%, merge by order 57% (a later hunk replaces any earlier hunk it overlaps), non-conflicting 39%. The smart merge reconciled conflicts but did not compress. 3. Global rewrites the most: it deletes the most Seed text (818 words) while adding the least. It is the only arm that rewrites existing Seed text to a noticeable extent. 4. Size has a running cost: per-trial agent tokens on train were 37.5k for the Seed, 59.6k for Global, and 52.1k / 66.3k / 88.0k for non-conflicting / merge by order / smart merge. Per-trial dollars were $0.124, $0.191, $0.168, $0.212 and $0.278. 5. Per-file numbers for every arm already exist (size_table.json) if a per-file table is wanted.


5. Table 4 — Running example: AAP2 Rule 4 ("don't re-fetch job data")

The bug. The Seed's AAP2 specialist says, in its Critical Rules:

  1. Don't re-fetch job data. If a prior query_aap2 call already returned job metadata or events, extract what you need (steps, playbook events, errors) from the existing result. Do NOT make a redundant second call to the same job.

Its Investigation Flow, step 5, then requires a second call on the same failed job: "If the job failed, also call get_job_events + failed_only=true". The agent resolves the contradiction by skipping the events call, so it misses the failing task's role, which only the events carry. Training task platform-004 exposes this: Seed 0.760 over 10 trials. The same fix was found on v4 and merged upstream as rhpds/parsec PR #57 (2026-10-14), which shows what a reviewed human fix of this bug looks like.

Table 4a — word counts (main text)

Rule 4 is the numbered item itself. "Job-read guidance" is any dedicated section that says which job action returns which field: "Choosing the job read", "get_job_log vs get_job_events", "What the log carries vs. what the events carry", and "get_job is the FIRST read". Words are whitespace tokens; added/removed come from a word-level diff against the Seed's Rule 4, as in Table 3.

Arm Rule 4 Rule 4 words (added / removed) Contradictory Seed sentence still present? Job-read guidance elsewhere (words) platform-004, train
Seed original 39 yes 0 0.760 (10)
PR #57 (upstream, from v4 T-opt) rewritten 111 (+94 / −22) no 0 —
Task-wise (T)
T-opt (platform-004 champion) rewritten 74 (+59 / −24) no 504 1.000 (5)
T-mrg · merge by order Seed text + appended exception 102 (+63 / −0) yes 1,263 1.000 (5)
T-mrg · smart merge rewritten, 3 tasks' exceptions folded in 232 (+219 / −26) no 1,682 1.000 (5)
T-mrg · non-conflicting unchanged 39 (+0 / −0) yes 1,263 1.000 (5)
Global (G)
Global (cand_0005) unchanged 39 (+0 / −0) yes 233 0.912 (4)

The last column is mean reward over valid trials (count in brackets). For Global it is its train incumbent sweeps. For T-opt it is the champion's own 5-trial evaluation, which selected it.

Table 4b — the texts (supplement)

Verbatim; emphasis as in the source. Only the Rule 4 item is shown, plus a one-line pointer to where the arm's other job-read guidance sits.

Seed (39 words):

  1. Don't re-fetch job data. If a prior query_aap2 call already returned job metadata or events, extract what you need (steps, playbook events, errors) from the existing result. Do NOT make a redundant second call to the same job.

PR #57, upstream (111 words):

  1. Don't re-fetch job data — but a different action is not a re-fetch. If a prior query_aap2 call already returned what you need, read it from that result instead of asking again. A call is redundant only when it repeats the same action with the same arguments. Different actions on one job are complementary: get_job_log returns the log text, get_job_events returns the structured failed-task rows (role, play, task, host, error_msg), and neither response contains the other's data — so "I already queried this job" is not a reason to skip a different action on it, including the get_job_events + failed_only=true call required by step 5 of the Investigation Flow below.

T-opt, platform-004 champion (74 words):

  1. Don't re-fetch job data. If a prior query_aap2 call returned a field, read it from that result instead of calling again. This forbids repeating the same action on the same job — it does NOT cover a different action. Different query_aap2 actions return different fields, so calling get_job_events after get_job_log on the same job is complementary, not redundant. Never skip a different-action call on the grounds that you already have "the job data".

Elsewhere: a new "get_job_log vs get_job_events" section with a field table and a worked example (504 words), plus a required events call added to Step 1 of the job workflow.

T-mrg · merge by order (102 words: the Seed sentence kept, another task's exception appended):

  1. Don't re-fetch job data. If a prior query_aap2 call already returned job metadata or events, extract what you need (steps, playbook events, errors) from the existing result. Do NOT make a redundant second call to the same job. This applies only to a call that returned data. A call that timed out returned nothing to extract, so re-issuing it is not a re-fetch — it is the first real attempt. One retry of a timed-out call is within budget: a gap you could have closed with a single retry is not an acceptable gap. Do not retry it a third time.

Elsewhere: the "different action is not a re-fetch" idea survives only in separate job-read sections (1,263 words), including platform-116's "Choosing the job read". Step 5 of the flow became conditional ("if the job failed and you expect a task-level error"). Why: three tasks edited these lines: platform-032 (replaced Rule 4 with a "two dead calls" rule; 20th in gain order), platform-004 (the in-place rewrite; 25th), and platform-158 (kept the Seed sentence and appended a timeout exception; 34th, the highest gain). Under -X theirs the last edit wins a contested region, so platform-158's version stands, Seed sentence included.

T-mrg · smart merge (232 words; renumbered as rule 7 in the merged list):

  1. Don't re-fetch job data — but a different action, or a failed call, is not a re-fetch. If a prior query_aap2 call returned a field, read it from that result instead of calling again. This forbids repeating the same action on the same job after it returned data. It does NOT cover: - A different action. Different query_aap2 actions return different fields, so calling get_job_events after get_job_log (or get_job alongside either) on the same job is complementary, not redundant. Never skip a different-action call on the grounds that you already have "the job data" — see "Choosing the job read". - A call that failed without returning data. A tool call that returned an error, a timeout, or no payload at all returned nothing to extract, so re-issuing that exact call once is not a re-fetch — it is the first real attempt (see the transient-failure retry rule in the shared context). A gap you could have closed with a single retry is not an acceptable gap. Do not retry it a third time; after the retry also fails, say which call failed and which specific question you therefore could not answer. Never substitute a link for a value the user asked you to read: if the user asked what a file contains and the fetch failed, the retry is mandatory, because handing back a URL answers none of the question.

Elsewhere: job-read guidance (1,682 words) and the required events call in Step 1, both cross-referenced from the rule. The three tasks folded in are platform-004 (different action), platform-158 (timed-out call, retry once) and platform-152 (retry before reporting; never substitute a link).

T-mrg · non-conflicting hunks: Rule 4 unchanged (the Seed text above). All three edits to it (platform-032, platform-004, platform-158) were contested, so all three were dropped. The uncontested job-read sections survive (1,263 words), so the bundle carries the prohibition and its exceptions in different places.

Global, cand_0005: Rule 4 unchanged (the Seed text above). Elsewhere: step 4 of the Investigation Flow now points to a new "Choosing the job read" section (233 words), which opens: "get_job, get_job_log, and get_job_events return DIFFERENT data about the same job. Calling more than one of them is not a redundant re-fetch — pick by what the question needs, and issue them in parallel", and says of the events call: "Call this for every failed job".

What Table 4 shows (comments for the text)

  1. Two ways to fix the same bug in English. Repair in place: T-opt and PR #57 rewrite Rule 4 so the contradiction disappears. Override elsewhere: Global leaves the rule and adds a later section that contradicts it. Both work on platform-004 (1.000 vs 0.912). The in-place fix is a 10-line change a reviewer can read and upstream as is: it is what PR #57 is.
  2. Two ways to integrate several fixes. The smart merge integrated by meaning: one rule carrying all three tasks' exceptions, with no contradiction left. Merge by order integrated by text position: the highest-gain edit to those lines won (platform-158), and it was written against the Seed sentence, so the prohibition stayed and the "different action" exception landed in another section. Non-conflicting hunks dropped all three contested edits.
  3. The prompt no longer has to be consistent to score. Three of the five arms (merge by order, non-conflicting, Global) keep the contradictory Seed sentence, yet every arm's bundle reaches 0.91–1.00 on platform-004. This is the paper's interpretability point made concrete: the score cannot tell an in-place repair from an override, but a reviewer can.
  4. Size. The rule itself stays small (39–232 words) in every arm. The job-read guidance around it ranges from 233 words (Global) to 1,682 (smart merge), which repeats Table 3's pattern at the scale of one rule.
  5. Upstream status. PR #57 came from the v4 T-opt edit; the v6 T-opt edit is near-identical in content. The paper may say that this repair was merged into the maintained PARSEC system (already allowed by the handoff, paper_notes.md §3.5).

The supplement's existing two-column Rule 4 table (T-opt vs Global) becomes Table 4b. The failed-event retrieval rates already in the paper (22% → 41% on test, 20% → 29% on validation) belong to Global only and stay as they are.


6. Supporting results (text or supplement)

6.1 How much of T-opt's local gain survives integration (train)

On the 34 optimized train tasks the Seed scores 0.788 and every T-opt champion 1.0 (+0.212).

Δ on the 34 optimized tasks share of T-opt's gain kept Δ on the 33 tasks T never touched of those 33, tasks down > 0.1
Smart merge +0.199 ± 0.023 94% −0.046 ± 0.020 6
Merge by order +0.154 ± 0.028 73% −0.013 ± 0.008 1
Non-conflicting +0.114 ± 0.024 54% −0.016 ± 0.010 2
Global +0.124 ± 0.026 (59%) −0.020 ± 0.014 4

The smart merge keeps almost all of the local gain but costs the most on tasks no edit was written for. Dropping contested hunks loses about half the gain. (Global's percentage is shown in brackets because Global did not start from the T-opt champions.)

6.2 Per-task movement on test (release-gate question)

tasks up > 0.05 tasks down > 0.05 worst task (Δ)
Merge by order 34 10 platform-141 (−0.27)
Smart merge 31 13 platform-135 (−0.37)
Non-conflicting 25 18 platform-141 (−0.62)
Global 27 6 platform-131 (−0.40)

Every deployable bundle has at least one severe per-task regression, so the paper's release-gate point applies to both paths, not only to Global. The T merges move more tasks in both directions. Global's platform-131 regression is much milder under the smart merge (−0.04) and merge by order (−0.13).

6.3 Routing-gap tasks (outside the editable surface)

Nine test tasks are sent by the regex fast router to a specialist that lacks a required tool. Every arm is slightly negative on them and positive on the other 57:

9 routing-gap tasks other 57
Merge by order −0.032 +0.058
Smart merge −0.023 +0.051
Non-conflicting −0.006 +0.018
Global −0.025 +0.047

This supports the paper's existing claim that skill optimization cannot repair the router.

The Rule 4 running example is Table 4 (§5).


7. Caveats to carry into Limitations

  1. The Seed was not re-measured for T-mrg. T-mrg was evaluated on 2026-10-21. Its Seed reference is the Global run's Seed: train on 2026-10-16, validation on 2026-10-17, test on 2026-10-18. Global's bundle and Seed were measured during the same run. Drift in the simulator, gateway or target model between those dates would bias T-mrg's Δ but not Global's. A same-day 5-trial Seed run on test would cost about $40–80 and remove this confound; recommended before submission.
  2. Three methods, one test split. All three merges were scored on the same sealed test split, so no method may be presented as chosen because it scored best there. Report all three, or name one as primary for an a-priori reason (merge by order is the natural no-judgment baseline). To select a single method by a held-out rule, score all three on validation (about $55–95 per method at 5 trials).
  3. No validation gate on T-mrg. The merges are static and ungated. That they match Global on test is a result, not part of the protocol. The paper may note, carefully, that Global's validation-gated search did not beat ungated T-mrg on test in this run.
  4. One run per arm: one T-opt run, one Global optimizer seed, one smart-merge run. The historical v4 pilot merge failed. v6 differs in task count, the per-task protocol and the merge methods, so v4 and v6 are not a replication. Keep v4 out of the main narrative as before.
  5. Errored trials are left out, never scored as zero. These tasks have fewer than 5 trials: - Merge by order: train cost-407 (3). - Smart merge: test cloud-026 and platform-114 (4 each). - Non-conflicting: train cloud-057 and cloud-074, test cloud-027 (2 each).

8. Paper locations that need to change (as of main at 4fef9ff)


9. Sources

In parsec-history (this branch): results/v6/v6_t4_e1/, artifacts/v6/v6_t4_e1/, results/v6/v6_t2_e1/, artifacts/v6/v6_t2_e1/, ui/v6_tables_*.html. Code: examples/parsec/v6_t4/, v6_t2/ on main (skillberry-ai/cap-evolve#904). The table below gives paths in the source parsec-intake_v6-t2 worktree

What Where (relative to the worktree)
T-mrg plan and blind rule examples/parsec/v6_t4/PLAN.md
Merge inputs (Seed, 34 diffs, scrubbed rationales) results/v6_t4_e1/merge_inputs/
Frozen merged bundles + merge logs results/v6_t4_e1/{m1_order,m3_smart,m2_nonconflict}/ (m1 = merge by order, m3 = smart, m2 = non-conflicting)
Merge time, tokens, cost results/v6_t4_e1/merge_cost.json, m3_smart_usage.json
Pooled 5-trial rewards results/v6_t4_e1/eval/v6t4_{m1,m3,m2}_5t.json
Table 1 analysis (deltas, CIs, subsets, head-to-head) examples/parsec/v6_t4/analyze_t4.py → results/v6_t4_e1/eval/analysis*.json
Table 2 examples/parsec/v6_t4/cost_table.py → results/v6_t4_e1/cost_table.json
Table 3 examples/parsec/v6_t4/size_table.py → results/v6_t4_e1/size_table.json
Table 4 examples/parsec/v6_t4/rule4_table.py → results/v6_t4_e1/rule4_table.json (each arm's Rule 4 verbatim, word counts, platform-004 reward)
Rendered tables (HTML) results/v6_t4_e1/{main_table,cost_table,size_table}.html
Narrative report results/v6_t4_e1/README.md
T-opt run (unchanged) results/v6_t2_e1/
Global run (unchanged) v6_g2_e1, PR skillberry-ai/cap-evolve#887

Global's validation reward for cand_0005 is computed from the sealed validation rollouts of the Global run: tag r004__cand_0005__seq* for the candidate and final__seed for the Seed.