SpreadsheetBench · docs/decisions.md source

exp895 decisions log (issue skillberry-ai/cap-evolve#895)

Every entry: date, decision, who decided, evidence. No credentials here.

2026-10-20 — Code pin

2026-10-20 — Data

2026-10-20 — Agent (Gemma) route — DECIDED by Boaz

2026-10-20 — Optimizer (GLM) route

2026-10-20 — Compute policy — DECIDED by Boaz

2026-10-20 — Scorer

2026-10-21 — Container start reliability (approved by Boaz)

2026-10-21 — P1 throughput probes (runs/p1_probe/1892653/probe_{gemma,glm}.jsonl, calls_*.jsonl)

One wave of N parallel requests per level, through the gateway, no gateway cap. Gemma-4-31B-it (RITS), ~7.6k in / 1k out, temperature 0: | N | ok | p50 s | p95 s | req/min | out tok/s | |---|---|---|---|---|---| | 4 | 4/4 | 30.8 | 30.9 | 7.8 | 129 | | 8 | 8/8 | 28.7 | 28.7 | 16.7 | 279 | | 16 | 16/16 | 29.9 | 30.0 | 32.0 | 533 | | 32 | 32/32 | 39.5 | 39.7 | 48.3 | 805 | | 48 | 48/48 | 40.4 | 40.6 | 70.8 | 1,181 | | 64 | 64/64 | 46.7 | 47.1 | 81.4 | 1,356 | | 96 | 96/96 | 49.4 | 49.8 | 115.2 | 1,920 | - 268 calls, all 200, 0 upstream retries. No saturation up to N=96 (latency +60% from N=16 to 96, throughput still rising).

GLM-5.3 (Curve Bender), ~28.2k in / ~2k out, reasoning_effort=low: | N | ok | p50 s | p95 s | req/min | out tok/s | |---|---|---|---|---|---| | 4 | 4/4 | 89.3 | 89.9 | 2.7 | 89 | | 8 | 8/8 | 59.2 | 60.1 | 8.0 | 263 | | 16 | 16/16 | 59.7 | 63.8 | 14.9 | 495 | | 32 | 32/32 | 61.5 | 67.8 | 26.0 | 856 | | 48 | 48/48 | 84.7 | 123.7 | 22.5 | 735 | | 64 | 64/64 | 93.0 | 164.4 | 22.7 | 748 | | 96 | 96/96 | 125.1 | 238.4 | 23.8 | 779 | - 268 calls, all 200 after retry; 10 upstream ReadTimeouts (180 s gateway timeout), all in one burst (t≈679 s, during the N=64 wave), all recovered. - GLM saturates at N≈32: throughput flat ~23 req/min beyond it while p95 latency climbs 68 s -> 238 s. Matches the issue's earlier "GLM peaked around 53 concurrent" only loosely — our ceiling is lower with 28k-token prompts.

2026-10-21 — P1 episode probe (runs/p1_episodes/1897272/episodes_joined.jsonl, calls_gemma.jsonl)

10 train tasks x 1 trial, concurrency 10, v2 seed, helpers ON, host (dedicated, span[hosts=1]). - 10/10 scored, 0 infra; pass 8/10. 91 Gemma calls; 9.1 calls/episode mean, 4 median. - Tokens/episode: prompt median 9,340, mean 117,451 (heavy tail: 2 episodes hit the 30-turn cap at 351k and 751k); completion mean 5,619. RITS reports cached_tokens = 0, so no cache discount is assumed. - Wall/episode as logged: median 400 s, max 1,170 s. Every episode's FIRST call stalled ~300 s (10 ReadTimeouts at the gateway's 300 s Gemma timeout, all at t+300 s, all retried and answered in 17-50 s). Non-retried call p50 = 21 s, so the steady-state episode wall is ~100 s median; the 300 s is a cold-burst stall to tune (gateway Gemma timeout -> ~90 s). - Machine (): max 10 containers, load 82.7 (host shared with others), /tmp free >= 774 GB. - Estimate (Gemma list price proxy 0.13/0.38 $/M): ~$0.017 per episode at the mean -> ~$38 agent cost per R=2,160 search, ~$1.1k for 30 searches, ~$0.5k for the 27,720-episode final window. GLM optimizer cost NOT yet measured (needs P2 smoke).

2026-10-21 — Gateway Gemma read timeout 300 s -> 90 s

2026-10-21 — Spec-key audit of cap-evolve @ 520ecbe8b (+ #903) and the decisions it forced

Audited every §6 spec key against the code (survey + spot checks). Findings that change the design: - actions, dataset_source (run tags only) and the V4 keys cost_lambda/cost_accept/cost_metric/hc_compress_every are not read: unknown top-level keys are silently dropped (specfile.read_yaml has no schema). The V4 keys are confirmed gone. - target_model does NOT pick the agent model. The adapter takes MODEL from env. We set both and check they agree. - max_metric_calls counts task x trial rollouts, seed included. By default it is only checked between iterations, so it can overshoot R by one full eval. Decision: budget_reserve: true, which reserves before dispatch and raises BudgetExhausted before an eval that would overshoot (harness.py:385-399, rundir.py:357). This is the hard counter §5 requires. - finalize default is true and env CAPEVOLVE_FINALIZE overrides it. We set finalize: false and unset the env in the job. - Paired gate: footprint gating is on by default (val tasks outside an edit's footprint get Δ=0, plus an SE floor). The issue's gate is "mean paired Δ > 0.5·SE". Decision: gate_footprint: false, so the gate is exactly the stock paired gate on the raw per-task vector. Recorded as the reading of "stock paired gate". - Helper lint/smoke never run under hill-climb (only agent-optimize's pregate calls lint_candidate/run_smoke). Decision (Boaz): a common gate in the shared rollout layer, identical for EVERY arm. It runs before any rollout; a failing candidate scores 0 with no Gemma calls, and its full eval (tasks x k) is still debited from R. - The claude-code optimizer is unsandboxed (full os.environ, Bash allowed, no CLAUDE_CONFIG_DIR). It would see ~/secrets, the gold data and the scorer. Decision (Boaz): a bwrap shim named claude, first on PATH: read-only system, only the work dir writable, a private HOME/CLAUDE_CONFIG_DIR, gold/scorer/secrets not mounted, network only to the gateway. Reused for every Claude-Code-driven arm (CapEvolve, EvoSkill, Meta-Harness).

2026-10-21 — Optimizer sandbox verified (runs/sandbox_test/1900403, logs/sbxtest.1900403.out)

2026-10-21 — Common helper gate implemented (branch exp/sbv400-895)

2026-10-21 — CapEvolve P2 smoke (runs/capevolve/smoke_s0, job 1903524, )

2 train tasks (10452, 105-24) x k=3, R=60, max_iterations=1, concurrency 6. RC=0, wall 13.6 min. WORKED (evidence in run_main/): - k=3: 6 rollouts per eval, trial_rewards per task [1,1,1]; rollouts/val has 12 files (2 tasks x 3 trials x {seed, cand_0001}). - R counter: spent.metric_calls=12 of 60, under budget_reserve. Iterations 1/1. Seed not re-evaluated on train (train==val note). - Optimizer: sandboxed Claude Code on GLM produced cand_0001: prompt.md + SKILL.md edited, new skills/.../scripts/cell_ops.py (206 lines). The common helper gate PASSED it ("helper gate passed for tree 131156adfed8"). - SKILL.md reached the agent: the cand_0001 rollout system prompt (4,135 chars) contains the skill name and cell_ops. - Gate: paired Δ=0, SE=0 -> STRICT fallback -> rejected; champion = seed. Correct, and expected on 2 already-solved tasks. PROBLEMS FOUND (fix before any real search): 1. GLM spend was not metered: 34/35 optimizer calls are STREAMED /v1/messages and the gateway only parsed usage on non-stream replies, so the ledger has 0 tokens. cap-evolve's own estimate: optimizer_tokens=1,606,606, optimizer_usd=4.17 (its internal price). FIX: gateway now parses SSE usage (message_start + message_delta; OpenAI final chunk) and adds prompt_total = cache-inclusive input tokens. Unit-checked on both shapes. Needs re-verification in the next smoke. 2. No optimizer transcript kept, so the §4.1 isolation scan had nothing to read. FIX: CAPEVOLVE_OPTIMIZER_TRANSCRIPT now points at runs/.../optimizer_transcript.jsonl (run-optimizer switches to stream-json and redacts secrets). 3. Gemma ReadTimeouts: 18 retries, in 3 bursts of 6 at t+90, 182 and 275 s. Each burst = the 6 parallel episodes' first calls. Same cold-start stall as P1, now with the 90 s timeout, and it took 3 rounds to clear. The episodes still finished (0 infra errors) but lost ~4.5 min. Next: probe whether the stall is per-episode-first-call or RITS cold start, before choosing between a longer first-call timeout and a pre-warm call. 4. Claude Code logs unrecognized_model rits/zai-org/glm-5-3 (it doesn't know the id). Harmless: all 35 calls were 200. The capability validator warned that SKILL.md lacks "Use when…" and cell_ops.py lacks --self-check. These are advisory, did not block, and are fed back to the optimizer. - Saturation: the 2 smoke tasks pass 3/3 at seed (headroom 0). Memory rule: a SATURATED baseline is not a real search. The real searches use all 120 tasks, where P1 measured 8/10 on train, so there is headroom.

2026-10-21 — Launch CapEvolve before the competitors are ported (Boaz)

2026-10-21 — After the VPN outage (18:20Z): state, and the Gemma-timeout decision (Boaz: keep 90 s, no restart)

2026-10-21 (Opus re-verification, ~19:30Z) — two correctness fixes in arms/common/rollout.py

Re-checked every Sonnet-era change before Boaz went offline. The 3 CapEvolve searches are correct: spec per seed, 120-id split == official train∪val with 0 test overlap, gated adapter (sha c035c75e…), budget_reserve active (360 reserved for val/seed). Two bugs found in the COMMON rollout (affect competitor arms only; CapEvolve uses cap-evolve's own harness): 1. Infra-missing trials were averaged in as 0.0, which violates §0.3. TaskResult.mean now averages MEASURED trials only. A task with no measured trial is .missing and EvalResult.mean drops it. Unit-tested with a fake adapter: infra trial excluded, all-infra task dropped, candidate error = real 0, refund correct. 2. Trial seeds collided. The issue's 10_000seed + 100call + trial wraps into the next seed once an arm makes >= 100 evaluation calls (GEPA minibatches do), and the +1000attempt retry term collided too: 23,070 collisions over the reachable space (seed<=2, call<720, attempt<=3, trial<3). Deviation from §5, recorded: trial_seed = 1_000_000seed + 1000eval_call + 100attempt + trial, which gives 0 collisions for seed<=2, call<720 (R=2160 / 3), attempt<=3, trial<3. CapEvolve's own harness seeds trials as base_seed + k (its native convention), so "one convention for every arm" holds only across the competitor arms. Reported, not hidden. - gepa / gepa_full smokes ran on the OLD rollout.py (before both fixes). Their smokes had 0 infra trials, so fix 1 did not change their numbers. Their seeds came from the old formula, which is fine for a 3-call smoke. Every real search runs on the fixed file. - Known shared limitation, NOT fixed (so every arm stays identical): adapter.score() labels EVERY errored rollout "Infrastructure error ... do not optimize against it", including candidate-caused ones (helper gate, template contract, helper crash). Scoring is right: these count as 0 under errored_trial_policy=fail and in CommonRollout. Only the feedback TEXT shown to optimizers is misleading. A fix was written and reverted: the 3 live CapEvolve searches load a frozen adapter copy (sha c035c75e…), so a fix would give competitors different feedback from CapEvolve (§0.4). Same handicap for all arms. To be fixed for all arms at once if searches are ever restarted.

2026-10-22 ~05:30Z — Competitor searches launched (Boaz: "launch the competitor searches; we discuss the gate after")

2026-10-22 ~17:30Z — GLM auth outage (14:03Z) killed 7 competitor searches; gateway fixed; relaunched

2026-10-22 ~18:00Z — Test scoring per search, with an in-window seed anchor (Boaz; DEVIATION from §0.2/§9)

2026-10-22 ~19:00Z — MIPROv2 searches launched (Boaz: yes)

2026-10-22 ~19:15Z — Slot count (-n) for search jobs: review only, NO change tonight (Boaz: if in doubt, change nothing)

2026-10-23 ~05:40Z — Morning housekeeping (Boaz approved)

2026-10-23 — Slot count: -n 2 for NEW submissions (Boaz: "2 should be more than enough, check first")

2026-10-23 ~08:20Z — Helper gate: LEAVE AS IS (Boaz: option 1)

2026-10-23 ~22:10Z — GEPA s1 search ENDED IN ERROR (not tested; awaiting Boaz)

2026-10-23 ~23:40Z — RITS Gemma OUTAGE (not acted on; Boaz asleep, instruction: do not change/kill running tasks)

2026-10-24 06:50Z - recovery of GEPA s0, GEPA s1, WikiSkill s2 (Boaz: "recover and continue the errored arms")

2026-10-25 — Preparing capevolve_v2 (main + PR #914, k_se 0.2) for a NEW dedicated session (Boaz / Osher)

2026-10-25 — [v2-session] capevolve_v2 intake, template, check, first smoke

2026-10-25 ~13:30Z — Relaunch of the 4 parked searches with fixes + dedicated RITS keys (Boaz: "fix ... run with a dedicated Rits key")

2026-10-25 ~22:20Z — SkillOpt s1/s2 test windows on key 6; Trace2Skill s2 test window resumed after a lost host (Boaz)

2026-10-26 ~09:30Z — Gemma gateway timeout 90 s -> 300 s for NEW gateways (Boaz: "You can raise the timeout to 300")

2026-10-26 ~14:50Z — FREEZE (bstop, not kill) of low-priority searches to free Gemma capacity (Boaz)

2026-10-26 ~19:50Z — FREEZE round 2 (bstop) so each competitor keeps one search seed running (Boaz: "freeze those you suggested")

2026-10-27 ~18:30Z — Three queued test windows (GEPA-full s2, MIPROv2 s1, WikiSkill s0) resubmitted; speed-ups agreed with Boaz

2026-10-28 ~08:40Z — Resumed 3 of the 7 frozen searches (Boaz: "lets do that. resume the three closest to done first")

2026-10-28 ~23:25Z — State after the resume round (Boaz: "resume the three closest to done")

2026-10-29 ~16:25Z — Complete three seeds per competitor (Boaz: "unsuspend the suspended ones ... complete the runs so each competitor has three seeds")

2026-11-01 ~06:10Z — Catch-up after a ~36 h gap (the Claude session stopped on a model budget limit; LSF jobs kept running)