Parsec · reports/v4/v4_t_e1/platform-004-events-then-config.md source
platform-004-events-then-config
task: platform-004-events-then-config
category: platform
tranche: regression
services: platform, github
status: optimized
run: .capevolve/v4_t2_e1_platform-004-events-then-config/run_20260920_083427 (n_runs: 1)
Read no cell without the n and split beside it.
| measurement | split | n | reward |
|---|---|---|---|
| JB baseline | test | 1 | 1.000 |
| our baseline (v4_t1_e1) | test | 3 | 0.733 |
| seed (val, v4_t2_e1) | val | 5 | 0.787 |
| cand_0001 (val, v4_t2_e1) | val | 5 | 0.400 |
| cand_0002 (val, v4_t2_e1) <- best (val) | val | 5 | 1.000 |
| final (test, v4_t2_e1) | test | 5 | 1.000 |
delta vs JB: 0.000 · delta vs our baseline: 0.267
T2 cost/time: $15.88, 517,799 tokens, 2.05h (eval $1.22/357,549tok · optimizer $14.67/160,250tok) — see ../../../results/v4/v4_t_e1/cost_time/
Optimizer run material (not committed here -- parsec-intake_v4 worktree, gitignored): .capevolve/v4_t2_e1_platform-004-events-then-config/run_20260920_083427/report.md, .capevolve/v4_t2_e1_platform-004-events-then-config/run_20260920_083427/JOURNAL.md
What this task is about
A cluster-provision job failed. The task that actually failed, and the role it belongs to, only show up in the job's events, not in the log alone, and a config file supplies how many retries the wait was allowed. The seed also includes a second, successful host, so naming the wrong host is a distinguishable mistake.
What the optimizer tried
Two iterations, both resolving one rule conflict: four different rules told the agent not to make a second query_aap2 call after get_job_log (a redundant-call ban, a round budget, two re-fetch echoes in shared_context.md), while only one narrow rule said to make it — and that one lived in a GUID-discovery flow this task never enters, since a separate tip routes a direct job ID straight to get_job_log. cand_0001 rewrote aap2_agent.md, shared_context.md, and orchestrator.md to make all of those rules agree (redundant now means same action and same arguments; a new Critical Rule 5 requires get_job_events(failed_only=true) after a failed job's log, before any GitHub fetch) and also fixed a Role field the report template required but no log in this task's trials could source. cand_0001 was rejected on paper (val 0.400), but JOURNAL.md's post-mortem — reading all 5 trials' raw records rather than the aggregate — found this was a LiteLLM gateway outage: 2 of 5 trials scored 1.0 with the byte-identical prompt, and the other 3 never received a model response at all (TLS handshake timeouts and a 600s request timeout, before any output was produced). cand_0002 re-delivered the same behavioral contract in about a third of the diff size (no changes to orchestrator.md) and was accepted at val 1.0.
Why the winning candidate won
JOURNAL.md's reward-detail.json reading shows all 5 baseline trials differ at exactly one decision point — whether a second query_aap2 call happens after get_job_log — and that one skipped call costs both the tool_calls and answer components simultaneously (0.267 of reward). cand_0002's conflict-resolved rules made that call fire consistently, moving val 0.787 → 1.0 (Δ+0.213) and fixing the task per the RESULT line. On the held-out test split, report.md records the baseline seed skills at 0.713 ± 0.008 versus the optimized skills at 1.0 ± 0.0, a test-side improvement of +0.287; the auto block's delta vs JB of 0.000 reflects that the single-trial JB baseline measurement already sat at reward 1.0, not that the optimizer made no difference against the team's own earlier (n=3) baseline of 0.733.
Caveats
n=5 val trials is a small sample, and this was single-task tuning (see results/v4/v4_t_e1/summary.md's "Coverage" section). JOURNAL.md's own account is a caveat on interpreting any single rejected candidate at face value: cand_0001's val=0.400 looked like a regression but was actually 2 clean 1.0 trials plus 3 infrastructure failures (a gateway outage) that the eval layer scored as 0.0 — a distinction JOURNAL.md says the framework currently cannot make automatically (filed as a framework escalation).
What changed (seed → best)
2 of 8 skill files changed. Full unified diffs, artifacts/v4/seed/ → artifacts/v4/v4_t_e1/platform-004-events-then-config/best/:
aap2_agent.md (+51/−10)
--- seed/aap2_agent.md
+++ platform-004-events-then-config/best/aap2_agent.md
@@ -21,9 +21,33 @@
failure and write the report. More fetching without analysis is worse than a
report with some gaps.
-4. **Don't re-fetch job data.** If a prior `query_aap2` call already returned job
- metadata or events, extract what you need (steps, playbook events, errors) from
- the existing result. Do NOT make a redundant second call to the same job.
+4. **Reuse values you already hold — but a different `action` is not a re-fetch.**
+ A `query_aap2` call is redundant only when it repeats the **same `action` with the
+ same arguments**; read that from the existing result instead. Different actions on
+ one job are complementary: `get_job_log` returns the log text, `get_job_events`
+ returns the structured failed-task rows (`role`, `play`, `task`, `host`,
+ `error_msg`). Neither response contains the other's data, so "I already queried
+ this job" is never a reason to skip a different action on it.
+
+5. **A failed job needs the log AND the failed events.** When a job's status is
+ failed or error, call `get_job_events` with `failed_only=true` after `get_job_log`
+ and before any GitHub fetch — including when the user hands you the job ID and you
+ skip the GUID lookup. Pass `failed_only` as the boolean `true`, not the string
+ `"true"` and not `1`. The log gives you the failure *text*; only the events rows
+ carry the **`role`** that owns the failed task. These two calls are the minimum
+ evidence set, so Rule 3's budget does not apply to them — it governs exploratory
+ fetching. Do the events call even when the log looks self-explanatory: the log
+ telling you *what* failed is not the log telling you *which role* it was in.
+
+ Worked example — user asks "why did job 11111 on the `west` controller fail?":
+ ```
+ query_aap2(action="get_job_log", controller="west", job_id="11111")
+ query_aap2(action="get_job_events", controller="west", job_id="11111", failed_only=true)
+ fetch_github_file(owner=…, repo=…, path=…) # config trace, using the job's Project URL
+ ```
+ Two `query_aap2` calls, different actions, same job — correct, not a re-fetch. The
+ second one is what lets you name the role; without it the report has a hole you will
+ be tempted to fill by guessing.
## Available Tools
@@ -58,7 +82,8 @@
3. Read `tower_jobs` from the AnarchySubject — contains controller hostname and job ID
4. Call `query_aap2` with `get_job_log` using `towerHost` as controller and `deployerJob` as job_id.
**Always use `get_job_log` instead of `get_job`.**
-5. If the job failed, also call `get_job_events` + `failed_only=true`
+5. If the job failed, also call `get_job_events` + `failed_only=true` (Critical Rule 5)
+ — do it here, before Step 6; the config trace cannot supply the failed task's `role`
6. Continue to trace the config hierarchy via the "Investigate AAP2 Job Failures" workflow Steps 2+
**If the AnarchySubject is gone**, use `query_aap2(action="find_jobs", template_name="<guid>")`
@@ -93,7 +118,9 @@
ask the user to double-check the number before sweeping all controllers. If you do
sweep, check all remaining controllers in a single batch — don't try them one at a time.
- **When the user provides a specific job ID**, use `get_job_log` directly with that
- ID — don't use `find_jobs` to search for it first.
+ ID — don't use `find_jobs` to search for it first. This shortcut skips the
+ GUID/Babylon *discovery* steps only; if the job failed, still make the
+ `get_job_events` call from Critical Rule 5.
### Job Not Found in Database
@@ -234,8 +261,13 @@
| Project Pattern | Version | GitHub Owner | GitHub Repo |
|----------------|---------|--------------|-------------|
-| `https://github.com/redhat-cop/agnosticd.git` | v1 | `redhat-cop` | `agnosticd` |
-| `https://github.com/rhpds/agnosticd-v2.git` | v2 | `rhpds` | `agnosticd-v2` |
+| `.../agnosticd.git` (legacy) | v1 | `redhat-cop` | `agnosticd` |
+| `.../agnosticd-v2.git` | v2 | `agnosticd` | `agnosticd-v2` |
+
+**Parse `owner`/`repo` out of the URL you were actually given** — the job's Project
+URL, a tool result, or the user's question — as `https://github.com/{owner}/{repo}.git`,
+and use exactly that. Fall back to this table only when you have no URL. An owner you
+guessed either 404s or silently returns a different repo's file.
When fetching agnosticd files, use the `ref` parameter to get the correct code version:
1. **If the job has a Revision SHA** — use it as `ref` to get the exact commit that ran.
@@ -276,7 +308,10 @@
**Key sections to examine in the log:**
1. **PLAY RECAP** — Summary of hosts and status
2. **fatal** or **FAILED** tasks — Actual error messages
-3. **TASK [role_name : task_name]** — Identify which role/task failed
+3. **TASK [...]** — the failed task's name. The `role :` prefix is frequently absent
+ from a rendered log, so a bare `TASK [some name]` does **not** mean the task has no
+ role — that is a rendering detail, not a fact about the playbook. Take the role
+ from the `get_job_events` rows (Critical Rule 5), never from the log's silence.
4. **Pod status details** — container states, waiting reasons, restart counts, exit codes
5. **Timing** — how long did the failing operation take? Short = auth/config error. Long = timeout.
@@ -401,8 +436,14 @@
- **Namespace:** `{namespace}` (if CNV)
**Failure Analysis:**
-- **Failed Task:** `{role} : {task_name}`
-- **Host:** `{host}`
+- **Failed Task:** `{task_name}`
+- **Role:** `{role}` — copy the `role` field from the `get_job_events` row for the
+ failed task; that is its only source. If the events rows carry no role, write
+ `not reported in job events`. An `env_type`/config directory, the catalog item, the
+ repo path, and the `PLAY [...]` name are **not** the role — not even when the user's
+ own question handed you that string.
+- **Host:** `{host}` — the host the PLAY RECAP shows with `failed=1`, not the first
+ host mentioned in the log
- **Error:** the actual error (not "pod failed" — the underlying cause)
For pod/container failures, include:
shared_context.md (+10/−4)
--- seed/shared_context.md
+++ platform-004-events-then-config/best/shared_context.md
@@ -184,7 +184,12 @@
loosen JOINs, widen date range) before adding complexity back.
- **Error results**: All tools return `{"error": "..."}` on failure. Report the error
and suggest alternatives.
-- **NEVER call the same tool with the same parameters twice in a conversation.**
+- **Never repeat an identical call — same tool, same parameters — in a conversation.**
+ If a prior result already holds the value you need, read it from that result instead
+ of calling again. This bans repeating a *call*, not touching a *resource* twice: a
+ different tool, or the same tool with different parameters (a different `action`, a
+ different filter), returns data the first call did not, so it is not a re-fetch and
+ is not redundant.
- **CRITICAL: Consult the "Reporting Database Reference" section before writing
SQL.** Do not guess column names — use ONLY columns listed in the schema
reference. If unsure, call `db_describe_table` to check. Common mistakes:
@@ -195,9 +200,10 @@
- `tower_job_log` column names are snake_case (`deployer_job`, not `deployerJob`)
- `provision_cost` is partitioned — always include a `month_ts` filter to avoid full partition scans
- When joining tables with shared column names (e.g. `category`), always use table aliases to avoid ambiguous column errors
-- **Don't re-fetch data already in context.** If a prior tool call returned data
- (e.g., job details, provision records), extract what you need from the existing
- result before making another call.
+- **Reuse values already in context.** If a prior tool call returned the value you
+ need (job details, provision records), read it from that result rather than calling
+ again — but see the identical-call rule above: a *different* action or filter on the
+ same resource returns new data and is not a re-fetch.
## Grounding — Use Tool Results, Never Hallucinate