Parsec · reports/v1/aap2-0052.md source
traces_parsec-aap2-0052 — the assertion list is empty — half this score measures nothing
Status: BASELINE_ONLY · Experiment: v1 (trace-extracted from a production session) · Category: aap2 / multi-tool
Prompt given to the agent:
can you check the user count also, some of this would be 1 users, while the failed one was 30 or 40 i think
Contract (tests/expected.json): subset match over 9 demanded tool calls · 0 assertions — an empty list scores a vacuous assertions: 1.0 · weights trajectory 0.5 / assertions 0.5 · gate requires status == ok and a non-empty answer.
Trajectory reachability at baseline: the contract demands db_describe_table and query_provisions_db, none of which any simulator running on 2026-08-12 exposed. Under a subset match one uncallable op fails the whole containment, so trajectory: 1.0 was unreachable by construction on this task and its reward was capped at 0.5 whatever the skill said.
Flags: capped-at-0.5-by-50/50-weighting · no-assertions-scores-vacuous-1.0 · trajectory-unreachable-tool-gap
| measurement | split | n | reward | |
|---|---|---|---|---|
| baseline sweep | all |
1 | 0.500 | single trial, no optimizer — v1 reference sweep |
The optimizer never ran on this task. It was not in the pilot's val split, so seed, best, delta and final_test are null by construction — not zero. Only 2 of 30 v1 tasks entered cap-evolve; the other rows are a one-trial reference sweep.
Every number above is generated from results/results.json. Read no cell without the n beside it: this branch's central finding is that the same unchanged skill measures differently at different trial counts.
Material:
- results/v1/baseline_all_1786543049.json — the n=1 sweep row this task's baseline comes from, with its completion / trajectory / assertions components
- results/v1/tasks.json — this task's full contract: the demanded tool-call sequence and every answer_contains string
- results/v1/per_task_scores.json — per-trial reward vectors for this task in every run of this experiment, with errored trials marked null
- artifacts/v1/seed/SKILL.md and the two rejected candidates — diff them to see what was tried and thrown away
- ui/heatmap.html — this row in context of all 40 tasks
What changed
Nothing was changed on this task, because the optimizer never saw it. The N=30 pilot's val split contained 2 tasks (traces_parsec-aap2-0047 and -0048); this is one of the other 28. It has exactly one measurement in the whole experiment: a single trial in the reference sweep.
What worked
N/A — nothing was attempted.
What didn't
The contract is defective. tests/expected.json lists zero assertions, and an empty list scores assertions: 1.0 vacuously. Combined with trajectory: 0.000 over 9 demanded calls, that produces a baseline of exactly 0.500 in which the agent demonstrably earned nothing: it did not reproduce the tool sequence, and there was no answer requirement for it to satisfy or miss.
And it could not have reproduced the sequence in any case. This task's contract demands 9 tool calls in a subset match, including db_describe_table and query_provisions_db — none of which any simulator running on 2026-08-12 exposed. One uncallable op fails the whole ordered containment, so trajectory 1.0 was unreachable here by construction, and half of this task's reward was never in play. Across v1 trajectory scored 1.0 on 5/5 tasks whose demanded sequence was fully callable and 0/25 of those that were not.
What we can (or can't) learn
Nothing about the skill, and this row should not be in a mean. It is the clearest single instance of the v1 benchmark's real problem: the contracts were extracted from traces mechanically and not reviewed, so a task can score 0.500 without measuring anything. Sum this task into a 30-task average and it drags the headline up. It is the only such task in v1 (traces_parsec-aap2-0052), which is why it is flagged on the row rather than dropped — dropping it silently would be the same class of error in the other direction.