Evaluate Matrix

/evaluate:matrix

The executability gate. Where /evaluate:legibility asks "can a fresh agent comprehend this skill," this skill asks "can a weak model actually do it" — run the skill's evals on haiku (and opus/sonnet) with real tool execution, grade the produced artifact, and surface the per-skill verdict executable_on_haiku. A skill that opus passes and haiku fails leans on reasoning the cheap model lacks.

This builds the orchestration the cross-model design (docs/cross-model-evaluation.md) calls a follow-up. It reuses, without duplicating: prepare_run.sh, grade_deterministic.py (zero-token first pass), the eval-grader agent (deferred fuzzy checks only), model-matrix.json, and render_matrix_report.py.

When to Use This Skill

Context

• Available plugins: !find . -maxdepth 2 -type d -name '*-plugin' -not -name '.claude-plugin'

Parameters

Aliases float across model generations, so --models accepts either an alias or a full id; pass the full id when the run must be reproducible (the model field on the dispatch accepts one) and record the id that actually ran — never the alias — in model-matrix.json metadata.models[].model_id (.claude/rules/skill-evaluation.md).

Execution

Execute this cross-model matrix:

Step 1: Resolve skill and evals

Read <plugin-name>/skills/<skill-name>/evals.json. If absent, report that the matrix needs eval cases (point at /evaluate:skill --create-evals) and stop. Validate it against the evals.json schema (references/schemas.md).

Step 2: Run the matrix (serialized)

Loop over (model ∈ --models) × eval × config ∈ {with_skill, cached_baseline} — skip cached_baseline if --with-skill-only, and reuse a baseline cached for the same model-version rather than re-running it. For each combination:

1. Scaffold the run dir:

   bash ${CLAUDE_PLUGIN_ROOT}/scripts/prepare_run.sh \
     --skill-dir <plugin-name>/skills/<skill-name> \
     --eval-id <eval-id> --run <N>
   

   If the eval carries a fixture block, apply it for an honest execution context — without one a context-needing skill fails on haiku purely for lack of fixtures, a false negative that poisons this gate:

   bash ${CLAUDE_PLUGIN_ROOT}/scripts/apply_fixture.sh \
     --fixture '<eval.fixture JSON>' --repo-root "$(pwd)"
   

   Parse WORKDIR=; the subagent operates there. Tear it down after the transcript is copied out (--teardown "$WORKDIR").

2. Dispatch one Task subagent with the model field set to the loop model (full Bash/Edit — it does real tool execution, not just reading):

   Task subagent_type: general-purpose
   model: <loop model alias>
   prompt: <eval prompt; with_skill runs also receive the SKILL.md content>
   

   Serialize the dispatches — one at a time, never a parallel batch. [1m] models hit cascading rate limits with concurrent subagents (.claude/rules/skill-fork-context.md). Spawns run in the background by default (Claude Code ≥ 2.1.232), so dispatch order alone no longer serializes them: pass run_in_background: false on each dispatch, or wait for that agent's result to arrive before issuing the next one — serialization is the wait, not the dispatch order.

   Keep the subagent off …/skills/… paths. Paste the SKILL.md body into the prompt instead of naming its path, and have the subagent return its artifact in the reply. Path-scoped rules load whole into any agent that touches a matching file, and a repo can scope many of them to **/skills/** (the claude-plugins marketplace scopes over a dozen); there a haiku subagent handed a skill path failed with HTTP 400 Prompt is too long before running anything (#2667). A 400 on the haiku arm is the context surface, not the skill — prepare_run.sh stages run dirs outside skills/ for the same reason, and check-context-engineering.py caps each rule's size so the surface cannot regrow unseen.

3. Write the subagent's produced artifact to $RUN_DIR/transcript.md.

Headless branch (--harness headless). Replace items 2-3 with one serial Bash call per combination; no subagent is dispatched and no SKILL.md is pasted. Write the eval prompt to $RUN_DIR/prompt.txt; use the fixture's WORKDIR (or mktemp -d outside the repo):

bash ${CLAUDE_PLUGIN_ROOT}/scripts/rollout_headless.sh \
  --run-dir "$RUN_DIR" --workdir "$WORKDIR" --prompt-file "$RUN_DIR/prompt.txt" \
  --model <loop model alias or full id> [--effort <e>] --max-budget-usd 0.25 \
  [--plugin-dir "$(pwd)/<plugin-name>"]      # with_skill only; omit for baseline

It writes transcript.md, trace.json and workspace/ into $RUN_DIR. Omit --effort on haiku. STATUS=ERROR is an ERROR cell, not a 0% pass.

Step 3: Grade — deterministic first, judge only on deferral

For each run, grade the produced output:

1. Run the zero-token deterministic grader first:

   python3 ${CLAUDE_PLUGIN_ROOT}/scripts/grade_deterministic.py \
     --evals <evals.json> --eval-id <eval-id> --output $RUN_DIR/transcript.md --json \
     [--trace $RUN_DIR/trace.json] [--workspace $RUN_DIR/workspace --allow-exec]
   

   Bracketed flags: headless runs only. On a subagent run trace/workspace checks are harness_deferred — excluded from the pass rate, never judged.
2. Only if it reports JUDGE_PENDING > 0, dispatch the eval-grader agent for the deferred fuzzy expectations:

   Task subagent_type: evaluate-plugin:eval-grader
   Prompt: Grade ONLY the deferred (judge) expectations for <eval-id> ...
   

   Most expectations grade deterministically — the judge fires on a fraction.

Step 4: Aggregate to model-matrix.json

Combine per-run pass rates into <skill-dir>/eval-results/model-matrix.json following the schema. Compute, per model alias, the mean with_skill and baseline, the delta, and prev_delta from any stored prior run. Record metadata.models[].harness; on a headless run model_id is trace.json's.

Step 5: Render the report

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/render_matrix_report.py \
  <skill-dir>/eval-results/model-matrix.json

The renderer emits the delta table, per-model verdicts, the portability flag (opus−haiku spread ≥20 pts), and the executability flag (executable_on_haiku=false when haiku's absolute with-skill rate is below the 0.5 floor while opus clears it). It also prints a mixed-harness warning when the file mixes subagent and headless rollouts (not comparable). Print the report and call out whether the executability flag fired.

Minimal Provable Increment

Run the matrix for git-plugin/git-commit only (it already has typed-check evals), --models opus,haiku --with-skill-only: grade deterministically, render, and confirm the executability callout lights up or stays dark correctly. This exercises every reused piece end-to-end before scaling to the golden set.

Agentic Optimizations

Quick Reference

Related

• /evaluate:legibility — the comprehension gate (cold-read, no execution)
• /evaluate:skill — single-model effectiveness with a baseline
• docs/cross-model-evaluation.md — the design this implements
• .claude/rules/skill-evaluation.md — tiered methodology, golden set, cadence
• .claude/rules/skill-fork-context.md — why subagent dispatch is serialized