Evaluate Plugin Batch

/evaluate:plugin-batch

Batch evaluate all skills in a plugin. Runs /evaluate:skill for each skill, then produces a plugin-level quality report.

When to Use This Skill

Context

• Available plugins: !find . -maxdepth 2 -type d -name '*-plugin' -not -name '.claude-plugin'

Parameters

Parse these from $ARGUMENTS:

Workflow harness (template)

workflows/evaluate-plugin-batch.workflow.js ships beside this skill. It is a TEMPLATE to adapt, not a script to run verbatim. Read it, then rewrite it for the work in front of you.

Adapt freely: the inventory and aggregation agent prompts, the effort tiers, the DEFAULT_WAVE starting width, the CELL_CAP handed to each child, the skipped[] reason strings, and the shape of the rows your project's report actually needs.

Preserve across any adaptation: (a) the loop bound comes from inspect_eval.sh --plugin-dir <plugin>'s === SKILLS === / === EVALS === listings, never from a prose "for each skill" — and eval-readiness stays the pure s.hasEvals || createMissingEvals filter over a file that exists or does not, so no agent classifies it; (b) REPORT_SCHEMA's closed complete | partial-sweep | aggregate-failed status enum and its one-value denominatorSource enum, which together make it structurally impossible to report a plugin pass rate without stating that the denominator came from the script; (c) the Aggregate stage is a barrier — aggregate_benchmark.sh walks the filesystem for every skill's eval-results/benchmark.json, so its denominators are only correct once the last cell has finished writing, and no single cell can see the plugin-level numbers. The harness arg is forwarded verbatim to every workflow('evaluate-skill', …) child — never mixed within one batch, because subagent and headless pass rates are not comparable.

Three consequences of (a)–(c) that are also non-negotiable:

• CAP (default 25) ABORTS; it never truncates. A silently-shortened sweep publishes a pass rate over a denominator nobody chose — which is precisely the number this skill exists to produce correctly. The abort states the count and the cap.
• --parallel N is the caller's and is never superseded. The wave width is derived from args.parallel; DEFAULT_WAVE applies only when the caller supplied nothing, and a non-integer value falls back loudly via log(). The platform separately caps concurrency at min(16, CPUs-2), and a nested child shares this run's cap and agent counter.
• The workflow's own count is only a cross-check. crossCheck.agrees compares the cells this run dispatched against the benchmarks the script found on disk. A disagreement emits partial-sweep — that disagreement is the anti-laziness signal, not a rounding error, and must never be smoothed into complete.

This harness runs near-empty until a golden set of evals.json files exists. Per .claude/rules/skill-evaluation.md eval coverage is deliberately scoped to ~15–25 canary skills, so most plugins today have zero or one eval-ready skill and the run aborts with reason: 'below-floor'. That is the designed outcome, not a bug — an empty sweep means the evals have not been written yet, and the fix is to author them (or pass --create-missing-evals), never to widen the harness.

Agent budget: 2 + skills x evaluate-skill budget — the inventory and aggregate agents, plus one evaluate-skill child per included skill (at most 25), each spending that skill's own budget. The scale guard asks before every run: a workflow() child's agents are never counted.

Skip the harness when: the plugin has fewer than 2 eval-ready skills — the modal case today, and FLOOR = 2 is a hard bound that aborts there rather than a tunable knob, because a single skill is exactly what /evaluate:skill already does without an inventory agent, a nested workflow and an aggregate agent on top. The steps below remain the authoritative description of what each stage must produce; the harness only fixes how the work is split.

context: fork stays off for this skill. The fan-out here is a caller-chosen --parallel N width, so the [1m] concurrent-subagent cascade hazard in .claude/rules/skill-fork-context.md applies; scripts/plugin-compliance-check.sh keeps this skill out of its context: fork pin list for that reason, and the harness does not change it.

Execution

Step 1: Discover skills

Find all skills in the plugin:

<plugin-name>/skills/*/SKILL.md

List them and count the total.

Step 2: Filter and prepare

For each skill, check if evals.json exists:

• Has evals: include in evaluation
• No evals + --create-missing-evals: include, will create evals during evaluation
• No evals, no flag: skip with a note

Report the breakdown:

Found N skills in <plugin-name>:
  - M with eval cases
  - K without eval cases (skipped | will create)

Step 3: Run evaluations

For each included skill, invoke /evaluate:skill via the SlashCommand tool:

SlashCommand: /evaluate:skill <plugin-name>/<skill-name> [--create-evals] [--harness <h>]

Forward --harness unchanged to every skill when it was given; omit it otherwise (the default is subagent).

If --parallel N is set and N > 1, batch evaluations into groups of N. Otherwise, run sequentially.

Report progress from the results themselves: after each /evaluate:skill returns, state which skill finished and its pass rate from benchmark.json. (Todo/task tools are not available on Opus 4.8+/Sonnet 5/Fable-generation models unless CLAUDE_CODE_ENABLE_TODO_TOOLS=1 is set; do not depend on them — see .claude/rules/agentic-permissions.md § Task-tool availability.)

Step 4: Aggregate plugin report

After all skill evaluations complete, read each skill's benchmark.json and aggregate:

bash evaluate-plugin/scripts/aggregate_benchmark.sh <plugin-name>

Write aggregated results to <plugin-name>/eval-results/plugin-benchmark.json.

Step 5: Report

Print a plugin-level summary table:

## Plugin Evaluation: <plugin-name>

| Skill | Evals | Pass Rate | Status |
|-------|-------|-----------|--------|
| skill-a | 4 | 100% | PASS |
| skill-b | 3 | 67% | PARTIAL |
| skill-c | 5 | 80% | PASS |

**Overall**: 82% pass rate across N eval cases

Rank skills by pass rate. Flag any below 50% as needing attention.

Agentic Optimizations

Quick Reference