How to Test an Agent Skill Before You Rely on It

Short answer: give the skill a few realistic tasks in a fresh session, run each one with the skill and again without it, and check both outputs against what you expected. Then test separately that the skill triggers on the requests it should and stays quiet on near-misses, on every model you use, and repeat the whole set whenever the skill or the model changes.

That is the method in agentskills.io's evaluation guide, Anthropic's skill best practices and the Claude Code docs, all checked on Oct 30, 2026. This guide walks through it in six steps.

Why "it worked once" isn't a test

The Claude Code docs put it plainly: seeing a skill trigger tells you Claude found it, not that it did what you intended. There are two separate questions. Does the agent load the skill when it should? And when it does, is the output better than what the agent would have produced without it? A skill can fail either one, and a skill can make results worse.

Step 1: Write a few realistic test cases

agentskills.io defines a test case as three things: a prompt, a plain-language description of the expected output, and any input files. It suggests keeping them in evals/evals.json inside the skill folder, for example:

{"id": 1, "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Can you find the top 3 months by revenue and make a bar chart?", "expected_output": "A bar chart image showing the top 3 months by revenue, with labeled axes and values.", "files": ["evals/files/sales_2025.csv"]}

How many: agentskills.io says start with 2-3 and expand later. Anthropic's best-practices checklist asks for at least three evaluations, and its enterprise guide asks skill authors for 3-5 representative queries per skill.

What makes them useful: vary the phrasing from casual to precise, include at least one edge case such as malformed input, and write prompts like a real user would, with file paths and column names. "Process this data" is too vague to test anything.

If you are writing your own skill, Anthropic recommends building these evaluations before writing much of the skill: run the agent without a skill first, note where it fails, and write just enough to fix those failures.

Step 2: Run each case with and without the skill

The baseline is the core of the method. Run each prompt once with the skill and once without it, then compare. If you are testing a new version of a skill, use the previous version as the baseline instead.

Start every run in a fresh session. Leftover context, especially from writing the skill, hides gaps in its instructions. In Claude Code, turn a personal or project skill off for the baseline run by setting it to "off" in skillOverrides. For a skill that comes in a plugin, use claude plugin eval, which repeats each run with no plugin loaded.

Also note the tokens and time each run used. agentskills.io's point: a skill that improves the result but triples token use is a different trade-off from one that is better and cheaper.

Step 3: Grade the outputs with assertions

After the first run, write assertions: specific statements the output should satisfy. Good ones are checkable, such as "The output file is valid JSON", "Both axes are labeled" or "The report includes at least 3 recommendations". Weak ones are vague ("The output is good") or brittle, requiring one exact phrase.

Mark each assertion pass or fail with evidence from the output. Use a script for mechanical checks such as valid JSON or row counts; agentskills.io says scripts are more reliable than LLM judgment there. Then look at the pattern. Assertions that pass with and without the skill tell you nothing, so drop them. The ones that pass only with the skill show where it earns its place.

Finally, read the outputs yourself. Assertions only check what you thought to write down.

Step 4: Test that it triggers, and when it shouldn't

Triggering depends almost entirely on the skill's description. agentskills.io suggests about 20 test queries: 8-10 that should trigger the skill and 8-10 that shouldn't. The best negatives are near-misses that share keywords but need something else, such as an Excel formula question for a CSV analysis skill.

Model behaviour varies between runs, so run each query several times (3 is a reasonable start) and compute a trigger rate. A should-trigger query passes above a threshold, 0.5 by default, and a should-not-trigger query passes below it. If you are tuning the description, hold back about 40% of the queries to check that your changes generalise.

One catch: agents often skip skills for simple one-step requests they can handle alone, so a perfect description may still not fire on "read this PDF". Test with tasks where the skill really adds something.

To see whether it fired, check your agent's logs or tool-call history for a load of the skill's SKILL.md. agentskills.io's sample script runs claude -p with --output-format json and looks for a Skill tool call.

Step 5: Test on every model you use

A skill adds to a model; it doesn't replace it. Anthropic's advice is to test a skill with every model you plan to use it with, and its checklist says Haiku, Sonnet and Opus. The questions differ by model: does a fast, small model get enough guidance, and does a large one get over-explained to? Instructions that work for Opus may need more detail for Haiku. The same applies if you run the skill in more than one agent.

Step 6: Re-test after every update

Skills change, and so do models. Anthropic's enterprise guide says to pin production skills to specific versions, run the full evaluation suite before promoting a new version, and rerun evaluations periodically to catch drift as workflows and models change. It also says to treat every update as a new deployment with a full security review. Keep your test cases next to the skill so re-testing is one command, not a project.

Tools that automate this

skill-creator: Anthropic's skill-creator, published as a skill and available as a Claude Code plugin, runs this loop inside a conversation. Per the Claude Code docs, it benchmarks pass rate, time and tokens with and without the skill, runs a blind A/B between two versions, generates should-trigger and should-not-trigger prompts to tune the description, and opens an HTML review page.

claude plugin eval: for skills that ship in a Claude Code plugin, it runs each prompt in an isolated session with and without the plugin, scores the results with graders, and exits non-zero below a threshold, so you can gate CI on it.

The Claude Code docs note that the two tools' eval formats aren't interchangeable. Anthropic's best-practices page also says there is no built-in way to run the simple evaluation format it shows. You don't need either tool to start: a folder of prompts and a notes file is enough.

What Skill Store 33's Verified badge tells you

On Skill Store 33, a skill marked Verified is one whose own tests ran in a sandbox and passed. The store's wording is "Nothing more." That is useful: it tells you the skill does what its author's tests expect. It does not tell you that those tests cover your tasks, your model or your agent, that the skill beats having no skill, or that anyone reviewed it for security. Other states are Not yet verified, Safety scanned (a safety scan only) and Tests failing.

Almost every skill in the store is imported from public sources and is not yet verified, so for now the steps above are how you find out whether a skill works for you. A badge is a reason to try a skill first, not a reason to skip your own with-and-without run. For the safety side, see our guide to the risks.

FAQ

How many test cases does a skill need?

Start with 2-3 for output quality, as agentskills.io suggests, and grow to the 3-5 Anthropic asks of enterprise skill authors. Trigger testing needs more: about 20 queries, split between should and shouldn't.

How do I know whether a skill triggered?

Look in your agent's logs or tool-call history for a load of the skill's SKILL.md. In Claude Code it appears as a Skill tool call, which you can see in claude -p --output-format json.

Can I test a skill without writing code?

Yes. Write a few realistic prompts and what a good result looks like, run each in a fresh session with the skill on and again with it off, and compare. Scripts only matter once you repeat runs many times.

Does a Verified badge mean I don't need to test a skill?

No. It means the author's own tests passed in the store's sandbox. Your tasks, model and agent may differ, so run your own comparison before you rely on it.

Let your agent search Skill Store 33

Sources

Agent Skills, Evaluating skill output quality, https://agentskills.io/skill-creation/evaluating-skills (read Oct 30, 2026)

Agent Skills, Optimizing skill descriptions, https://agentskills.io/skill-creation/optimizing-descriptions (read Oct 30, 2026)

Anthropic, Skill authoring best practices (Test with all models you plan to use; Evaluation and iteration; Checklist), https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices (read Oct 30, 2026)

Anthropic, Skills for enterprise (Evaluating Skills before deployment; Versioning strategy), https://platform.claude.com/docs/en/agents-and-tools/agent-skills/enterprise (read Oct 30, 2026)

Claude Code docs, Extend Claude with skills (Evaluate and iterate on a skill; Troubleshooting), https://code.claude.com/docs/en/skills (read Oct 30, 2026)

Skill Store 33, verification badge wording, https://ss33.ai/skills (read Oct 30, 2026)

This article was created with the help of AI (AI-Assisted). Every config key and command was checked against the official docs in the Sources list. ReadyAI is accountable for the content.

Keep reading