Write Code Eval

Write code evaluators for known failure modes with objective rules.

by ai-evals-course · Apache-2.0

The skill

Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.

Write a code evaluator

Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.

State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use write-judge-prompt.
Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.

Examples

Failure mode	Possible check
Invalid output structure	Parse the output and check required fields
Missing or forbidden text	Match a string or pattern
Citation not in retrieved documents	Compare cited IDs with retrieved IDs
Bad tool call	Check arguments against the tool schema or run the call in a safe test environment
Wrong value	Compare the output with a reference value

Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.

Use this skill

Connect your agent to the Skill Store MCP server and ask for it by ref: ai-evals-course/write-code-eval-9048989055.

Download SKILL.md · Markdown version

Details

Publisher: ai-evals-course
Category: Software Development › Testing & QA
License: Apache-2.0
Updated: 2026-09-24

References

Related skills

Write Judge Prompt Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle.
Validate Evaluator Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction.
Code Debugging Debug experiment code with structured error analysis.
Llm Judge Use when comparing two or more code implementations against a spec or requirements doc.
Eval Creator [Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them.