AI Evals Course publishes skills that help AI agents evaluate LLM systems, from auditing an existing eval pipeline for missing error analysis and unvalidated judges to designing LLM-as-Judge prompts and code-based evaluators for specific failure modes. Their catalogue also covers RAG retrieval and faithfulness testing, synthetic test data generation, and building browser-based annotation interfaces for reviewing traces and collecting human labels. The skills stand out for their clear routing, with most stating when to use them and when to hand the task to a more specific skill.
Evals Skills
9 skills · 1,715 stars · Apache-2.0
Build Review Interface Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces…
Error Discovery Run error analysis on a dataset.
Eval Audit Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vani…
Evals Start Entry point for evals.
Evaluate Rag Guides evaluation of RAG pipeline retrieval and generation quality.
Generate Synthetic Data Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple gene…
Validate Evaluator Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction.
Write Code Eval Write code evaluators for known failure modes with objective rules.
Write Judge Prompt Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle.