hypothesis: Property-Based Testing for LLM Outputs
Property-based testing with hypothesis is a natural fit for non-deterministic LLM output: instead of asserting a specific answer, you assert invariants that must hold across all valid outputs regardless of phrasing.
Read MoreToolCallCheck: Mock Your Tool Server and Assert on Trajectories
Mock MCP server that runs fully offline, records every tool call, and lets you assert on tool selection, argument correctness, call order, and structural invariants — no LLM call needed.
Read MoreDeepEval: CI-Gated Typed Metrics for LLM Outputs
DeepEval is an open-source Python framework with 50+ ready-made metrics for LLM outputs: faithfulness, hallucination, toxicity, bias, G-Eval rubrics, agent and conversation metrics, plus deterministic checks. You build test cases, score them with metrics, and fail CI when a score drops below its threshold. This is the complete reference.
Read MoreBraintrust: Production Eval History, Dashboards and CI Gates
Braintrust bridges the gap between local eval scripts and production monitoring — dataset management, commit-linked eval history, regression dashboards, and human annotation queues.
Read MorePromptfoo: Model Comparison and Red-Teaming for LLM Outputs
Promptfoo is the go-to tool for comparing prompts and models side by side, running adversarial tests, and catching regressions before they hit production. YAML-driven, no Python required.
Read Morepytest for LLM Evaluation: Parametrized Tests, Fixtures, and CI Gates
pytest is the runner that ties the eval pyramid together — parametrized golden datasets, fixtures for model connections, invariant checks, and sampling loops. No extra framework needed.
Read MoreTesting LLM Outputs: Evals for Models, Agents, and Skills
LLM outputs are non-deterministic, which breaks the assumptions most of our testing tooling is built on. This document covers how to actually test model outputs, agent trajectories, and Claude Code-style skills — the layers of evaluation, the current 2026 tool landscape, and a working CI harness you can copy.
Read More