hypothesis: Property-Based Testing for LLM Outputs
Property-based testing with hypothesis is a natural fit for non-deterministic LLM output: instead of asserting a specific answer, you assert invariants that must hold across all valid outputs regardless of phrasing.
Read MoreToolCallCheck: Mock Your Tool Server and Assert on Trajectories
Mock MCP server that runs fully offline, records every tool call, and lets you assert on tool selection, argument correctness, call order, and structural invariants — no LLM call needed.
Read MoreDeepEval: CI-Gated Typed Metrics for LLM Outputs
DeepEval is an open-source Python framework with 50+ ready-made metrics for LLM outputs: faithfulness, hallucination, toxicity, bias, G-Eval rubrics, agent and conversation metrics, plus deterministic checks. You build test cases, score them with metrics, and fail CI when a score drops below its threshold. This is the complete reference.
Read Morepytest for LLM Evaluation: Parametrized Tests, Fixtures, and CI Gates
pytest is the runner that ties the eval pyramid together — parametrized golden datasets, fixtures for model connections, invariant checks, and sampling loops. No extra framework needed.
Read More