Blog Posts

Home /
Blog Posts

hypothesis: Property-Based Testing for LLM Outputs

Property-based testing with hypothesis is a natural fit for non-deterministic LLM output: instead of asserting a specific answer, you assert invariants that must hold across all valid outputs regardless of phrasing.

ToolCallCheck: Mock Your Tool Server and Assert on Trajectories

Mock MCP server that runs fully offline, records every tool call, and lets you assert on tool selection, argument correctness, call order, and structural invariants — no LLM call needed.

DeepEval: CI-Gated Typed Metrics for LLM Outputs

DeepEval is an open-source Python framework with 50+ ready-made metrics for LLM outputs: faithfulness, hallucination, toxicity, bias, G-Eval rubrics, agent and conversation metrics, plus deterministic checks. You build test cases, score them with metrics, and fail CI when a score drops below its threshold. This is the complete reference.

Braintrust: Production Eval History, Dashboards and CI Gates

Braintrust bridges the gap between local eval scripts and production monitoring — dataset management, commit-linked eval history, regression dashboards, and human annotation queues.

Promptfoo: Model Comparison and Red-Teaming for LLM Outputs

Promptfoo is the go-to tool for comparing prompts and models side by side, running adversarial tests, and catching regressions before they hit production. YAML-driven, no Python required.

pytest for LLM Evaluation: Parametrized Tests, Fixtures, and CI Gates

pytest is the runner that ties the eval pyramid together — parametrized golden datasets, fixtures for model connections, invariant checks, and sampling loops. No extra framework needed.

Spec-Driven Development With AI

Spec-Driven Development (SDD) is a 2026 software engineering workflow where structured specifications become the source of truth and AI agents derive implementation, tests and documentation from them.

Amazon MQ System Design — ActiveMQ and RabbitMQ on AWS

Amazon MQ design guide for ActiveMQ and RabbitMQ.

AWS Aurora Serverless

Deep dive into Aurora Serverless v1 and v2 architecture, auto-scaling, Data API, pricing, best practices, limitations and migration strategies.

Database Change Data Capture (CDC)

Deep dive into Change Data Capture — log-based, trigger-based and timestamp-based patterns, Debezium, AWS DMS, DynamoDB Streams, the outbox pattern, dual-write problem and exactly-once semantics.

Blog Posts

Tags