High Availability Patterns Reference

This reference collects pattern tables from the main High Availability post and adds pattern families not covered there. Each section compares strategies or patterns across dimensions relevant to availability. Sources are cited per section.

Consistency Models

Interservice consistency

When service A calls service B synchronously, A’s availability is coupled to B’s — if B is slow or down, A degrades too. This is the CAP trade-off at the service boundary: synchronous propagation gives immediate consistency at the cost of availability. The patterns below decouple services by accepting eventual consistency.

The same consistency models that govern database replication apply directly to interservice communication — any time state is replicated or propagated across nodes (DB replicas, caches, or microservices), the same trade-offs and vocabulary apply.

Sources: Microservices Patterns by Chris Richardson (Ch. 4, 7); Building Event-Driven Microservices by Adam Bellemare (Ch. 3, 6)

How the standard consistency models map to a two-service scenario:

• Strong consistency — every reader sees the same value immediately after a write, regardless of which node or service they query. The synchronous call gave this: A waited for B to confirm before the operation was considered done, so anyone reading from B afterward saw the update. Availability suffered because enforcing this across a slow or unreachable node means blocking or failing.
• Eventual consistency — no guarantee about when B converges, only that it will given no more writes. Async messaging gives this: A returns immediately, B catches up whenever the message is processed. Availability goes up, but there is a window where A and B disagree.
• Read-your-writes consistency — the writer itself sees its own write on subsequent reads, even if other readers might not yet. This matters when the same client that triggered a change in A immediately queries B and expects to see it (e.g. a user updates a profile in service A, then the UI fetches it from service B and the update appears missing). Solved by routing that specific read back to A’s primary or a cache A writes through, rather than waiting on B’s eventual sync.
• Causal consistency — causally related operations (B’s event depends on A’s event) are seen by everyone in that order, even if unrelated operations can be reordered. If A emits “OrderCreated” then “OrderCancelled,” the consumer must see Created before Cancelled. Achieved via partition keys (Kafka partitioning by order ID guarantees ordering within that key) or version vectors.

Patterns to consistency mapping:

The practical question is not which pattern is correct in the abstract — it is which guarantee the business requirement actually needs. If nothing bad happens when B lags behind A by a few hundred milliseconds to a few seconds, eventual consistency with async messaging is the right call. If the caller needs to see its own write immediately, that is a read-your-writes requirement on a specific path, not a reason to make the whole system strongly consistent again.

Operational necessities for any async pattern:

• Idempotency — consumers deduplicate by event ID because brokers offer at-least-once, not exactly-once
• Dead-letter queues — events B cannot process must not vanish silently; DLQs capture them for inspection and replay
• Lag monitoring — track the staleness window so “eventual” is measurable in practice
• outbox pattern + SQS/SNS (or Kafka) + idempotent consumer. - This removes synchronous coupling without needing full saga orchestration — save that for multi-step business transactions, not a two-hop propagation.

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 5, 9)

Note: eventual vs read-after-write: With eventual consistency even the writer may not see their own write immediately — the read could land on a replica still catching up. With read-after-write consistency the writer always sees their own writes, but other clients may still read stale data for a while.

CAP Theorem

CAP Theorem: In a distributed data store, you can have at most two of Consistency (every read returns the most recent write), Availability (every request receives a non-error response) and Partition Tolerance (the system continues operating despite network failures).

Note — two caveats to keep in mind:

1. Writes during a partition: In a CP system the majority side keeps accepting writes but the minority side rejects writes (returns errors) to prevent divergence. Those writes fail until the partition heals — they are not lost, simply denied. In an AP system both sides accept writes but data may diverge until the link is restored.

2. Why CA is impossible: Partitions are inevitable — network failures will happen. A CA-classified system is only CA when the network is perfect, which is never true at scale. The real choice in production is CP or AP.

How real systems pick their trade-off:

Key insight: The “sacrificed” pillar is never truly zero. AP systems find ways to give you consistency when you need it (DynamoDB strongly consistent reads, Cassandra QUORUM). CP systems invest heavily in infrastructure to make partitions rare (Spanner’s private network, Raft lease mechanisms). No system fully ignores any pillar — they just prioritise.

No database guarantees 100% availability. The best achievable is five 9s (99.999%) — about 5 minutes of downtime per year. Even AP systems go down during a full-region outage (no replicas left) or a software/ops failure that affects all nodes simultaneously. Real-world outages happen regardless of architectural trade-off: global DNS failures, bad schema migrations, shared control-plane bugs, misconfigured firewalls.

DynamoDB-specific write behaviour (being AP): DynamoDB does not have dirty writes or dirty reads — every write is atomic and there is no uncommitted state. However, its default last-writer-wins policy means concurrent writes silently overwrite each other (lost updates). To prevent this, use conditional writes (ConditionExpression) for optimistic locking, or DynamoDB Transactions for serializable isolation. The trade-off: preventing lost updates requires a read-before-write round trip, reducing throughput.

RDBMS replication modes (sync vs async): In async replication (default in PostgreSQL, MySQL) the primary commits immediately and ships the WAL to replicas asynchronously. A replica failure never stalls the primary — the replica catches up when it returns. In sync replication, the primary waits for at least one replica to acknowledge before committing. If the replica is down, the primary stalls. Most production deployments use semi-sync — the primary waits for a configurable timeout (e.g. 100ms), then falls back to async to avoid blocking writes on a slow or unreachable replica.

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 9); “Brewer’s Conjecture and the Feasibility of C, A, P” (Eric Brewer, PODC 2000)

Write Anomalies

Write anomalies: Types of data corruption that can occur when multiple clients read and write concurrently without proper isolation. The choice of consistency model and database architecture determines which anomalies are possible.

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 7)

Partitioning (Sharding)

Partitioning: Splitting a dataset across multiple independent nodes so no single node holds everything. Each partition is responsible for a subset of the data. In HA terms:

• Limits blast radius — if one partition fails, only the data on that node is affected; the rest stays up.
• Enables horizontal scaling — more partitions = more nodes = more room for redundancy.
• Determines rebalance complexity — adding/removing nodes requires redistribution (e.g. consistent hashing minimises moves).

Partitioning strategies (choices):

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 6); The Art of Scalability by Abbott & Fisher (Ch. 15)

Caching

Caching: Temporarily storing frequently accessed data in a fast, nearby layer to reduce load on slower backend systems. In HA context:

Placement choices:

Write policy choices:

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 12)

Replication Patterns

Replication has two orthogonal dimensions: topology (who can accept writes) and mode (when the write returns to the caller).

Topology patterns:

Replication modes (applies within any topology):

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 5); Database Internals by Alex Petrov (Ch. 9-10)

Leader Election / Distributed Coordination

Mechanisms for electing a single leader among multiple replicas to coordinate writes, assign partition ownership or manage task distribution:

How Raft consensus works. A cluster elects a single leader by majority vote. The leader is the only node that accepts writes — followers just replicate its decisions. No split-brain is possible because only one leader can exist per term.

Replicated write-ahead log (WAL). Every write is first appended to a durable log file on the leader. That log entry is replicated to a majority of followers before the leader commits it. Each follower appends the same entry to its own WAL in the same order. If the leader crashes, the new leader replays its WAL so no committed write is lost. Because the log is write-ahead (recorded before the state is updated) and replicated (copied to N servers), you get durability even if half the cluster dies. This is the foundation of Aurora, DynamoDB Global Tables, CockroachDB, Kafka’s KRaft mode and many other HA data services.

Ephemeral (in lease-based elections) means temporary — the lease auto-expires if not renewed. The leader periodically renews its lease to stay alive; if it crashes, the lease expires and triggers re-election. No manual cleanup needed.

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 8, 9); Database Internals by Alex Petrov (Ch. 7-8)

Load Balancing Patterns

Distribution algorithms:

Architectural patterns (LB placement):

Instance lifecycle / draining:

Sources: Site Reliability Engineering by Beyer et al. (Ch. 20)

Connection Pooling Patterns

Size strategy:

Health validation:

Exhaustion behaviour:

Bulkhead

Isolating system resources into isolated pools so a failure in one pool does not cascade to others — named after the watertight compartments (bulkheads) on a ship:

Sources: Building Microservices by Sam Newman (Ch. 7); Site Reliability Engineering by Beyer et al. (Ch. 22)

Retry Logic Patterns

Sources: Site Reliability Engineering by Beyer et al. (Ch. 22); Designing Data-Intensive Applications by Martin Kleppmann (Ch. 8)

Health Check Patterns

Probe type:

Check depth:

Aggregation:

Direction:

Sources: Kubernetes in Action by Marko Lukša (Ch. 8); Site Reliability Engineering by Beyer et al. (Ch. 20)

Traffic Shifting Patterns

Cross-cutting rule: Never shift more traffic than you can absorb if the new version fails. Stage the rollback before the shift starts.

Sources: Continuous Delivery by Humble & Farley (Ch. 10); The DevOps Handbook by Kim et al. (Ch. 13)

TTL Management Patterns

Disaster Recovery Strategies

DR strategies trade off recovery speed against cost. Each has different RTO and RPO characteristics:

Sources: Site Reliability Engineering by Beyer et al. (Ch. 29); AWS Well-Architected Framework Reliability Pillar

Auto-scaling Strategies

Auto-scaling ensures enough compute capacity is available to handle traffic without over-provisioning:

Sources: Site Reliability Engineering by Beyer et al. (Ch. 20)

Rate Limiting / Throttling

Rate limiting protects backend services from resource exhaustion by rejecting excess requests:

Sources: Building Microservices by Sam Newman (Ch. 7); Site Reliability Engineering by Beyer et al. (Ch. 21)

Backpressure / Load Shedding

When the system cannot keep up with demand, backpressure mechanisms slow intake or shed load rather than degrading unpredictably:

Sources: Site Reliability Engineering by Beyer et al. (Ch. 21); Building Microservices by Sam Newman (Ch. 7)

Timeout Patterns

Timeouts prevent a single slow dependency from consuming resources indefinitely:

Sources: Site Reliability Engineering by Beyer et al. (Ch. 22); Designing Data-Intensive Applications by Martin Kleppmann (Ch. 8)

Service Discovery

Service discovery lets clients find healthy instances of a dependency without hard-coding addresses:

Sources: Building Microservices by Sam Newman (Ch. 8); Kubernetes in Action by Marko Lukša (Ch. 5)

Chaos Engineering

Chaos engineering validates that a system withstands unexpected failures by injecting faults in a controlled manner:

Sources: Chaos Engineering by Casey Rosenthal & Nora Jones; Site Reliability Engineering by Beyer et al. (Ch. 27)

Supervisor / Watchdog

A monitoring process that detects failures in a worker process and restarts it or triggers escalation:

Sources: Kuberentes in Action by Marko Lukša (Ch. 8); Site Reliability Engineering by Beyer et al. (Ch. 11)

Saga / Compensating Transactions

Sagas manage data consistency across multiple services without distributed transactions. Each step has a compensating action to undo it:

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 9); Microservices Patterns by Chris Richardson (Ch. 4)

Feature Flags / Toggles

Feature flags separate deployment from release, enabling instant kill-switches, gradual rollouts, and operational flexibility:

Sources: Continuous Delivery by Humble & Farley (Ch. 14); The DevOps Handbook by Kim et al. (Ch. 14)

Circuit Breaker

Prevent cascading failures by failing fast when a downstream service is unhealthy, then probing periodically for recovery:

Circuit breaker states:

Configuration dimensions:

Sources: Site Reliability Engineering by Beyer et al. (Ch. 22); Building Microservices by Sam Newman (Ch. 7); Microservices Patterns by Chris Richardson (Ch. 4)

Graceful Degradation / Fallback

When a dependency fails or degrades, the system continues operating at reduced functionality rather than failing entirely:

Sources: Building Microservices by Sam Newman (Ch. 7); Site Reliability Engineering by Beyer et al. (Ch. 22)

API Gateway / Backend for Frontend (BFF)

A single entry point for client requests that handles cross-cutting concerns, reducing round-trips and centralizing HA policy enforcement:

Sources: Building Microservices by Sam Newman (Ch. 8); Microservices Patterns by Chris Richardson (Ch. 8)

CQRS (Command Query Responsibility Segregation)

Separating read and write data models so each can be optimized independently for its workload:

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 11); Microservices Patterns by Chris Richardson (Ch. 11)

Transactional Outbox

Ensures reliable message or event publishing by writing the message to the same database as the business data within the same transaction, eliminating the need for distributed transactions:

Sources: Microservices Patterns by Chris Richardson (Ch. 7); Designing Data-Intensive Applications by Martin Kleppmann (Ch. 11)

Dead Letter Queue (DLQ)

A holding area for messages or events that cannot be processed successfully after repeated retries:

Sources: Building Event-Driven Microservices by Adam Bellemare (Ch. 6); AWS Well-Architected Framework Reliability Pillar

Event Sourcing

Storing the full sequence of state-changing events as the authoritative source of truth rather than only the current state:

Sources: Designing Data-Intensive Applications by Martin Kleppmann (Ch. 11); Microservices Patterns by Chris Richardson (Ch. 10)

This reference is part of the High Availability series.