Failure Scenarios

A single application server crashes due to hardware failure, OOM kill, or software bug. All active requests on that server are lost.

The primary database server crashes. Writes are completely blocked. Reads may still work from replicas, but data could be stale.

Network connectivity is lost between parts of the system. Servers are running but can't communicate. This is the 'P' in the CAP theorem.

One component fails, causing increased load on other components, which then also fail. Like dominos falling. Often starts with a single overloaded service.

A cache entry expires and thousands of requests simultaneously hit the database to regenerate it. The DB gets overwhelmed by requests that would normally be served from cache.

One database partition or cache key receives disproportionate traffic. The node hosting that partition becomes a bottleneck while others sit idle.

Data becomes corrupted or deleted due to software bugs, bad deployments, or accidental operations. Unlike server failures, this can't be fixed by just restarting.

A third-party API or downstream service becomes slow (not down, but 10x slower). This is worse than a full outage because requests hang, consuming threads and connections.

Application memory usage grows over time until the process is killed by the OS (OOM killer). Service becomes increasingly slow before crashing.

DNS servers become unavailable or return incorrect records. Services can't resolve hostnames, breaking all inter-service communication and external API calls.

A new code deployment introduces a bug that causes errors or performance degradation. The blast radius depends on the deployment strategy.

A sudden spike in traffic (legitimate or attack) overwhelms the system. Without protection, all users experience degraded service.