Failure library

Good architecture is mostly about what happens when something goes wrong. These incidents are among the best documented in the industry, because the organizations involved published detailed reviews. Each entry summarizes what happened in plain terms, then names the patterns that would have limited it.

A note on fairness: every organization here has excellent engineers. These failures happened because large systems fail in surprising ways, and because the companies were willing to explain them publicly. That openness is what makes them worth studying.

At a glance

Incident	What failed	Pattern that limits it
Amazon S3, 2017	An operator command removed too many servers	Guardrails on tools, smaller blast radius (cells)
Knight Capital, 2012	A partial deployment woke up dead code	Consistent automated deployment, kill switches
GitLab, 2017	A data directory deleted on the wrong server; backups didn’t work	Tested restores, point-in-time recovery
Cloudflare, 2019	One bad regular expression, deployed everywhere at once	Staged rollouts, runtime limits
Facebook, 2021	A maintenance command cut the backbone; tools depended on it	Independent recovery paths, command auditing
Fastly, 2021	A valid customer change triggered a latent bug	Isolation between tenants, fast global kill
Roblox, 2021	One overloaded coordination cluster; monitoring depended on it	Independent monitoring, sharded control planes
Slack, 2021	A traffic surge saturated a network gateway; autoscaling made it worse	Capacity headroom, load-tested provisioning
GitHub, 2018	A 43-second network split caused a cross-country database failover	Region-aware failover, tested assumptions
Atlassian, 2022	A script deleted customer sites using the wrong IDs	Soft deletes, bulk restore drills
CrowdStrike, 2024	A content update with one field too many crashed Windows hosts	Staged rings, input validation, customer control
AWS DynamoDB, 2025	A race condition left a DNS record empty	Safe automation, limits on automated removal

Amazon S3 (us-east-1), February 2017

What happened. While debugging a slow billing system, an engineer ran an established playbook command but mistyped an input, and far more servers were removed than intended. They supported two core S3 subsystems: the index, which tracks where every object lives, and placement, which allocates space for new objects. Both had to restart fully, which they hadn’t done in years in large regions. Restart and integrity checks took hours.

Why it spread. Many AWS (opens in a new tab) services depend on S3 in that region, including EC2 instance launches, EBS restores from snapshots and Lambda. Even AWS’s status dashboard depended on S3, so it couldn’t be updated at first.

What limits it. AWS changed the tool to remove capacity more slowly and refuse to go below a minimum, audited other tools for similar safeguards, and accelerated splitting the index into smaller cells so a restart covers less. Your own status page should not depend on the system it reports on.

Source: Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region (opens in a new tab), AWS

Knight Capital, August 2012

What happened. Knight deployed new trading code for a stock-exchange programme, but the deployment to its order router was done incorrectly. Some orders then triggered an old function that had been left in the code for years without use. In the first 45 minutes after the market opened, the router sent more than 4 million orders while trying to fill 212 customer orders, trading over 397 million shares. Knight lost more than $460 million.

Why it spread. There was no automated check comparing the orders leaving the router with the orders entered, and no control stopped orders that exceeded capital limits. Automated emails flagged the problem before the market opened, but nobody acted on them.

What limits it. Automated, identical deployment to every server; removing dead code; pre-trade limits enforced outside the system that generates orders; and a kill switch someone is empowered to use. The SEC fined Knight $12 million.

Source: SEC Charges Knight Capital With Violations of Market Access Rule (opens in a new tab), US Securities and Exchange Commission, October 2013

GitLab, January 2017

What happened. Heavy load caused a database replica to fall behind. While rebuilding it, an engineer ran a command to wipe the replica’s data directory on the primary by mistake, deleting about 300 GB. When the team turned to backups, the nightly database dumps had been failing silently because of a version mismatch, and the failure emails had been rejected. Disk snapshots weren’t enabled on the database servers.

Why it spread. Every backup mechanism had a gap nobody had noticed. The best option was a six-hour-old copy made for the staging environment, which took about 18 hours to copy back over slow disks. Around 5,000 projects, 5,000 comments and 700 user accounts created in that window were lost; Git repositories were not affected.

What limits it. Test restores regularly and automatically, monitor that backups succeed, use continuous archiving for point-in-time recovery, and make destructive commands hard to run against production. GitLab also assigned a named owner for data durability.

Source: Postmortem of database outage of January 31 (opens in a new tab), GitLab, February 2017

Cloudflare, July 2019

What happened. Cloudflare deployed a new web application firewall rule containing a regular expression that backtracked heavily. It drove CPU to nearly 100% on the machines handling HTTP traffic worldwide, and Cloudflare lost about 80% of its traffic for 27 minutes.

Why it spread. Firewall rules were allowed to go to every machine at once, unlike Cloudflare’s other software, which rolls out in stages. Tests didn’t check for excessive CPU use, and a CPU safeguard had been removed in an earlier refactor.

What limits it. Staged rollouts for configuration as well as code, with an emergency path kept for real attacks; a regular-expression engine that guarantees bounded run time; and performance tests in the rule pipeline.

Source: Details of the Cloudflare outage on July 2, 2019 (opens in a new tab), Cloudflare

Facebook, October 2021

What happened. During routine maintenance, a command meant to check backbone capacity instead disconnected all of Facebook’s data centers from each other. A tool that should have stopped the command had a bug. Facebook’s DNS servers, designed to withdraw themselves from the internet when they can’t reach the data centers, did exactly that, so Facebook, Instagram and WhatsApp disappeared from the internet.

Why it spread. The internal tools engineers would use to fix the network also depended on the network. Engineers had to go to data centers in person, where strong physical security slowed access. Services then had to be brought back gradually to avoid power surges.

What limits it. Recovery paths that don’t depend on the thing being recovered, reliable auditing of dangerous commands, and drills that simulate losing the whole backbone.

Source: More details about the October 4 outage (opens in a new tab), Engineering at Meta

Fastly, June 2021

What happened. A software release in May contained a bug that only certain customer configurations could trigger. On June 8 a customer pushed a perfectly valid change that did, and about 85% of Fastly’s network started returning errors, taking many major websites with it. Monitoring detected it within a minute and 95% of the network was back within 49 minutes.

Why it spread. All customers shared the same software on the same machines, so one customer’s configuration could affect everyone.

What limits it. Stronger isolation between tenants, testing with a wide range of real-world configurations, and the ability to disable a single configuration quickly everywhere. Fastly’s fast detection and response kept a global failure under an hour.

Source: Summary of June 8 outage (opens in a new tab), Fastly

Roblox, October 2021

What happened. Roblox was down for 73 hours. Its backend relied on a single cluster of HashiCorp Consul (opens in a new tab), a service-discovery and configuration system. A newly enabled streaming feature caused contention under very high load, and a storage-layer problem made writes slow on the cluster’s leaders. Both were deep inside Consul and hard to see.

Why it spread. Every backend service depended on the same Consul cluster. Roblox’s monitoring also depended on it, so when Consul struggled, the data needed to diagnose it disappeared too. Bringing millions of players back at once risked overloading caches and databases.

What limits it. Monitoring that doesn’t depend on the systems it watches, splitting critical control-plane systems into separate clusters, and reopening gradually: Roblox restored players in steps of about 10%.

Source: Roblox Return to Service 10/28-10/31 2021 (opens in a new tab), Roblox

Slack, January 2021

What happened. On the first working day after the holidays, Slack’s traffic surged and clients with cold caches pulled extra data. An AWS Transit Gateway connecting Slack’s networks didn’t scale fast enough and dropped packets. Slack’s web tier slowed, so autoscaling tried to add about 1,200 servers, but the service that sets up new servers was overwhelmed by the same network problems and hit its own limits.

Why it spread. Recovery automation depended on the failing network, and Slack’s dashboards and alerts ran in a different network from their databases, so they failed too.

What limits it. Capacity headroom for known peaks (Slack now asks for gateways to be scaled up before holidays), load-testing the provisioning system itself, and placing monitoring where it doesn’t depend on shared network links.

Source: Slack’s Outage on January 4th 2021 (opens in a new tab), Slack Engineering

GitHub, October 2018

What happened. Maintenance on network equipment cut the link between GitHub’s East Coast hub and its main data center for 43 seconds. In that time, the database failover system promoted West Coast databases to primary. When the link returned, the application wrote to the West Coast, while some recent East Coast writes had not yet copied across. Each side now had data the other lacked.

Why it spread. Running with cross-country database latency made the site largely unusable, but simply switching back would lose data. GitHub chose to protect data over speed of recovery, restoring from backups and replaying writes. Service was degraded for 24 hours and 11 minutes.

What limits it. Failover rules that respect geography, so a brief network blip can’t move primaries across regions; tested assumptions about failover; and serving from several data centers so one site’s loss doesn’t matter.

Source: October 21 post-incident analysis (opens in a new tab), GitHub

Atlassian, April 2022

What happened. A script meant to remove a retired app was given site IDs instead of app IDs. The deletion API accepted either without warning, and in 23 minutes it deleted 883 customer sites. 775 customers lost access, some for up to 14 days.

Why it spread. Deletion was immediate, with no soft-delete step. Atlassian’s restore tooling handled one site at a time, not hundreds across several products in shared data stores, so restores were partly manual. Customer contact details were deleted too, which slowed communication. Atlassian reported that no customer lost more than five minutes of data.

What limits it. Soft deletes with a retention period, APIs that reject ambiguous identifiers, staged rollouts for bulk operations, and drills that restore many tenants at once.

Source: Post-Incident Review on the Atlassian April 2022 outage (opens in a new tab), Atlassian

CrowdStrike, July 2024

What happened. CrowdStrike sent a content update for its Falcon security sensor to Windows hosts. The update defined 21 input fields, but the sensor code expected 20, causing an out-of-bounds memory read that crashed affected machines. Because the sensor runs deep in the operating system, machines failed to start, disrupting airlines, hospitals and banks worldwide.

Why it spread. Content updates were treated differently from software releases and reached hosts quickly, without staged acceptance rings.

What limits it. Bounds checks and input validation in the code that interprets content, staged deployment rings for content as well as code, and giving customers control over when updates reach their machines. CrowdStrike adopted all three.

Source: Channel File 291 Root Cause Analysis: Executive Summary (opens in a new tab), CrowdStrike, August 2024

AWS DynamoDB (us-east-1), October 2025

What happened. DynamoDB’s automated DNS management had a race condition. A delayed worker applied an old DNS plan over a newer one; a clean-up process then deleted that old plan, leaving the main regional DynamoDB endpoint with no IP addresses at all. It needed manual intervention to fix.

Why it spread. Many AWS services depend on DynamoDB internally. EC2’s system for managing physical hosts couldn’t renew its leases and then struggled to recover; network load balancers removed healthy capacity because new instances failed health checks. The event lasted about 14.5 hours across many services.

What limits it. Safety checks in automation, so it can never publish an empty or stale configuration; rate limits on automated capacity removal, which AWS added to its load balancers; and recovery workflows that are themselves tested under load.

Source: Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region (opens in a new tab), AWS

Patterns that keep showing up

Roll out everything in stages, including configuration and content. Cloudflare, Fastly and CrowdStrike all pushed a change everywhere at once.
Limit the blast radius. Cells, shards and tenant isolation turn a global outage into a partial one.
Keep recovery and monitoring independent of the system they protect. Facebook, Roblox, Slack and S3 all lost visibility or tools when they needed them most.
Put guardrails on powerful tools and automation: minimum capacity, rate limits, confirmation of identifiers, and soft deletes.
Test the recovery, not just the backup. GitLab and Atlassian both had backups; restoring them was the problem.
Bring systems back gradually. Facebook and Roblox both had to avoid overloading systems as traffic returned.

Written and edited by Abhinav Taduka, enterprise software architect.