A distributed-system collapse may begin with an error that appears limited, but it can quickly turn into a chain of failures when many components depend on one another. In a presentation by Sam Newman, the independent software engineering consultant connects the concept of Progressive Collapse in civil engineering with resilience problems in digital systems, concluding that preventing all errors is impossible, but the chances of their spreading and their impact can be reduced.
From the Ronan Point Building to Cloud Services
Newman returns to the incident involving the partial collapse of the Ronan Point tower in the Canning Town area of London in 1968. A limited gas explosion inside Mrs. Ivy Hodge’s apartment dislodged an exterior wall that supported part of the building. The four floors above the explosion site collapsed, which then caused part of the tower’s corner to fall in a cascading effect. Four people died, and the timing of the incident, shortly before six in the morning, helped limit the number of casualties.
The significance for software is not the physical similarity between a building and a digital service, but the failure pattern: a small initial fault disables a component on which other components depend, expanding the scope of the damage. This chain is difficult to understand in distributed systems because the relationships among services, databases, load balancers, and DNS systems are not always visible as a row of dominoes.
Digital Examples of Cascading Failure
During an outage affecting AWS’s us-east-1 region in October, a subcomponent of DynamoDB encountered a problem while updating DNS routes within AWS’s infrastructure. A fault in the process for updating plans led to routes being deleted in Route 53, after which services that depended on those routes began to fail, including network load balancers, compute, queues, and EKS. Because other services use these foundational services, the problem spread to user-facing products and companies, including Alexa, Ring, Slack, Snapchat, Zoom, and Shopify, with some affected partially and others completely.
According to Newman’s explanation, based on an AWS report, the system included a planner that created DNS change plans and several executors that applied them. One plan took longer than usual, and then another plan was created and executed quickly by a different executor. When the older plan was later completed, it removed the routes that the newer plan had created. The result was a race condition that caused DNS entries to be deleted.
The second example was a website selling used cars, motorcycles, and trailers, operating through an application with the code name Sauron on ten servers. The application normally handled 30 to 60 concurrent requests, but it faced more than 800 requests. One of the dependent sites accepted connections and then hung without responding, while the application waited 30 seconds before terminating the request. The connection pool was exhausted, after which threads accumulated and processors spent their time managing them, causing the entire system to stop. Users increased the pressure by repeatedly clicking the refresh button.
Three Paths to Reducing Collapse
Newman believes that focusing on a single “root cause” leads to harmful oversimplification. Preventing the spark may be useful, but it does not address all the conditions that allow the fire to spread. Therefore, resilience engineering focuses on accepting the possibility of failures and preparing to limit their effects through three interconnected categories:
- Reducing risks: Removing sources of danger or limiting their ability to exhaust resources. In the AWS case, this included temporarily stopping automated DNS management until the problem was addressed and adding better tests for the component. In the Sauron application, load shedding could have been used to accept a specific number of requests and reject those exceeding the system’s capacity, rather than allowing them to accumulate until collapse.
- Strengthening components: Increasing a service’s ability to withstand the failure of part of itself through redundancy, monitoring, testing, and improved implementation. This may mean running multiple copies of a service behind a load balancer, but the decision is tied to the component’s importance and cost; the Sauron application was nearing retirement and represented only part of the revenue, making the operation of additional backup copies an unattractive commercial option at the time.
- Reducing coupling: Preventing a failure in one component from spreading to the rest of the system. This includes using bulkheads, timeouts, alternative paths, and local-first designs where possible. The less one part of a system depends on another, the more a failure can be contained rather than propagated.
What Changes Practically for Technology Teams?
Resilience does not automatically mean running a system in two regions or two clouds. Newman explains that the options range from backup and recovery, to a Pilot Light architecture that replicates data without running the full infrastructure, then warm recovery, and finally running two active sites at the same time. These options progressively reduce downtime and data loss, but they increase cost and complexity.
He also warns against treating bidirectional synchronization between sites as simple, especially when it is added to an existing application that was not designed for it. The same applies to multicloud environments; they may reduce dependence on a single provider, but they require different skills and operations for each platform, and deploying the same code on AWS and Azure is not enough to guarantee independence, because the same software defect could disable both environments.
Conclusion
The central lesson is that resilience is not about seeking an environment that never fails, but about designing a system so that part of it remains operational when another part fails. Risks, capacity limits, bottlenecks, and dependency paths should therefore be assessed, followed by balancing redundancy, isolation, and cost. Decisions such as multiregion or multicloud operation remain engineering and business choices tied to the importance of the service, not a general formula suitable for every system.