“Windswept hawthorn tree in Ireland.” Photo: Eoin Gardiner (2008). CC BY 2.0.
Resilience
At some point, most engineers encounter an incident their design should have prevented. Everything looked fine. Circuit breakers were in place and retries were configured, but the system failed anyway.
That’s the failure mode resilience engineering was built to understand.
Resilience vs Robustness
Robustness is the capacity to handle known failure modes. Resilience is the capacity to handle failures beyond what the designer foresaw. Robustness is designed in advance. Resilience is something a system does when conditions change[1].
The distinction matters because complex systems regularly produce failures that fall outside what their designers imagined.
Testing Has Limits
Rigorous testing, redundancy, and defensive programming all rest on the assumption that every failure mode can be anticipated and prevented. For complex distributed systems, that assumption breaks down[2].
Complexity and emergent behavior mean that distributed systems create interactions beyond what individual components predict. Testing can only explore a fraction of all possible system states[3].
Cascading failures occur when localized problems propagate through system dependencies. A slow database query triggers connection pool exhaustion, causing application timeouts, leading to retry storms that overwhelm the database further.
Human-computer interaction under pressure means operators are making decisions with incomplete information during the moments when systems are most likely to fail them[3]. Experienced operators improvising around system brittleness is what keeps complex systems running. Human expertise is the adaptive capacity.
Resilience Engineering grew from studying safety-critical industries like aviation, healthcare, and nuclear power. Pioneered by Hollnagel, Cook, Dekker, and Woods, it shifts focus from preventing all failures to building systems that hold up when something unexpected happens.
Core Principles
Four capabilities define a resilient system[2].
Anticipation. Knowing what to expect and preparing for potential disruptions. This extends beyond traditional risk assessment to include scenario planning, load testing, and chaos engineering.
Monitoring. Knowing what to look for in system behavior. Effective monitoring observes leading indicators of system stress, not just lagging indicators of failure, using observability that reveals how systems actually behave under varying conditions.
Response. Knowing what to do when anomalies occur. Response capability depends on technical mechanisms like circuit breakers and organizational practices like incident command structures.
Learning. Knowing what has happened and understanding why. This goes beyond documenting incidents to understanding how system behavior, human decisions, and organizational factors shaped the outcome.
Resilience and Safety
Safety-I focuses on eliminating failure causes. Safety-II focuses on understanding successful adaptations, grounded in the observation that things go wrong and things go right for the same basic reasons[4]. Traditional software reliability is Safety-I thinking. You learn more about how a system actually works by studying the thousands of times people adapted successfully than by studying the handful of times it failed.
Your Role in Building Resilience
Your design decisions shape how much adaptive capacity a system has.
Design for graceful degradation. Circuit breakers, bulkheads, and timeouts are the implementation[5]. Think beyond happy paths. What happens when a dependency is slow, unavailable, or returning corrupt data? Build mechanisms that respond through fallbacks, cached data, or reduced functionality.
Build observability in from the start. Good instrumentation reveals how the system behaved in the lead-up to a failure, not just that it failed[6]. Instrument for the operator who will be debugging this system at 3 AM, not just for normal operations.
Participate in incident response. Volunteer for on-call and get involved in incidents outside your area. Incident commanders and escalation paths provide structure under pressure[6], and game days make that structure practiced rather than improvised.
Invest in learning. Blameless postmortems surface how people adapted and held things together under pressure. That’s the richest source of information about how a system actually works. People report accurately only when they feel safe doing so[1].
Balance velocity with resilience. Only some systems need chaos engineering or elaborate circuit breakers[7]. Assess risk based on blast radius, recovery time objectives, and business impact.
Protect the conditions for human resilience. Psychological safety, shared mental models, and the freedom to speak up about risk are what allow adaptive capacity to surface. An experienced operator who stays quiet about a problem because of how it will be received is a system with a silenced warning light.
The work extends beyond the code. Model resilience thinking in design reviews, ask questions about failure modes, and share stories of how systems failed.
How Teams Get This Wrong
Confusing redundancy with resilience. Adding more servers, databases, or availability zones improves fault tolerance without guaranteeing resilience. Resilient systems adapt to unexpected conditions, a different skill than tolerating anticipated failures[8].
Treating over-engineering as resilience. Over-engineering is a false substitute for adaptive capacity. Resilience often needs simple, well-understood mechanisms instead of elaborate ones for every possible failure. The most resilient solution is often the one that’s easy to reason about during an incident.
Searching for a single root cause. Complex system failures result from combinations of conditions, decisions, and circumstances[9]. Chasing a single cause impedes learning.
Trusting aggregate metrics alone. Metrics can hide system health. Aggregate metrics like average response time may look healthy while a subset of users experiences severe degradation. Percentile metrics, error budgets, and user-centric measurements give better visibility into what the system is actually doing[6].
Put It Into Practice
Pick one dependency your system relies on and ask what happens when it’s slow, unavailable, or returning corrupt data. The answer should be “we degrade gracefully and we’d know within minutes.” Getting there is the work. Build the observability, add the circuit breaker, write the runbook, and run the game day.
Then talk to the people who operate this system under pressure. What do they know that the runbook missed? What are they quietly working around? Learning how people cope with surprise is the path to finding sources of resilience.
Where is your system weakest? Where would your team struggle most, and what do they know about it beyond what you do? Ask those questions in design reviews and postmortems. Resilience takes more than a single project or sprint to emerge. It’s built by the people who ask consistently, and who create the conditions for truthful answers.
References
- Allspaw, John (2012). "Blameless PostMortems and a Just Culture." Etsy Code as Craft Blog. https://web.archive.org/web/20230601000000/https://codeascraft.com/2012/05/22/blameless-postmortems/
- Hollnagel, Erik (2011). Resilience Engineering in Practice: A Guidebook. Ashgate Publishing. https://www.routledge.com/Resilience-Engineering-in-Practice-A-Guidebook/Hollnagel-Paries-Wreathall/p/book/9781472420749
- Cook, Richard I. (2000). "How Complex Systems Fail." Cognitive Technologies Laboratory, University of Chicago. https://how.complexsystems.fail/
- Hollnagel, Erik (2014). Safety-I and Safety-II: The Past and Future of Safety Management. Ashgate Publishing. https://www.routledge.com/Safety-I-and-Safety-II-The-Past-and-Future-of-Safety-Management/Hollnagel/p/book/9781472423085
- Nygard, Michael T. (2018). Release It! Design and Deploy Production-Ready Software, 2nd ed. Pragmatic Bookshelf. https://pragprog.com/titles/mnee2/release-it-second-edition/
- Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media. https://sre.google/sre-book/
- Basiri, Ali, et al. (2016). "Chaos Engineering." IEEE Software, 33(3): 35-41. https://arxiv.org/abs/1702.05843
- Woods, David D. (2015). "Four Concepts for Resilience and the Implications for the Future of Resilience Engineering." Reliability Engineering & System Safety, 141: 5-9. https://maritimesafetyinnovationlab.org/wp-content/uploads/2021/06/4sensesofresiliencepublic.pdf
- Dekker, Sidney (2006). The Field Guide to Understanding Human Error. Ashgate Publishing. https://sidneydekker.com/the-field-guide-to-understanding-human-error
Changelog
2026-06-07 Added hero image.
2026-05-01 Expanded from a notes-and-quotes format. Removed the Goldratt, Bergström, and Dekker quotes.
2019-10-30 Initial publication.