Most production incidents we’ve reviewed after the fact weren’t caused by one service failing - they were caused by one service failing and every other service depending on it failing along with it, in a cascade nobody designed for. Resilient system design isn’t about preventing every failure, which is impossible. It’s about making sure one failure stays one failure.
Why a single failing service becomes a system-wide outage
The default, naive behavior when Service A calls Service B and B is slow or down: A’s request hangs, waiting for a response that isn’t coming, tying up A’s own resources (threads, connections) while it waits. If enough requests to A pile up waiting on a struggling B, A itself becomes exhausted and starts failing too - and if other services depend on A, the failure keeps propagating outward. One slow dependency becomes a system-wide outage, not because the original failure was catastrophic, but because nothing stopped it from spreading.
The circuit breaker pattern, explained without the electrical metaphor
A circuit breaker wraps calls to a dependency and tracks whether those calls are succeeding or failing. When failures cross a threshold, the breaker “trips” - it stops sending requests to the struggling service entirely for a period, immediately returning a fallback response instead of waiting on a call that’s very likely to fail or time out anyway. After a cooldown period, it allows a small number of test requests through to check if the dependency has recovered, and closes again (resuming normal traffic) if they succeed. The concrete benefit: instead of every caller hanging and consuming resources waiting on a failing service, they fail fast and predictably, and the struggling service gets breathing room to recover instead of being hammered with a full load of requests it can’t currently handle.
What “implementing retries” actually needs to look like
Naive retry logic - just try the failed call again immediately - often makes outages worse, not better, because a struggling service getting hit with an immediate flood of retries from every failed caller is exactly the load pattern that prevents it from recovering. Proper retry logic needs:
- Exponential backoff - increasing the delay between retry attempts, rather than retrying immediately, so retries don’t compound into another load spike on an already-struggling service.
- Jitter - randomizing that delay slightly across different callers, so many clients retrying a failed call don’t all retry at the exact same moment, which would itself create a synchronized load spike.
- A sensible retry limit, combined with the circuit breaker - retries should stop and the breaker should trip once it’s clear the dependency isn’t just having a brief blip but is genuinely down, rather than retrying indefinitely.
What a good fallback actually looks like
The other half of this pattern that’s often skipped: what happens when the circuit breaker trips and a call genuinely can’t succeed. A good fallback returns something useful and honest - cached or slightly stale data with a clear indicator it’s not live, a degraded but functional experience, or a clear, specific error rather than a generic failure. A recommendation engine that’s down should fall back to showing popular items, not an error page; a feature genuinely not core to the primary user flow should degrade gracefully rather than taking the whole page down with it.
Where we actually apply this discipline
Not uniformly across every internal call - that’s overkill for low-stakes internal services where a failure genuinely doesn’t cascade. We apply circuit breakers and careful retry logic specifically at the boundaries that matter most: calls to third-party services outside your control, calls between services where a failure would visibly affect users, and any dependency with a history of intermittent instability. Knowing where this discipline actually pays off, rather than applying it everywhere reflexively, is itself part of the design skill.
We build this resilience into every production system we architect, at the boundaries where it actually matters. If a single struggling dependency has ever taken down more of your system than it should have, get in touch and we’ll look at where the cascading failure risk actually lives in your architecture.