I have spent enough time around distributed systems to be suspicious of any resilience mechanism that operates without understanding where a failure originated. I remind why that matters with Uber’s write-up on retry storms.

In a deep service graph, retries that look reasonable at the level of one caller can multiply as an error propagates upstream, increasing load on the component that is already failing. Uber addresses this by introducing error ownership into shared infrastructure. A service distinguishes an error it originated from one it is propagating, and callers use that information to decide whether another attempt can plausibly help. In one production incident, Uber reports that this mechanism prevented roughly 9.5 million spurious requests, while preserving at-least-once retry behavior where appropriate.

The architectural point extends beyond retries. Reliability controls become more effective when they operate on causal context rather than local symptoms; rate limits, circuit breakers, backpressure, and retries all make better decisions when the system can preserve enough provenance across service boundaries to identify where intervention belongs. For large service meshes, resilience is then partly a context-propagation and attribution problem, not only a configuration problem.

https://www.uber.com/us/en/blog/protecting-against-retry-storms/