BA community

Fault tolerance isn't "we have a 99.9% SLA" — it's a specific checklist Behind the slogan "the system needs to be reliable" is a set of concrete decisions — some in code, some in infrastructure. Here's the checklist, with real numbers where they help. SOFTWARE LEVEL ➡️ One shared retry rule for all service-to-service calls If every service retries its own way, a struggling node gets hit by everyone at once and a small problem becomes an outage. 💡 Google SRE's guidance: exponential backoff with jitter, 3-5 tries max, wait capped at 10-30 seconds, and a system-wide retry cap (~10% of normal traffic, or a fixed number like 60/min). Without shared limits, retries become the outage. ➡️ A weekly check on memory usage per service Memory leaks rarely show up in load testing — they build up over weeks. Watching for services whose memory keeps climbing instead of leveling off catches the problem before a 2am crash and restart cascade. ➡️ Business logic that knows what to do when something's unavailable If the payment service goes down, the code needs a plan — queue the request, fall back to a simpler flow, degrade one feature — instead of just erroring out and losing the transaction. ➡️ Every service is stateless, no exceptions. A service holding local data can't be restarted or scaled without losing it. One exception eventually becomes the thing that brings everything down. INFRASTRUCTURE LEVEL ➡️ Duplicate infrastructure or spread it across regions One data center is a risk code can't fix. You need a backup ready to take over fast, or traffic spread across locations. ➡️ A separate read replica per data store. Splitting reads and writes takes load off the primary and keeps reads working during a failover. ➡️ Spare room in the connection pool, not just enough for normal load A pool sized exactly for everyday traffic has nothing left when a failed node's traffic gets redirected. Headroom is what makes that moment survivable. ➡️ Alerts that catch problems early. 💡 Google's four golden signals — latency, traffic, errors, saturation — are the standard baseline. Alert on p99 latency, not the average, and on error rate and saturation trend, not raw counts. The goal is catching warning signs before the system actually goes down. Fault tolerance means preparing in advance for failures that are individually unlikely but, together, inevitable. Miss any one item on this list, and you've usually found the title of your next post-mortem: "unexpected cascading failure." SystemArchitecture Reliability SRE Backend DevOps Scalability