7.1Approaches to High Availability

Downtime costs revenue and trust directly. That is why operational design centers on high availability: not just staying up, but recovering fast when something breaks. This section walks through setting numeric targets, finding weak points, and closing them with redundant architecture.

7.1.1Failure patterns and eliminating SPoF

• Failure patterns fall into three groups: physical failures (hardware, power, network outages), logical failures (software bugs, misconfiguration, overload), and planned maintenance (scheduled downtime for patching or hardware swaps). Each calls for a different kind of countermeasure.
• SPoF (single point of failure) is a component whose failure stops the whole system: a lone load balancer, a single power feed, a single network path. The first step in HA design is drawing the architecture and hunting down every SPoF to eliminate with redundancy.
• Resiliency is the property of continuing to serve, or self-recovering, despite failures—not just redundancy, but automated failover and planned failback as well.

7.1.2Availability metrics and clustering

• MTBF (mean time between failures) is the average time from one failure to the next (how rarely it breaks). MTTR (mean time to repair) is the average time from failure to recovery (how fast it is fixed). Availability = MTBF / (MTBF + MTTR), and an SLA (service level agreement) contractually commits to a target value.
• RPO (recovery point objective) is how much data loss is acceptable (e.g., RPO of 1 hour tolerates losing the last hour of data). RTO (recovery time objective) is how long recovery may take (e.g., RTO of 4 hours means service must return within 4 hours). They measure different axes—data loss vs. elapsed time—and must not be conflated.
• The implementation path for redundancy is clustering: multiple nodes provide one service together. On Linux the standard pairing is Pacemaker (the cluster resource manager, deciding and executing failover) with Corosync (the membership/heartbeat and messaging layer between nodes). Combine this with load balancing (spreading traffic across nodes) and geographic distribution (redundancy across sites/regions, surviving even a whole data center outage).

A concrete scenario helps this stick. Suppose an e-commerce site agrees to an SLA of "RPO 15 minutes, RTO 2 hours." Meeting RPO 15 minutes requires replicating or backing up data more frequently than every 15 minutes (e.g., semi-synchronous DB replication or frequent WAL shipping). Meeting RTO 2 hours requires automation that completes detection, failover, and service restoration within 2 hours (monitoring, alerting, automated failover via Pacemaker, or a documented manual runbook). The two targets are independent: tightening RPO does not automatically tighten RTO—more frequent backups do nothing for RTO if the recovery procedure is still manual and slow. In a cluster, Corosync continuously exchanges heartbeats between nodes to track who is alive; when it detects a lost node, Pacemaker executes a predefined failover policy (which resources move where, how the old node is fenced off) to keep the service running. A single-node cluster, or all nodes wired through one network switch, is itself a SPoF—only by multiplying power feeds, network paths, and, where possible, entire data centers (geographic distribution) does a design approach "truly no single point of failure."

7.1.3Section summary

• Eliminate SPoF + availability = MTBF/(MTBF+MTTR). RPO = data-loss tolerance / RTO = recovery-time tolerance (different axes, independent targets)
• Redundancy is implemented via clustering (Pacemaker = resource management, Corosync = heartbeat/membership) + load balancing + geographic distribution