4.3Availability and disaster preparedness

The previous section covered redundancy within a single data center (series/parallel availability calculation), but in practice, an architect must also prepare for the loss of an entire data center due to a large-scale disaster (earthquake, fire, or wide-area power outage), which redundancy alone cannot address. What matters here is quantitatively setting two recovery targets—"how quickly to recover" and "how much data loss is acceptable"—according to the criticality of the business. This section builds the judgment to design the level of availability and disaster preparedness by weighing it against cost, grounded in HA clusters for higher availability, the DR/BCP mindset, and the two metrics RTO and RPO.

4.3.1Configuration judgment from a target availability

• A system architect sets a target availability (e.g., 99.9% / 99.99% / 99.999%) from business requirements and works backward, via the series/parallel calculations from the previous section, to a configuration that meets it. Each additional nine of availability cuts allowable downtime by roughly a factor of ten (99.9% is about 8.76 hours/year, 99.99% is about 52.6 minutes/year, 99.999% is about 5.3 minutes/year), so the architect must be able to explain to stakeholders that raising the target by just one nine sharply raises the cost of achieving it.
• Setting the target availability excessively high causes cost to grow exponentially—through the number of components to parallelize, or through keeping a DR environment running at all times, as covered below. The design judgment, therefore, is to choose a configuration by the priority of discerning the availability the business actually requires, avoiding over-engineering, while still ensuring no single point of failure remains.

4.3.2HA clusters

• An HA cluster (High Availability Cluster) links multiple servers to behave as a single system, using failover to automatically hand processing to a standby server when the active one fails. Active-standby has only one unit processing during normal operation, with the other purely on standby (switching takes some time but the setup is simple and cheaper). Active-active has multiple units sharing the processing load simultaneously, so if one fails, the rest continue processing (faster switchover and full use of capacity, but a more complex setup).
• What matters in judging failover is whether the time required to switch over (the failover time) fits within the business's allowable downtime. Active-standby keeps costs down, but starting the standby unit and syncing state takes time, making some downtime likely. Active-active can push this switchover time close to zero, but it comes with the cost of running two or more units at all times and the complexity of a synchronization mechanism to keep data consistent.

4.3.3DR/BCP and RTO/RPO

• DR (disaster recovery) is preparation for a data-center-scale disaster such as an earthquake, fire, or wide-area power outage, aimed at restoring the system at a geographically separate remote site. BCP (business continuity plan) is a company-wide plan—not limited to IT systems—for continuing or quickly restoring the business itself during a disaster (covering personnel, sites, and business processes); DR is positioned as one IT-side means of realizing BCP.
• RTO (Recovery Time Objective) is the target for the duration allowed from the moment of failure until the system is restored (the upper bound on "how long it may remain down"). RPO (Recovery Point Objective) is the target for the acceptable extent of data loss at recovery, expressed as how far back from the moment of failure the data may be lost up to (the upper bound on "how much data may be lost"). RTO is a time axis (by when must it be back) and RPO is a data axis (back to which point)—two distinct axes that must not be conflated.

Suppose a regional bank's core-system architect is designing disaster preparedness for the case that the production data center is completely knocked out by an earthquake. Interviews on business requirements yield two targets from management: "account balance update data must not lose more than 1 minute (RPO <= 1 minute)" and "a system stoppage must be recovered within at most 4 hours (RTO <= 4 hours)." From the RPO <= 1 minute requirement alone, the architect can judge that a scheme of periodic backups (e.g., daily) could lose up to 24 hours of data just before the disaster, failing to meet the requirement. Keeping RPO within 1 minute requires near-real-time synchronous or semi-synchronous replication of data to a remote DR site. Next, from the RTO <= 4 hours requirement, the architect can judge that a cold standby, where the DR site sits idle in normal times and the environment is built from scratch after the disaster, could exceed 4 hours on environment setup alone, and is insufficient. Keeping RTO within 4 hours requires at least a warm standby configuration, where the DR site's system stays running at all times and recovery is achieved simply by switching over. The typical misjudgment here is assuming "as long as backups are being taken, both RPO and RTO are fine"—because backup frequency governs RPO, while the time needed to recover from a backup (rebuilding the environment) governs RTO, each acting on a separate axis, a configuration must be chosen that satisfies both targets individually (both the replication scheme and the DR site's standby readiness). Also, because keeping a DR site running at all times is costly, for peripheral systems whose business criticality does not demand as strict an RTO/RPO, choosing a cold standby or a looser backup scheme—that is, not uniformly applying the highest-grade DR configuration to every system—is also an important cost-optimization judgment for the architect.

4.3.4Section summary

• Each additional nine of the target availability cuts allowable downtime by roughly a factor of ten and sharply raises cost—avoid over-engineering while leaving no single point of failure
• Choose an HA cluster mode—active-standby (lower cost, slower switchover) or active-active (costlier, near-instant switchover)—based on the business's allowable downtime
• RTO (time axis, time to recovery) and RPO (data axis, acceptable data loss) are separate targets; choose a replication scheme and DR-site readiness level that individually satisfy both