4.1Performance design and capacity planning

For a system architect, performance design is not the simplistic story of "add more servers and it gets faster." The response time a user experiences and the throughput the whole system can process per unit time are distinct metrics that are often in tension. Furthermore, without understanding a key property of queueing theory—that as the ratio of load to capacity rises, wait time degrades not linearly but sharply (nonlinearly)—an architect can be lulled into false reassurance by reasoning like "utilization is still only 50%, so there is headroom." This section builds the judgment to diagnose bottlenecks from load-test results and estimate the capacity needed, grounded in the response time/throughput distinction and the M/M/1 queueing model.

4.1.1Response time and throughput

• The response time is the interval from sending one request to receiving its reply—a metric the user experiences directly. Throughput is the number of requests the system can process per unit time (e.g., req/sec)—a metric describing the capacity of the system as a whole. The two are not independent: as detailed below, as load (utilization) rises, response time degrades sharply, and throughput eventually plateaus as a result.
• A common mistake in performance-design judgment is to evaluate quality by looking only at the average response time. In practice, architects use percentile values (95th percentile, 99th percentile, etc.) to detect a skewed distribution in which "most requests are fast, but a subset are extremely slow (tail latency)." Looking at the average alone can miss a situation where a minority of users are having a markedly bad experience.

4.1.2The M/M/1 queueing model and utilization

• M/M/1 is a single-server queueing model in which both the inter-arrival interval and the service time follow an exponential distribution (random arrivals, random processing time). Writing the arrival rate as λ (mean arrivals per second) and the service rate as μ (mean requests the server can process per second), utilization (ρ) = λ / μ (with ρ < 1 as the stability condition).
• The mean wait time (mean time in the system) under the M/M/1 model is W = 1 / (mu - lambda). The heart of this formula is that as utilization rho approaches 1, the denominator (mu - lambda) approaches 0, and wait time diverges sharply (nonlinearly). For example, with a server whose mu = 10 requests/sec, at lambda = 8 requests/sec (rho = 0.8), W = 1 / (10 - 8) = 0.5 sec; but raising the load to lambda = 9.5 requests/sec (rho = 0.95) gives W = 1 / (10 - 9.5) = 2 sec, a fourfold degradation—wait time quadruples from just a 15-point rise in utilization.

4.1.3Bottleneck diagnosis and capacity planning

• When a load test shows response time degrading sharply, a system architect identifies from the measurement data which resource (CPU, memory, disk I/O, network, database connection pool, etc.) has a utilization approaching rho = 1. A bottleneck is the resource among several that is the rate-limiting stage of processing capacity (the first one to saturate); reinforcing any resource other than the bottleneck does not improve the throughput of the system as a whole.
• Capacity planning is the activity of forecasting future load growth (more users, more data) and provisioning the necessary processing capacity in advance. Rather than simply matching current peak load, the architect provisions margin that accounts for growth rate, seasonal variation, and sudden spikes (e.g., campaigns), and judges whether to address growth via scale-out (horizontal) or scale-up (vertical) expansion based on the tradeoff between cost and extensibility.

Suppose an e-commerce system architect receives a report that the order API's response time, normally 0.3 seconds, degraded to an average of 2.5 seconds during a sale. Checking the monitoring data first shows the web server tier's CPU utilization sitting comfortably around 60%, but the database connection pool that order processing relies on had reached 95% utilization (rho = 0.95). Applying the M/M/1 intuition here—treating the DB connection pool as a server with a fixed service capacity mu—explains that near rho = 0.95, the denominator of W = 1 / (mu - lambda) approaches nearly zero, so even a small change in lambda (incoming requests) causes wait time to degrade sharply. The misjudgment to avoid here is the fix "add more web servers since CPU has headroom"—the bottleneck is the DB connection pool, and adding web servers would only increase demand on that pool, potentially making things worse. The correct judgment is one of (or a combination of): (1) raising the DB connection pool's limit, (2) improving connection-pooling efficiency by reusing connections, or (3) offloading read queries to a cache or read replica to reduce the load on the connection pool itself. Furthermore, for a predictable spike such as a sale, rather than permanently provisioning resources (raising cost), the design decision to scale out temporarily via autoscaling, keeping normal-time cost down, is also an important angle of capacity planning.

4.1.4Section summary

• Response time (user experience) and throughput (processing capacity) are distinct metrics; look at percentiles (tail latency), not just the average
• Under the M/M/1 model, as utilization rho = lambda / mu approaches 1, wait time W = 1 / (mu - lambda) rises nonlinearly and sharply
• Identify the bottleneck (the rate-limiting stage) through measurement, and judge that reinforcing any other resource will not help. For predictable spikes, balance cost and performance via autoscaling