6.3Testing and quality assurance

The idea that "testing can all be lumped together at the end" does not hold for a level-4 designer. A systems architect is responsible for designing which test at which stage of development (unit, then integration, then whole-system, then requester acceptance) guarantees what, and for judging from data whether testing progress itself is converging normally. Getting the test-process design wrong lets fatal defects surface only after go-live, sending rework cost soaring.

6.3.1Unit, integration, system, and acceptance testing

• Unit testing verifies the smallest unit—a function, class, or module—in isolation. Developers themselves conduct it, and it can closely check internal logic branches and boundary values, but it cannot detect defects in the interaction between modules (such as interface mismatches).
• Integration testing verifies, by combining multiple already unit-tested modules, whether the interfaces between them (data hand-off, call order, etc.) work correctly. System testing verifies, in an environment close to actual operation, whether the whole integrated system satisfies the business requirements and non-functional requirements (performance, availability, etc.).
• Acceptance testing is a final confirmation, led by the users or requester who commissioned the development, of whether the system is complete as required and usable in actual business operation. Unlike the developer-side tests, it differs in who conducts it, and the pass/fail criterion is whether the acceptance criteria agreed to by the requester are met.

6.3.2Performance/load/security testing and the reliability growth curve

• Performance testing confirms, under the expected normal usage conditions and user count, whether the system meets the response time and throughput defined in the requirements. Load testing applies near-limit or beyond-limit load—access spikes or bulk data input beyond what was assumed—to see how far the system can hold up and how it behaves (degrading gracefully or stopping abruptly).
• Security testing uses vulnerability assessments, penetration testing, and similar methods to confirm the system behaves safely against unanticipated input or attacks. Unlike other tests that confirm functional requirements are met, its distinguishing feature is actively confirming that the system does not break or leak data even when used in unanticipated ways.
• Reliability growth curve: a graph, plotted over time, of the cumulative number of defects detected during testing. Ideally, the rate of newly detected defects gradually slows, converging in an S-curve. If defects fail to decline and instead stay high or keep increasing as the deadline approaches, that is a red flag that testing is insufficient and quality has not converged, and it becomes a basis for judgments such as delaying release.

Suppose a systems architect is assessing the quality status of a new members-only e-commerce site shortly before release. Checking the reliability growth curve first, the architect notices that even two weeks before the planned release date, the number of newly detected defects has not declined and remains high—in an ideal S-curve, the detection count should have dropped substantially by this point. From this sign, the architect judges that current testing is insufficient and quality has not converged, and investigates the cause. Scrutinizing the testing status reveals that developer-run unit testing and integration testing combining modules were complete, but system testing, which verifies the whole set of business requirements in an environment close to actual operation, was behind schedule—so the kinds of defects only found at the system-test level (inconsistencies in business flows spanning multiple functions, etc.) had been surfacing in a cluster near the end. It also turns out that load testing simulating an access spike during a sale had not been run, leaving unknown how the system would behave (degrading gracefully or crashing outright) under access beyond what was assumed. Judging that releasing as-is risks a fatal failure during a sale, the architect delays release by two weeks to complete the remaining system-test items and the load test, confirms the reliability growth curve shows a converging trend, and finally revises the plan so that acceptance testing by the requester (the marketing department) confirms usability in actual business operation before making the go decision. Separating out what each test level must guarantee, and judging whether quality has converged using the quantitative data of the reliability growth curve, is what the practice looks like.

6.3.3Section summary

• Test levels ascend unit to integration to system to acceptance, with the conductor and scope changing at each level
• Performance testing = does it meet requirements under normal load, load testing = confirms behavior at the limit, security testing = resilience against unanticipated use
• A reliability growth curve that fails to converge (stays high or keeps rising) even near the deadline is a red flag for immature quality