3.5Sampling (statistical / non-statistical)

When a population reaches tens of thousands or millions of items, manually examining every one is often impractical, so auditors use testing (sampling)—examining a subset. What matters is not "sampling = cutting corners" but a judgment based on risk and materiality: which areas testing suffices for, which require a complete examination (every item), and if testing, whether to use a statistical method to infer the population. This section covers testing vs. complete examination, the difference between statistical and non-statistical sampling, and the risk that comes with selecting a sample.

3.5.1Testing and complete examination

• A complete examination examines every item in the population. It has the highest coverage, but is impractical in cost and time when the population is large. If generalized audit software reconciles all production data, an exhaustive examination close to a complete one can be done efficiently even for mass data.
• Testing examines a subset (sample) drawn from the population and infers the state of the whole population from the result. It reduces cost, but if the drawn sample does not correctly represent the population, the conclusion is wrong (sampling risk, below). The higher the risk and materiality of an area, the more the testing scope is widened or brought closer to a complete examination.

3.5.2Statistical and non-statistical sampling

• Statistical sampling draws samples randomly based on probability theory and objectively infers the population's error rate and the like at a certain confidence level from the sample result. Sample-size determination and result evaluation can be justified mathematically, giving high objectivity and explainability to the conclusion.
• Non-statistical sampling draws samples by the auditor's experience and judgment, targeting items expected to be high-risk or material (judgmental sampling). It can efficiently hit problems, but it is weak as a basis for probabilistically inferring the whole population, and the auditor's subjectivity enters the selection.
• Sampling risk is the risk that the drawn sample does not represent the population, so the auditor reaches a conclusion different from the one that examining the whole population would have yielded. It is inherent to testing. It is reduced by increasing the sample size, using a statistical method, or performing a complete examination in high-risk areas. When an exception is found in testing, the possibility that it extends across the whole population is evaluated.

Suppose a systems auditor audits the approval control of an expense-reimbursement system. There are about 50,000 reimbursement requests over the year, so a complete examination by hand of every item is impractical in time and cost. The auditor therefore uses testing. The choice of selection method is the crux of the judgment. If the objective is "objectively evaluate to what degree the approval control was upheld across the whole population (error rate) and produce a conclusion that can be explained in the report," the auditor chooses statistical sampling, mathematically determining the sample size from the required confidence level and tolerable deviation rate and selecting at random. This lets him objectively infer "the population's missed-approval rate is at most X% at a certain confidence level," raising the explainability of the conclusion. On the other hand, if prior risk assessment reveals "signs of fraud in high-value reimbursements or a particular department," non-statistical sampling (judgmental selection) that targets high-value and that department's requests can hit problems more efficiently with limited effort. In practice the two are often combined. In any testing, sampling risk—the risk that the drawn sample happens to contain no problem and misses a missed-approval that exists across the whole population—is inherent, so the auditor increases the sample size or moves toward a complete examination the higher the inherent and control risk of the area. And if even one missed approval is found in testing, he does not dismiss it as "just a single exception" but evaluates the extent to which the same kind of error may extend across the whole population. Thus the auditor's correct judgment in sampling is to choose testing vs. complete examination and statistical methods according to the objective (objective inference vs. targeting high risk) and to risk and materiality, and to evaluate the extension of any found exception to the population.

3.5.3Section summary

• A complete examination examines every item with the highest coverage; testing draws a subset to infer the population, and the higher the risk and materiality, the more the testing scope is widened or brought toward a complete examination
• Statistical sampling infers the population objectively at a certain confidence level via random selection; non-statistical sampling efficiently targets high-risk items by auditor judgment
• Testing carries inherent sampling risk, and a found exception must not be dismissed as "just one" but evaluated for its extension to the population