3.3Choosing among CAATs (computer-assisted audit techniques)

The more the system is the actor in processing, the more the evidence lives inside the system. CAAT (computer-assisted audit techniques) is a family of techniques for verifying the correctness of computer processing itself, or the entirety of production data. What matters is not the technique names but judging which technique best satisfies evidence sufficiency and appropriateness for the audit objective—"do you want to verify that the processing logic is correct," "exhaustively reconcile all of a large data set," or "continuously monitor a live production system?" This section precisely distinguishes the five techniques and covers choosing among them by situation.

3.3.1The five CAAT techniques

• The test data method feeds test data the auditor prepared in advance into the audited system and checks whether the processing results are as expected. It suits verifying processing logic (program correctness) but only covers the cases the auditor prepared, and cannot exhaustively verify the entire production data set.
• Parallel simulation has the auditor re-process the same input data as production using the auditor's own verification program and compares the output against the production system's output to see whether they agree. It verifies whether the production program's logic is correct by reconciling against the production output.
• The ITF (integrated test facility) sets up a fictitious audit entity (a dummy department, account, etc.) in the live production system and injects dummy audit data into the flow of production processing, checking the results. It verifies continuously without stopping production operation, but requires strict management to keep the dummy data from contaminating real production data and to remove it.
• The embedded audit module builds audit functions into the production program (a mechanism that automatically extracts and records transactions meeting specific conditions) to continuously collect audit evidence while running. It presupposes incorporation from the design/development stage and can capture abnormal transactions in real time.
• Generalized audit software has the auditor extract, aggregate, reconcile, and screen for anomalies across all records of the production data file (database) with a general-purpose tool. It verifies mass transactions exhaustively (every record) and catches exceptions that sampling would miss, making it ideal when you want to verify the accuracy and completeness of the whole population with high sufficiency.

Suppose a systems auditor audits a bank's interest-calculation system. The objective is to verify—with sufficient evidence—whether interest was calculated and posted correctly per the rules for all several million deposit accounts over one year. The auditor first considers the test data method: preparing representative test data for each interest bracket and feeding it in confirms that the interest-calculation program's logic is correct. But the test data method verifies only the cases the auditor prepared and does not guarantee that all several million real records were processed correctly (completeness). Unexpected data that arose in production (negative balances, mid-term cancellations, spanning a rate revision, etc.) would be missed, because such cases are not in the test data. Since the objective is "exhaustive verification of all accounts," the auditor therefore chooses generalized audit software. Extracting all records from the production deposit database, recomputing interest himself per the prescribed rates, and reconciling one by one against the interest amounts the system posted verifies the accuracy and completeness of the whole population with high sufficiency, and catches exceptional errors that sampling might miss. If instead the objective were "verify that a newly released interest-calculation program's logic is correct before it goes live," parallel simulation (re-process with the auditor's verification program and reconcile) or the test data method would fit; and if the objective were "continuously monitor daily processing in the live system," ITF or an embedded audit module would fit. The crux of CAAT selection is thus to choose the technique against the objective (logic verification vs. exhaustive coverage vs. continuous monitoring) and evidence sufficiency. It is important not to err in the judgment that exhaustive verification of mass transactions calls for generalized audit software, not the test data method.

3.3.2Section summary

• The test data method verifies processing logic with auditor-prepared data; parallel simulation re-processes with the auditor's program and reconciles against production output
• ITF injects dummy audit data into production; an embedded audit module builds audit functions into the production program to collect evidence continuously
• For exhaustive verification of mass transactions, choosing generalized audit software that reconciles all production data—not the test data method—is correct from a sufficiency standpoint