Data science and assessment
Separate rising risk from rising volume
Build a baseline, test it on later data and measure how much work an alert creates.
A branch with 25 exceptions in 500 transactions has a 5% exception rate. Another has 40 in 2,000, or 2%. Raw counts send you to the second branch. Rates tell you to ask why the first branch has a larger share of exceptions before you decide what to fix.
Set up the denominator
Define what qualifies as a transaction, when it enters the count, and what constitutes a unique exception. Check for missing branches, duplicates and classification changes. Compare equivalent services and time windows. Ask the data owner whether the apparent change could come from a new channel or a different product mix. The AQuA Book calls for analysis that is “fit for purpose”. For an alert, that purpose includes the team’s capacity to review it. Government Analysis Function, The AQuA Book
Start with a transparent baseline such as a threshold on a comparable historical rate. Give AI the approved schema and ask it to draft a query and edge-case tests. An analyst should inspect the calculation and reconcile the counts before anyone uses the rate to allocate review work.
Back-test the alert with later data
Keep the test period later than the development period. If a field only becomes known after the alert would fire, exclude it from the model. Have reviewers verify a sample of both flagged and unflagged cases. This reveals whether the model misses problems that the team never investigates. Government Analysis Function, The AQuA Book
| Practice check | Result | Meaning |
|---|---|---|
| 40 alerts, 28 verified exceptions | 70% precision | 28 of 40 alerts were valid. |
| 40 actual exceptions, 28 detected | 70% recall | The method missed 12 actual exceptions. |
| 40 alerts at 10 minutes each | 400 review minutes | The team needs about 6 hours 40 minutes. |
These figures are teaching inputs. Decide the review budget and the cost of a miss before you tune a threshold. Keep results by branch if uneven review capacity could hide missed cases.
Copy this prompt
Review these approved schema definitions and aggregate event totals. Draft pseudocode or SQL that counts distinct eligible transactions and distinct eligible transactions with exceptions by branch and period. Include reconciliation queries and tests for duplicates, missing data, denominator changes and information that arrives after scoring. Propose a later-period back-test against our existing rule, with precision, recall and minutes of review. Flag assumptions for the analyst.
Make the comparison a pilot
Compare the proposed rule with the current process on the same cases. Ask the risk owner what would justify adopting it. Write down the alert volume the team can handle and which type of exception it has to catch.