Adrian DunkleyAI Risk Management

Data science and assessment

Separate rising risk from rising volume

Build a baseline, test it on later data and measure how much work an alert creates.

A branch with 25 exceptions in 500 transactions has a 5% exception rate. Another has 40 in 2,000, or 2%. Raw counts send you to the second branch. Rates tell you to ask why the first branch has a larger share of exceptions before you decide what to fix.

Set up the denominator

Define what qualifies as a transaction, when it enters the count, and what constitutes a unique exception. Check for missing branches, duplicates and classification changes. Compare equivalent services and time windows. Ask the data owner whether the apparent change could come from a new channel or a different product mix. The AQuA Book calls for analysis that is “fit for purpose”. For an alert, that purpose includes the team’s capacity to review it. Government Analysis Function, The AQuA Book

Start with a transparent baseline such as a threshold on a comparable historical rate. Give AI the approved schema and ask it to draft a query and edge-case tests. An analyst should inspect the calculation and reconcile the counts before anyone uses the rate to allocate review work.

Back-test the alert with later data

Keep the test period later than the development period. If a field only becomes known after the alert would fire, exclude it from the model. Have reviewers verify a sample of both flagged and unflagged cases. This reveals whether the model misses problems that the team never investigates. Government Analysis Function, The AQuA Book

Practice checkResultMeaning
40 alerts, 28 verified exceptions70% precision28 of 40 alerts were valid.
40 actual exceptions, 28 detected70% recallThe method missed 12 actual exceptions.
40 alerts at 10 minutes each400 review minutesThe team needs about 6 hours 40 minutes.

These figures are teaching inputs. Decide the review budget and the cost of a miss before you tune a threshold. Keep results by branch if uneven review capacity could hide missed cases.

Copy this prompt

Review these approved schema definitions and aggregate event totals. Draft pseudocode or SQL that counts distinct eligible transactions and distinct eligible transactions with exceptions by branch and period. Include reconciliation queries and tests for duplicates, missing data, denominator changes and information that arrives after scoring. Propose a later-period back-test against our existing rule, with precision, recall and minutes of review. Flag assumptions for the analyst.

Make the comparison a pilot

Compare the proposed rule with the current process on the same cases. Ask the risk owner what would justify adopting it. Write down the alert volume the team can handle and which type of exception it has to catch.

Sources and references