Skip to content
Adrian DunkleyAI Safety and Alignment

National Agentic Risk Matrix

Before you give an AI agent permission to act, work out what could go wrong and how long it would take to put things right.

A truck waits at the port in one of the scenarios below. Customs received the broker’s filing during an outage and sent no receipt. The broker’s AI agent tried again when service returned. The officer sees several copies of the same filing and has to work out which one to clear.

The delay we’re rating starts when retries clog the queue. To place it on the grid, check who can resubmit and how fast Customs can clear the backlog.

Three-by-three chart of national agentic risk. Likelihood rises left to right and severity rises bottom to top. Each cell names the threat category and the main factors behind the placement. A table in the article lists all six factor ratings.
Download the matrix (PNG)

Start with what the agent can do

Use one failure per row. Identify who could be harmed and the action that would make it happen. Record what has to go wrong first, what the agent has permission to do and which check could interrupt the chain.

I rated nine hypothetical cases over 12 months for a small island state. The ratings are provisional. Change them if your agents have different permissions or your teams can recover sooner than the case assumes. The cited incidents show comparable failures. The Guide for Conducting Risk Assessments separates the chance of a threat event from the harm it can cause. The National Risk Register uses its own thresholds. The rules below apply to this matrix.

Four ways an agent can contribute to harm

An action error changes a system or moves money incorrectly. A decision error closes a case or changes eligibility on the wrong evidence. A service dependency lets one interruption spread when workflows depend on the same system or restart process. A public information failure sends people wrong or unapproved instructions. Use the category that best explains how the harm starts. Add a second category if it calls for a different safeguard. Mark whether the agent starts the failure or makes it worse.

The six checks behind each rating

Rate each factor low, medium or high. Check outage logs, agent permissions and recovery-test results. Where evidence is missing, mark the factor unknown and say what evidence would let you rate it.

Use the same scale for all nine scenarios.
FactorLowMediumHigh
ExposureSeveral unusual conditions have to coincide.An occasional event creates the opportunity.Routine work or frequent outages create repeated opportunities.
Agent authorityThe agent can advise. Someone else must act.The agent can act within limits or with approval.The agent can act at scale, or critical work depends directly on it.
Control gapAn independent safeguard has passed a test for this kind of failure.A safeguard exists and can fail in these conditions.No effective safeguard can stop this failure in time.
ReachOne site or district.Several districts or one national service.Widespread national disruption or several essential services.
HarmTemporary lost time or inconvenience.Material delays, missed payments or disrupted supplies.Serious harm, immediate danger or prolonged loss of an essential service.
Recovery difficultyA prompt correction or routine recovery restores service.Teams must coordinate to clear the backlog or restore service.No timely fallback before a critical deadline, or the harm is irreversible.

What puts a threat in its cell

Likelihood takes the lowest rating among exposure, agent authority and control gap. A failure needs an opening, a way for the agent to influence the outcome and a safeguard that can fail. Score other ways the harm could happen separately and use the highest rating to decide what to test first. For example, staff may routinely approve a recommendation without checking it. Treat the ratings as priorities for testing.

Severity is high when harm is high, or when both reach and recovery difficulty are high. Otherwise, severity is medium if harm or recovery difficulty is medium or high. It is low in the remaining cases. A false notice can reach the whole country and still rate low if a quick correction limits the harm to lost time.

At the port, outages create occasional chances for a backlog, so exposure is medium. Brokers’ agents can resubmit at scale and there is no effective duplicate check after restart. Authority and control gap are both high. The lowest of those three ratings gives medium likelihood. Food and medicine deliveries fall behind across several districts. Clearing the backlog takes several teams, so the harm and recovery difficulty both rate medium. The port goes in the centre cell.

Nine threats and the factors behind their ratings

Ratings apply to the conditions described in each scenario.
ScenarioThreat categoryExposureAgent authorityControl gapReachHarmRecovery difficultyLikelihood × severity
A faulty update cuts powerAction errorLowHighMediumHighHighMediumLow × High
Crews miss a rescue callDecision errorMediumHighHighMediumHighHighMedium × High
Referrals and payments stallService dependencyHighHighHighHighHighHighHigh × High
Relief paid twiceAction errorLowHighHighMediumMediumMediumLow × Medium
Repeat filings delay cargoService dependencyMediumHighHighMediumMediumMediumMedium × Medium
A blank field stops benefitsDecision errorHighHighHighHighMediumMediumHigh × Medium
A false school closure noticePublic informationLowHighMediumHighLowLowLow × Low
Voters get the old polling sitePublic informationMediumHighHighMediumLowLowMedium × Low
Applicants make two tripsPublic informationHighHighHighHighLowLowHigh × Low

A bad grid rollout and a missed rescue

A faulty update cuts power

A dispatcher loses contact with several sites during maintenance. The AI agent had permission to install a class of upgrades across the grid. Before the utility can stop the rollout, a faulty update has reached several sites. Other services switch to backup power while the dispatcher works to restore the grid.

Low likelihood · high severity · agent triggers the event. A bad update can cut power countrywide when the agent can change live systems and the utility loses the chance to stop the rollout. Office of Utilities Regulation report on the April 2016 outage

The utility can let the agent prepare changes and keep live installation under operator control. Test upgrades away from the live grid and require independent approval before rollout. The team should be able to pause between stages and restore the previous configuration even if the deployment system goes down. Operational technology security guidance

Crews miss a rescue call

Two families call from different flooded crossings with the same place name. The agent marks the reports as duplicates and closes one. The dispatcher sends a crew to the only location still on the list.

Medium likelihood · high severity · agent amplifies the emergency. A major storm fills the queue. In this scenario, the agent can close a report before a dispatcher verifies it. Crews can miss people who need urgent help wherever the same rule applies.

The emergency operations centre can keep every report visible and let the agent suggest matches for dispatchers to check. Keep radio and paper intake available when networks fail, and ask crews to confirm which locations they reached. In a drill, count how many open calls never reached a crew and how long they went unseen. UN OCHA data responsibility guidelines

When referrals and benefits stop

Referrals and payments stall

A hospital clerk can open a patient record but can’t send an urgent referral. Staff at a payments desk can see transactions waiting for release. The hospital and payments desk bought different applications. Their agents depend on the same platform, which has been dropping out. Staff take over by hand and miss the referral and payment deadlines.

High likelihood · high severity · agents amplify the failure. The platform fails frequently in this scenario. Each outage can put urgent referrals and payments back in the manual queue. Treasury under secretary Nellie Liang warned in 2024 that AI may “increase reliance on a relatively small number of cloud service providers.” Financial Stability Board remarks

One supplier’s software update affected many organisations at once in 2024. CrowdStrike’s incident analysis

The national service continuity lead can map which agencies share a model provider, sign-in service or network. Simulate an outage at peak demand across those agencies. Count the urgent requests staff can still handle and check whether the backup provider depends on the service that failed. FSB third-party risk toolkit

A blank field stops benefits

A woman brings the letter she’s used for years to prove she qualifies for benefits. A recent data import left a field empty. The agent treated the blank as a failed eligibility check and stopped her payment. Other district offices use the same system. Households across the country miss a payment cycle before the agency sees the pattern.

High likelihood · medium severity · agent triggers the event. Incomplete records are common in this scenario, and the agent can suspend payments without review. Back payments and appeals can restore the money. If people go without essentials for a prolonged period, raise the severity.

The benefits agency can route incomplete records to staff for review and keep paying while staff resolve the discrepancy, where programme rules permit. A written reason should tell the recipient how to reach someone who can reverse the decision. Monitor suspensions by district and pause the rule if they spike. The Robodebt inquiry recommended clearer review routes after investigating an automated welfare debt scheme. Robodebt Royal Commission report

When the agent tries again

Relief paid twice

A treasury officer sees that a disaster relief payment batch timed out. The agent has already tried again. The bank’s records show that both batches went through. Some families received two payments. The next batch is on hold while the treasury works out what it can recover.

Low likelihood · medium severity · agent triggers the event. A significant storm prompts a large payout. The bank’s payment status has to be unclear, the agent has to be able to retry and the payment system has to miss the duplicate. The treasury’s recovery work delays the next group of households. Government Accountability Office report on disaster assistance

Set the payment system to pay each approved identifier once, even if the agent retries. The treasury can require staff to check the bank’s settlement record before releasing a payment with unclear status. Set a limit on the whole batch and require a second approver for exceptions. Interrupt a test batch and check that resuming it pays each household once.

Repeat filings delay cargo

The customs officer can hold the trucks until she reconciles the duplicate filings. She can also clear priority cargo by hand and check those records against the system later.

Medium likelihood · medium severity · agents amplify the outage. If many brokers’ agents start resubmitting when Customs brings the system back, the filings can pile up at the main port. A backlog that lasts days can delay food and medicine deliveries.

Customs can assign each filing an identifier that blocks repeat submissions without blocking valid amendments. Slow retries during recovery and keep a manual route for priority cargo. In an exercise, time how long it takes to clear perishables and critical stock. Customs continuity process maps

A false school notice and an old polling address

A false school closure notice

A parent reads in the school group that schools are closed nationwide. An agent sent a draft that the ministry hadn’t approved. The correction arrives after buses have left and some parents have rearranged shifts.

Low likelihood · low severity · agent triggers the event. The draft has to be marked ready by mistake, an independent release check has to fail and the agent must have publishing access. A quick correction limits the disruption to time lost.

Require an authorised officer to release notices and a second person to check nationwide closures. The ministry can use the same channels, from school groups to radio, to correct a false notice. Exercise the correction and measure when it reaches families. An investigation into a false emergency alert documented a delay in issuing the public correction. FCC investigation

Voters get the old polling site

A voter reaches the school named in a text from an assistant and finds the gate shut. The election authority moved the polling site. The assistant kept sending people to the old address. Election workers spend part of the morning redirecting voters across several districts. The correction reaches them in time to vote.

Medium likelihood · low severity · agents amplify outdated information. The scenario assumes an election within the year and a late venue change that the assistants miss. Voters make an extra trip and election staff have to redirect them. If people lose a meaningful chance to vote, the severity rises.

The election authority can publish a dated lookup that assistants consult for each request. If an assistant can’t verify an address, it can direct the voter to the election authority. Test how quickly call centres, broadcasters and assistants pick up a correction. Electoral Commission voting procedures

The cost of an out-of-date checklist

Applicants make two trips

An applicant brings the checklist the national renewal agent sent him. The clerk has to turn him away for a document the checklist misses. By lunchtime, applicants with the right papers are waiting behind people who came back a second time. Offices across the country use the same checklist.

High likelihood · low severity · agent triggers the repeat visits. Frequent changes to the requirements, heavy use of the assistant and slow updates make repeat trips likely. Offices can still process applications, and people can return without losing access or eligibility.

The service owner can give the agent one current source for requirements and display the date on each reply. Track how many people have to come back after a change. If the agent sends out old instructions, take the affected answer offline and put the corrected checklist where people first asked for it. Government service standard on keeping channels aligned

Rehearse an outage and time the recovery

Ask the service owner and frontline staff to rehearse one failure from the matrix. Time how long it takes to spot the problem and suspend the agent. Simulate losing the usual system, then count how many cases staff can still handle. Give those results to the person who decides whether the agent can keep operating.