An uncontrolled acceptance of AI.

Before you expand an AI system's authority, estimate the financial exposure before someone can stop it. Then test the controls against the people and decisions they are meant to protect.

My work on AI adoption, staff training and return on investment keeps bringing me back to a question: what exactly are we accepting when we accept an AI answer? A useful draft, a recommendation and permission to act are different things. I want this series to examine the movement between them.

For this opening issue, take one workflow you are considering automating. Write down its current permissions. Then describe the worst action it could take within those permissions. That gap between a helpful answer and an authorised action is where I would begin the safety discussion.

The objective can be wrong even when the answer looks right

Here, I use alignment to mean whether an AI system's behaviour serves the intended purpose while respecting the people affected and the constraints agreed for its use. Safety asks what harm the system could cause, including when it follows its instructions. These are working definitions for organisational decisions. They leave open the much wider research problem of aligning advanced AI.

Consider an illustrative customer-service assistant rewarded for closing cases quickly. It can improve the closure count by discouraging difficult complaints. The metric improves while the service deteriorates. Changing the wording of its replies will not fix an incentive that rewards the wrong outcome. Amodei and colleagues' Concrete Problems in AI Safety distinguishes problems involving objectives, supervision and behaviour outside familiar conditions. That distinction helps identify which part of the workflow needs repair.

Acceptance becomes uncontrolled when a satisfactory demonstration is treated as evidence for a broader permission. A model that drafts a sound response has not thereby demonstrated that it should send messages, alter records or settle disputes. Each added permission changes the possible consequences and needs its own evidence.

Borrow a failure from automated trading

The Knight Capital incident is a useful warning about automation, although it was not a generative-AI incident. The US Securities and Exchange Commission reported that erroneous automated trading in August 2012 left the firm with losses exceeding US$460 million. Its findings included inadequate deployment controls and safeguards against excessive exposure. The SEC account matters here because it connects speed, authority and weak containment.

The lesson I take for AI is to test the path from output to consequence. If an agent can issue instructions faster than your team can inspect them, the inspection process needs an enforceable limit on what happens while people investigate. An alert with no effective response is a record of danger, not containment.

This does not establish that a language model will fail in the same way. SEC enforcement official Daniel M. Hawke asked what safety nets are in place to limit the harm. That identifies a control question that survives the change in technology: how much can the system commit before the organisation regains control?

My proposed 4S cycle makes the decision inspectable

I propose the Dunkley 4S Safety and Alignment Cycle: Specify, Stress-test, Safeguard and Scrutinise. The contribution is a practical teaching and decision structure. Its underlying ideas draw on established risk management and safety engineering; I am not claiming to have invented those disciplines or validated a new safety standard.

NIST's AI Risk Management Framework 1.0 organises its core around Govern, Map, Measure and Manage. My cycle is a reader-facing application alongside that work, not an equivalent replacement. Governance and consultation apply throughout all four stages.

The four Ds of AI fluency were developed by Rick Dakan and Joseph Feller, who partnered with Anthropic on the course. Fluency concerns how people work with AI. This proposed 4S cycle concentrates on the evidence and authority surrounding a particular deployment.

Measure exposure before arguing about a safety score

Start with an intentionally simple stress scenario. Let a be consequential actions per minute, t the minutes until effective containment and c the maximum direct commitment per action. An illustrative exposure envelope is E = a × t × c. The units are currency. It assumes a constant action rate, maximum commitment on every action and no recovery during that interval.

If an assistant can authorise 12 payments a minute, each capped at US$500, and containment takes 10 minutes, the envelope is US$60,000. Reducing effective containment to two minutes makes it US$12,000. These are hypothetical commitments at risk, not observed losses, expected losses or probabilities. Recovery, fraud, knock-on effects and legal consequences could change the eventual loss substantially.

A separately enforced aggregate cap can limit the direct commitment in this scenario. If that cap is US$5,000 across all relevant accounts and workers, the model takes the smaller of the calculated envelope and the cap. A cap applied independently to several agents does not automatically create a shared limit. Test its scope, concurrent actions and behaviour when the cap is reached.

The accompanying calculator lets you vary these assumptions and export the result. It also calculates an annual expected direct loss for a different, explicitly stated model: actions per year × assumed harmful-action probability × average direct loss per harmful action. Keep the annual estimate separate from the short-interval envelope. An assumed probability is not evidence about your deployment.

I would use a range of assumptions to decide what deserves further testing. I would also set non-negotiable constraints on unacceptable harm. A small estimated average financial loss cannot settle questions about discrimination, privacy or an irreversible decision affecting a person.

Ask who can challenge the decision

Quantification needs a qualitative challenge. Ask someone who does the work to describe a failure the dashboard would miss. Ask an affected person what a fair appeal would require. Then record disagreement in the decision log, including the evidence that would change each person's view.

A useful implementation might keep an assistant drafting routine replies while staff retain sending authority. A second workflow might permit tightly capped actions after independent approval. The right boundary depends on the task and evidence. Treat both as design options to evaluate, rather than assuming that greater autonomy is the natural destination.

My contrarian position is that a restriction can help adoption when it lets people attempt a useful task without accepting the full consequences of an unrestricted system. That proposition needs testing too: compare review burden, task completion, missed harms and staff willingness to report problems. A control that merely drives work into an unapproved tool can make observation worse.

Choose one workflow today. Complete the canvas with its owner and a colleague who can challenge the proposed objective. Test the stop mechanism using non-sensitive examples. Leave consequential actions disabled if the owner, permission boundary or recovery route remains unresolved.

This issue builds on assigning an owner to AI exceptions and testing questions a demonstration would avoid. It brings those checks into a permission decision with a quantified exposure envelope. The next issue, Lessons from Colonization: AI Misalignment., will examine who sets the objective, who receives the benefit and who retains the ability to refuse.

Sources checked 1 November 2026. The numerical examples are hypothetical and the 4S cycle is a proposed educational synthesis. AI assisted the preparation and editorial review of this article.