Treatment and control testing
Choose the control you can prove works
Compare control options by the harm they prevent, the work they add and the evidence you can collect.
CrowdStrike reported that a July 2024 configuration update for its Windows sensor “resulted in a widespread outage”. Its published root-cause analysis provides a starting point for discussing staged rollouts, rollout checks, rollback and customer recovery. Apply the lesson to work your team owns and is authorised to test. CrowdStrike, Channel File 291 incident root-cause analysis
Compare controls before you implement one
For a claims process, compare a second check on every payment with a second check on flagged cases. Checking all payments expands coverage and adds work. A targeted rule reduces routine review and may miss exceptions outside the rule. A risk owner needs evidence on both before choosing. HM Treasury, The Orange Book
Define a pilot using cleared cases and an approved environment. If a policy test reports that a check ran, ask whether it stopped the bad outcome or only recorded a warning. Keep separate evidence for control design, operation and effectiveness.
Include a test that can fail
| Test | Evidence that would count |
|---|---|
| Sample flagged cases | Reviewer decision and source fields |
| Sample unflagged cases | Independently checked missed exceptions |
| Test an unavailable approver | Decision log showing the fallback worked |
| Stop or roll back the pilot | Witnessed restoration and reconciliation |
Measure the extra time to review ordinary cases. Set an acceptable miss rate and maximum delay before the pilot. Treat an unanswered review request as an open case. During a tabletop, an observer should write down who can stop, approve or restart the process and what each person has to see. Use only approved defensive exercises and representative records. NIST SP 800-61 Rev. 3, Incident Response Recommendations
Copy this prompt
Compare the two supplied controls for the stated failure mode. For each, describe who operates it, what it can prevent, likely delays and the evidence we can collect. Draft a test plan with representative ordinary and exception cases, an unavailable approver and a rollback check. Identify who decides whether the control is effective and what evidence is missing. Use only the provided scope and sample data.
Put residual risk in the decision record
Record what the control still misses and who can accept that exposure. A good pilot has a stopping rule, a review owner and a way to compare the old and new process under similar conditions.