AI incident response: what to do when AI goes wrong

AI fails differently

Traditional incident response assumes systems that are down or breached. AI adds failure modes that are subtler and often invisible: a model that quietly drifts, a confident but wrong answer acted upon, a biased outcome, or confidential data leaking into a prompt. You need a response process built for these.

What counts as an AI incident

Treat any of the following as reportable:

• Inaccurate or harmful output that was, or could have been, acted upon.
• Privacy exposure — personal or confidential data entered into, or revealed by, an AI tool.
• Bias or discriminatory outcomes affecting individuals or groups.
• Security issues — compromise, misuse, or a vulnerability in an AI system.
• Unauthorized use — shadow AI operating outside policy.
• Unexpected behavior — a system operating materially outside its documented purpose.

The response lifecycle

1. Detect & report. Give people an easy way to raise an incident, and watch error monitoring and model outputs.
2. Triage. Assign a severity and an owner; scope the impact and who's affected.
3. Contain. Limit harm — pause the use case, revoke access, or add a human checkpoint.
4. Investigate. Find the root cause and the extent of impact; preserve logs and evidence.
5. Remediate & recover. Fix the cause, restore correct operation, and add or strengthen safeguards.
6. Notify. Inform affected stakeholders, and where required, customers and regulators, consistent with contracts and law.

Severity, at a glance

• Critical — confirmed harm to individuals, a data breach, or legal exposure.
• High — likely harm or a significant control failure.
• Medium — limited-impact issue needing correction.
• Low — minor issue or near-miss; log and address routinely.

Learn, don't just close

For higher-severity incidents, run a blameless post-incident review: timeline, root cause, impact, and corrective actions with owners and dates. Feed the lessons back into your risk rules and safeguards, and review incident trends to catch systemic issues.

Make it part of the program

Incident response works best when it's connected to your AI use register — so an incident links to the specific tool and use case, and remediation strengthens that use case's safeguards. AI Assurance Hub captures incidents, links them to use cases, and keeps the audit trail. See a live demo.