Before any alert-closing agent earns authority in your estate, it should survive an audit. The discipline now gathered under the name AIOps for Security lends itself to exactly that: its claims are graded, its gates are numeric, and its failure patterns are documented well enough to check against. What follows is a five-part checklist you can run against any vendor product or internal model, with the specific thresholds and traps that recur across the field, from gate arithmetic to data-handling defaults that quietly differ between tenants. Print it, schedule it quarterly, assign an owner, and refuse exceptions without written justification.
AIOps for Security audit, part one: scope and claims
- Inventory the action classes. List everything the system may do — score, recommend, close, act — and record each separately, because authority is granted per class and never per product. Clearance earned on the phishing queue confers no standing over identity events, and the inventory makes that boundary visible before anyone assumes otherwise.
- Test the observe claim. Ask whether a genuine observe-only mode exists, then verify it in configuration rather than in the sales deck. In 2026 the honest answer for several shipping products is no: Microsoft’s Phishing Triage Agent begins auto-closing the moment it runs, with no documented observe setting, so tenants wanting a silent phase have to engineer one themselves.
- Retire the vanity metric. “Percent automated” measures enthusiasm, not value. Replace it with verified workload removed at a measured error cost — the framing that one published triage evaluation put at 93.5% of analyst toil, which is a number you can negotiate against.
- Check public benchmarks. Against ITBench’s compliance-and-security scenarios agents resolved 25.2%; the CyberSOCEval runs scored roughly a fifth to a third of malware-analysis items and about half of threat-intel questions. Claims that sit far above that band deserve source documents, not slides.
Part two: the AIOps for Security evidence gates
- Recommendation floor. Confirm at least 200 labelled alerts and 30-plus true positives back any claim to advise, with the labelling done by people who do not benefit from the answer. Below that floor, the system is guessing and should be described as guessing in every document it appears in.
- Auto-close gate. Require a miss ceiling of 2% demonstrated over 150-plus true positives, a scored blind holdout, planted canaries already running, and reopen in one click. If any element is missing, the agent stays at recommendation until it is not.
- Reversible action gate. Suspending sessions or quarantining mailboxes requires false actions held to 1% or lower across at least 300 opportunities to act, rollback rehearsed end to end, and volume throttled so a bad day cannot become a bad estate.
- Irreversible refusal. Verify in configuration, not conversation, that deletion and permanent lockouts remain human-only. This line does not move for scores, savings, or executive impatience.
Part three: measurement that outlives go-live
- Per-category reporting. One blended miss rate has already hidden 42.3% cloud-IAM failures behind a 9.1% phishing figure, and nobody found it until somebody asked for splits. Demand them monthly, or accept that the reporting exists as decoration.
- Published prevalence. Every accuracy figure must travel with the base rate of the population it was measured on; without prevalence, precision has no meaning, and demo populations curated for balance will flatter anything.
- Canaries, not spot-checks. Fifty planted cases per quarter, mixed into live traffic so no one can route around them, with the standing rule that three or more wrongly closed withdraws the autonomy tier on the spot. Random sampling fails here for arithmetic reasons: a five percent audit of 1,845 quarterly closures that finds zero errors still bounds the miss rate near 38%, which is assurance in name only.
- Drift monitors. Attacker behaviour shifts; a gate proven last quarter speaks about last quarter, and nobody notices the expiry date until something is missed. Watch inputs and outputs, and re-run every gate when the model version changes, because a swap is a new claim that has earned nothing yet.
Part four: exposure and data handling
- Outbound channels. Any agent that fetches links, uploads samples, or replies to requesters reads attacker-controlled content and can be steered by it. Prompt injection is not an input-filtering defect with a patch on the way — plan for deterministic guards and least privilege, because cleverer wording is not a mitigation, and deny-lists lose to infinite rephrasing.
- Sample leakage. Uploading suspicious files to public scanners can broadcast their contents beyond your tenant; the standard services warn plainly that submissions may reach their paying customers. Classify every artefact before it leaves, and prefer in-tenant detonation where the estate allows it.
- Data residency defaults. Check the sharing switches by name and position: Security Copilot’s customer-data sharing switch starts in the on position without an M365 E5/E7 agreement, and Gemini in Security Operations processes on global endpoints — without training on your prompts, which is a guarantee about model improvement, not about where bytes travel.
Part five: governance and drift of a different kind
- Keep a signed register. One row per action class: tier, evidence, dates, owner, next review. Auditors and insurers ask for this document eventually; vendors also change behaviour between contract and deployment, and the register is what catches them.
- Preserve the human duty. The evaluation chains that matter most — MITRE’s guidance among them — keep a person responsible for judging the machinery’s output, and no configuration option ever relieves them of it. Document who holds that role per class, and what they are expected to review weekly rather than “periodically”.
- Track platform deadlines. Sentinel’s Fusion engine yields to Defender XDR correlation with every workspace migrating by 31 March 2027. Correlation logic changes during migrations; gates should be re-verified afterwards, not assumed to survive the move intact.
- Watch for transfer myths. Vendor accuracy does not carry across estates, and scores published near perfection have sagged repeatedly once evaluation bias was squeezed out of them. Your labels are the only ones that count, and your prevalence is the only one that contextualises them. Budget for producing those labels; it is the one cost in this programme that never gets cut usefully.
Running the audit without stalling the rollout
A checklist this blunt can freeze a programme, so stage it deliberately. Items in parts one and two gate any go-live decision; part three begins the week the agent starts observing, not the week it starts acting; parts four and five are continuous obligations, owned by name, reviewed at a fixed cadence that survives reorganisations. Teams that already operate this way — the habits built in progression from junior to senior DevOps work — will recognise the shape from data reliability practice, and the governance rhythm matches cloud, DevOps and reliability engineering as it is actually run: small checks, often, by someone accountable for them. The reference behind every threshold above — the full ladder, the sampling arithmetic, the worked cases — is the Ops Atlas reference titled AIOps for Security, and it updates often enough to warrant a full re-read at the start of every audit cycle. Audit once, and the second audit writes itself; skip the first, and the incident report writes it for you.