Security teams keep hearing a new phrase in vendor briefings and conference halls: AIOps for Security. Unlike the older, Gartner-defined AIOps that grew up in IT operations, this label has no standards body behind it and no founding milestone to cite. It is an editorial umbrella — useful, but loose — covering machine-learning and LLM assistance inside security work: anomaly detection and behaviour analytics, alert scoring, model-assisted investigation, and the discipline of evaluating all of it before it acts. Nine questions come up most often. Here they are, answered with numbers rather than adjectives.
What does AIOps for Security actually cover?
The label gathers several practices under one roof: anomaly detection and user-and-entity behaviour analytics (UEBA, an analyst category since roughly 2015), machine-learning alert scoring, LLM-assisted investigation, and software agents that draft or execute triage steps. Just as important is what stays outside it: IT event correlation, rule-based detection engineering, SOC staffing and process, telemetry pipelines, containment playbooks, LLMOps, and vulnerability prioritisation each carry their own names. Market vocabulary overlaps heavily — “AI SOC agents” is a category name Gartner itself now tracks, and its 2026 hype cycle parked such agents at the Peak of Inflated Expectations — while “AI-assisted security operations” means much the same thing. The most useful single overview we have found is this AIOps for Security field guide, which is also where the autonomy ladder below originates.
How is it different from the AIOps we already know?
Classic AIOps correlates IT events to shrink noise; when it misses, the usual damage is a slow incident or an unnecessary page. Security data behaves differently on two counts. First, the source is adversarial and adaptive — attackers probe whatever the detector sees and reshape their behaviour in response, so yesterday’s distributions lie about tomorrow’s traffic. Second, the cost of a miss scales toward breach territory rather than a longer outage queue. We took the IT-operations side on its own in our plain-language tour of classic AIOps; the machinery overlaps, the stakes do not.
What can AIOps for Security safely automate?
Think in autonomy levels, granted per action class rather than per product:
- L0 — observe. The system scores and comments; nothing leaves the dashboard.
- L1 — recommend. Allowed once the model has been scored against at least 200 labelled alerts containing 30 or more confirmed true positives.
- L2 — auto-close. Permitted after a demonstrated miss rate of 2% or lower across at least 150 true positives, with canaries running, a blind holdout set aside, and one-click reopen wired in.
- L3 — reversible action. Demands that wrong actions stay under 1% across a sample of 300 or more would-have-acted verdicts, with a rollback that has actually been rehearsed and rate limits on volume.
- L4 — irreversible. Never granted to AI. Destructive or unrecallable moves stay human, full stop.
A vendor who cannot state which level its agent holds — and the evidence behind it — is asking you to run an experiment blindfolded.
Why does the base rate decide everything?
Because rare positives make precision collapse. Axelsson’s 2000 worked example still teaches best: with attacks arriving at a base rate of 0.00002 and a false-alarm rate of 0.00001, two thirds of alarms turn out real — precision of 66.7%. Loosen the false-alarm rate to 0.001 and precision craters to roughly 1.4%. At modern scale the picture sharpens: on a stream of 50 million events per day carrying 50 genuinely malicious items, a false-alarm rate of 0.0001 produces alerts that are only 0.9% correct, meaning an analyst reads about 112 items to reach each true attack. Push the false-alarm rate down to 0.000001 and precision climbs to 47.4%. Every argument about “accuracy” is really an argument about this arithmetic.
How much evidence is enough?
Less than intuition promises, and more than most audits collect. A spotless run of n cases still leaves the true error rate bounded near 3/n — the old rule of three. Demonstrating a 2% miss ceiling with one miss already observed in the sample demands 236 labelled cases; two misses push the requirement to 313. Random sampling of closed tickets barely helps at all: audit 5% of roughly 1,845 quarterly auto-closes, find zero misses, and the miss rate is still only bounded near 38%. Canary items are the stronger instrument — fifty seeded records per quarter, under a blunt contract: three or more closed by the agent and the auto-close privilege is withdrawn.
Do published benchmarks settle the question?
No, but they frame it honestly. ITBench, unveiled at ICML 2025, left its agents solving only 25.2% of the compliance-and-security set. CyberSOCEval, a 2025 effort from Meta and CrowdStrike, landed around 23–34% on malware analysis and 43–53% on threat-intel reasoning, where outright guessing scores 0.63% and 1.7%. Capable, nowhere near autonomous — and non-transferable besides. The only accuracy figure that matters is the one measured on your own labelled traffic, in your own tenant.
Where do teams get burned?
Four recurring traps. Aggregated scores hide disasters: one deployment reported a comfortable 1% overall miss rate while its cloud-IAM category was missing 42.3% of genuine cases, the number diluted by a 9.1% phish rate next door. “Percent automated” treated as a target rewards volume over safety. Gartner even has a name for inflated accuracy claims — AI washing. And anomaly scores paged on their own grind on-call staff into the ground, because most anomalies resolve into routine changes and new hires, not intruders.
Can it run the SOC without analysts?
No, and the research literature says so plainly. MITRE’s guidance on evaluating AI systems keeps a person in the loop for judging whatever the machinery outputs — a model’s verdict is evidence, never a ruling. Operations researchers saw this coming long before the current wave: back in 2010, Sommer and Paxson had already flagged “a very high cost of errors” as the reason security tooling rarely carried machine learning into production. The healthy pattern pairs scoring engines with the humans who run response workflows — the arrangement we described in SRE Services to Automate Monitoring and Incident Response. The machine clears the repetitive bulk; people keep the residue, which happens to be the part with real consequences.
What should the first quarter look like?
Export last quarter’s dispositions and count the true base rate for two alert categories you actually care about. Shadow-score one candidate model against them using blind labels while the dashboards stay read-only. Publish per-category numbers monthly, and let the gate values — miss ceilings, false-action ceilings — emerge from that record instead of from a slide deck. Keep the categories separate when you report; a single blended score is where problems go to hide. Conversation about automation can wait until the evidence file is thick enough to survive an audit, and it should be written by the people who will carry the pager when the model is wrong.
Regular readers know we like forecasts that get checked against what actually happened. Security automation deserves identical treatment: measure first, grant autonomy slowly, and keep every receipt. The teams that do well with this technology are rarely the ones running the most models — they are the ones holding the most labelled evidence per decision.