Autonomous SOC (AI SOC) evaluation questionnaire

This evaluation is stored in your browser only. We cannot see it, and it is not tied to any account. Save it to a link or create an account to keep it across devices — you can export it at any time either way.

• baselineWhich specific SOC tasks does the AI actually perform end-to-end (alert triage, investigation/enrichment, containment action) versus merely summarizing or recommending for a human to execute — and what proportion of alerts are fully closed without human review today, with a customer reference?weight 12345Remove
• baselineDescribe the model architecture (LLM-based reasoning, deterministic playbooks, or a hybrid) and, critically, what guardrails prevent the AI from taking an incorrect autonomous action (e.g., isolating a critical production server on a false positive) — with a real incident where a guardrail actually stopped a bad action.weight 12345Remove
• baselineHow does the platform handle novel attack patterns outside its training data or existing playbooks — does it degrade gracefully to human escalation, or risk confidently taking a wrong autonomous action on something it hasn't seen before?weight 12345Remove
• baselineWhat is the measured false-positive suppression rate (alerts correctly closed as benign without human review) versus the false-negative rate (real threats incorrectly auto-closed), and how is the false-negative rate measured given it requires knowing about missed threats after the fact?weight 12345Remove
• baselineDetail the audit trail for autonomous actions — is every AI decision (and its reasoning) logged in a way a human analyst or auditor can review after the fact, and can an analyst reverse/undo an autonomous containment action easily?weight 12345Remove
• baselineHow does the platform integrate with existing SIEM/SOAR/ticketing rather than requiring a full rip-and-replace of the existing SOC stack, and can autonomy be dialed up incrementally (e.g., start with recommend-only, graduate to auto-close low-severity, then higher severities) as trust is established?weight 12345Remove
• baselineExplain data handling for the AI models — is customer telemetry used to train shared models across customers, kept fully isolated per-tenant, or processed by a third-party LLM API — and what are the resulting data-residency and confidentiality implications?weight 12345Remove
• baselineWhat is the actual headcount/workload impact reported by reference customers — analyst time reallocated to higher-value work, reduced MTTR, or headcount reduction — with specific, named, verifiable figures rather than vendor-authored case studies alone?weight 12345Remove
• baselineDescribe what happens during a platform outage or model-service degradation — does SOC coverage silently degrade with alerts going unprocessed, or is there a documented failover to human-only operation?weight 12345Remove
• baselineWhat is the pricing model — per alert processed, per analyst seat replaced/augmented, or a flat platform tier — and how does cost compare to the equivalent human-analyst cost it's meant to offset, with a real customer-validated ROI figure?weight 12345Remove
• baselineGiven autonomous containment/response actions could later be scrutinized in an audit or legal proceeding, is every AI decision's reasoning captured in a way that's genuinely defensible and explainable after the fact, or only a high-level action log with no interpretable rationale?weight 12345Remove
• baselineDetail historical trend reporting on autonomy-level progression — how has the proportion of alerts closed autonomously (versus escalated to a human) changed over the deployment lifetime, and is this trend tracked and reported to demonstrate growing trust and program value?weight 12345Remove
• baselineWho within the SOC organization has the authority to grant the AI additional autonomy (escalating from recommend-only to auto-close for higher-severity alert classes), and is there a documented, auditable approval process for expanding autonomous authority?weight 12345Remove
• baselineHow does this platform relate to the customer's existing detection-engineering tooling — does the AI SOC platform author or tune its own detection logic, and if so, is that logic transparent/reviewable by the customer's own detection engineers, or an opaque black box?weight 12345Remove
• baselineHas the platform been red-teamed or adversarially tested specifically to see if an attacker could manipulate the AI into taking an incorrect autonomous action (e.g., crafting an attack designed to trigger a false-negative auto-close, or to trigger a disruptive auto-containment against a legitimate target)?weight 12345Remove
• baselineHow consistent is autonomous-action reliability across multi-cloud/hybrid environments — does the AI have equally reliable context and action capability across on-prem, AWS, Azure, and GCP, or meaningfully weaker autonomous capability in one environment?weight 12345Remove
• baselineWhen the underlying AI model is updated or changes versions, does the vendor communicate this transparently and allow the customer to review/test behavior changes before they take effect in production, or can model updates silently change autonomous behavior with no customer visibility?weight 12345Remove
• baselineContractor shall evaluate and implement emerging tools designated by the organization that utilize AIOps and behavioral analytics to automate incident containment and prevent attacks before they manifest, and support adoption of machine learning-based security analytics to identify patterns indicative of advanced persistent threats (APTs).weight 12345Remove