Start from a neutral baseline and add what matters to you. Criteria are labeled by source — the platform baseline is architecture-neutral; buyer-contributed criteria are shown separately.
Build your evaluation
no account needed
Match on your requirements
no account needed
This evaluation is stored in your browser only. We cannot see it, and it is not tied to any account. Save it to a link or create an account to keep it across devices — you can export it at any time either way.
1. Weight what matters
28 criteria
baselineHow many pre-built integrations/connectors does the platform ship with for common security tools (SIEM, EDR, firewall, ticketing, identity), and how is a custom integration built for a tool without an out-of-box connector?
baselineDescribe playbook authoring — is it a visual/no-code builder, code-based (Python), or both — and provide a customer reference for time-to-build and time-saved on a specific common playbook (e.g., phishing triage).
baselineWhat percentage of alerts can realistically be fully auto-closed without analyst involvement for common use cases (e.g., known-benign phishing reports), and what safeguards exist to prevent an incorrect auto-close of a true positive?
baselineHow does the platform handle playbook version control, testing, and rollback — can a playbook be tested in a dry-run/simulation mode before being enabled in production?
baselineDetail case management capabilities — does the platform provide native incident case management (timeline, evidence, collaboration) or does it only orchestrate actions across other tools that retain the case record?
baselineWhat metrics/reporting does the platform provide on automation ROI — mean-time-to-respond (MTTR) reduction, analyst hours saved — and is this measured automatically or does it require manual tracking?
baselineExplain how the platform handles multi-tenant or MSSP use cases — can playbooks and integrations be shared across tenants while keeping tenant data and credentials isolated?
baselineHow are third-party API credentials and secrets used by playbooks stored and rotated — is there a built-in secrets vault, or does it rely on the customer's own secrets management?
baselineWhat is the pricing model — per playbook, per analyst seat, per event/alert volume processed — and how does cost scale as alert volume grows significantly (e.g., after onboarding a new high-volume data source)?
baselineDoes the platform offer AI/LLM-assisted playbook generation (describing a desired workflow in natural language and having a draft playbook generated), and how much manual refinement is typically needed before that draft is production-ready?
baselineHow does the platform support cross-team workflows beyond the security team itself — e.g., automatically routing a data-privacy incident to legal, or a compliance finding to the GRC team — with appropriate access scoping for non-security stakeholders?
baselineWhat happens to in-flight automated incident response if the SOAR platform itself has an outage mid-incident — is there a documented fallback/manual-continuation process, and has this actually been tested?
baselineDetail the availability of a staging/testing environment separate from production for validating new playbooks and integrations before they run against live security data and can take real automated actions.
baselineIs there a community or marketplace for sharing pre-built playbooks and integrations across customers, and how vetted/maintained is that content versus being unmaintained community contributions?
baselineExplain the audit trail for automated actions — is every action a playbook takes (e.g., disabling a user account, blocking an IP) logged with full context for compliance/forensic review, and can this audit trail be exported to an external SIEM or GRC system?
baselineWhat is the typical learning curve for a new analyst to become proficient building and maintaining playbooks — is formal training included, and what is a customer-referenced timeline from onboarding to a team member independently authoring production playbooks?
baselineIntegrate with key components of the existing IT ecosystem, including Heimdal EDR, Cisco Meraki, Microsoft 365, Azure AD, and on-prem Active Directory.
baselineProvide robust correlation, investigation, and automation workflows to reduce response time and improve analyst efficiency.
baselineDevelop and deploy automated response playbooks using Logic Apps or other Microsoft-native tools, aligned with pre-approved containment actions and escalation paths (Playbook and Automation Setup).
baselineShould support automated response actions (e.g., isolating compromised endpoints, triggering firewall rules, revoking user access).
baselineVendors should provide pricing for different deployment models and include cost breakdowns for licensing (per user, per endpoint, per GB, etc.), support and maintenance, and additional features (e.g., SOAR, extended retention, custom integrations).
baselineAlerting and Incident Integration: configure alert notifications, integrate with ITSM/SOAR platforms (e.g., in-house ticketing, Jira, Zoho), and configure incident workflows and escalation.
baselineBidder to provide workflow automation so that applications and infrastructure are integrated automatically, with minimal manual intervention.
baselineThe vendor should bring workflows and solutions that can automate the majority of incident response activities such as false positive management, managing whitelists, escalation workflow, and SLA management; alerts should be notified to the financial institution only after a proper triage process and enriched with context data, environmental data, vulnerability data, historical data, and threat intelligence.
baselineSolution should support centralized incident management and triaging of alerts from multiple security products (SIEM, DLP, IPS, WAF, Anti-APT, ETDR); investigation module should integrate with log sources (SIEM, ETDR, EPP, Data Lake) on demand to pull data related to the investigated alert with charting and graphing to analyse data.
baselineSolution should support full workflow for incident classification, coordination, escalation, exception approvals, and tracking of security exception approvals; SP to provide a ticketing tool with SOC operations access to the financial institution.
baselineProposal must clearly describe authority boundaries for three tiers of response action: (1) non-disruptive playbook actions authorized without prior agency approval, (2) actions requiring standing agency approval (e.g. account disable/reset, endpoint isolation, blocking IOCs at firewall/proxy/email), and (3) actions requiring explicit agency authorization (changes outside Sentinel, destructive actions, external communications) — including an emergency exception process for confirmed active incidents threatening public safety or critical agency services.
baselineZero Trust Phase 1 planning: Contractor shall develop technical requirements/implementation plans for Defender for Servers & Cloud Apps (CASB) integration, migration from traditional VPNs to a SASE model, and Security Orchestration, Automation and Response (SOAR) with Conditional Access — engineering automated response playbooks and identity-centric access policies.
Signing up adds sharing with your team, sending this as an RFP to vendors, and private document sharing. Nothing above is taken away, and nothing here is sent anywhere until you choose to.
Platform baseline
neutral · staff-reviewed
RFIHow many pre-built integrations/connectors does the platform ship with for common security tools (SIEM, EDR, firewall, ticketing, identity), and how is a custom integration built for a tool without an out-of-box connector?Answer key — what a strong answer shows
Look for a specific integration count and a documented path (API/SDK) for custom connectors, not just 'we integrate with everything.'
RFPDescribe playbook authoring — is it a visual/no-code builder, code-based (Python), or both — and provide a customer reference for time-to-build and time-saved on a specific common playbook (e.g., phishing triage).Answer key — what a strong answer shows
Strong answers state a concrete time-to-build figure and time-saved metric from a real customer, not generic 'faster response' claims.
RFIWhat percentage of alerts can realistically be fully auto-closed without analyst involvement for common use cases (e.g., known-benign phishing reports), and what safeguards exist to prevent an incorrect auto-close of a true positive?Answer key — what a strong answer shows
Look for an honest percentage (not '100% automation') and a described safeguard, e.g. confidence thresholds or human-in-the-loop for ambiguous cases.
RFIHow does the platform handle playbook version control, testing, and rollback — can a playbook be tested in a dry-run/simulation mode before being enabled in production?Answer key — what a strong answer shows
Dry-run/simulation testing before production enablement is a meaningful safety feature; a vendor with no answer likely expects customers to test in production.
RFPDetail case management capabilities — does the platform provide native incident case management (timeline, evidence, collaboration) or does it only orchestrate actions across other tools that retain the case record?Answer key — what a strong answer shows
Native case management with a full timeline/audit trail is stronger than a pure orchestration engine with no persistent case record of its own.
From other buyers
crowdsourced · anonymized
💬
No buyer-contributed criteria yet
Verified buyers can suggest criteria (anonymized before pooling).
RFIWhat metrics/reporting does the platform provide on automation ROI — mean-time-to-respond (MTTR) reduction, analyst hours saved — and is this measured automatically or does it require manual tracking?Answer key — what a strong answer shows
Look for automatically-tracked MTTR/hours-saved metrics tied to specific playbooks, not a one-time consulting estimate.
RFPExplain how the platform handles multi-tenant or MSSP use cases — can playbooks and integrations be shared across tenants while keeping tenant data and credentials isolated?Answer key — what a strong answer shows
Relevant for MSSPs specifically; look for genuine tenant isolation of credentials/data alongside playbook reuse, not a single shared workspace.
RFIHow are third-party API credentials and secrets used by playbooks stored and rotated — is there a built-in secrets vault, or does it rely on the customer's own secrets management?Answer key — what a strong answer shows
A built-in, audited secrets vault with rotation support is stronger than plaintext credential storage or reliance on the customer's own layer.
RFIWhat is the pricing model — per playbook, per analyst seat, per event/alert volume processed — and how does cost scale as alert volume grows significantly (e.g., after onboarding a new high-volume data source)?Answer key — what a strong answer shows
Look for transparent, predictable scaling economics; event-volume-based pricing that spikes unpredictably when a new noisy data source is onboarded creates real budget risk.
RFPDoes the platform offer AI/LLM-assisted playbook generation (describing a desired workflow in natural language and having a draft playbook generated), and how much manual refinement is typically needed before that draft is production-ready?Answer key — what a strong answer shows
Strong answers are honest about how much manual work remains after AI-assisted generation — a vendor claiming 'fully automated playbook creation' with no refinement needed should be tested directly, not taken at face value.
RFIHow does the platform support cross-team workflows beyond the security team itself — e.g., automatically routing a data-privacy incident to legal, or a compliance finding to the GRC team — with appropriate access scoping for non-security stakeholders?Answer key — what a strong answer shows
Look for genuine cross-team routing with proper access controls for non-security users; a SOAR platform that only knows how to talk to other security tools misses real incident-response workflows that cross departments.
RFIWhat happens to in-flight automated incident response if the SOAR platform itself has an outage mid-incident — is there a documented fallback/manual-continuation process, and has this actually been tested?Answer key — what a strong answer shows
A vendor without a tested fallback plan for their own platform's downtime during an active incident is a meaningful operational risk — ask if this has been tested, not just theorized.
RFPDetail the availability of a staging/testing environment separate from production for validating new playbooks and integrations before they run against live security data and can take real automated actions.Answer key — what a strong answer shows
A genuine staging environment (not just a 'dry run' toggle within production) is materially safer for testing new automation before it can take real, potentially destructive actions.
RFIIs there a community or marketplace for sharing pre-built playbooks and integrations across customers, and how vetted/maintained is that content versus being unmaintained community contributions?Answer key — what a strong answer shows
Look for an actively maintained, vendor-curated marketplace rather than an abandoned community forum — unmaintained shared playbooks can be a real security risk if they embed outdated assumptions.
RFPExplain the audit trail for automated actions — is every action a playbook takes (e.g., disabling a user account, blocking an IP) logged with full context for compliance/forensic review, and can this audit trail be exported to an external SIEM or GRC system?Answer key — what a strong answer shows
A complete, exportable audit trail of automated actions is essential for both incident forensics and compliance — a platform with only partial or non-exportable logging creates real accountability gaps.
RFIWhat is the typical learning curve for a new analyst to become proficient building and maintaining playbooks — is formal training included, and what is a customer-referenced timeline from onboarding to a team member independently authoring production playbooks?Answer key — what a strong answer shows
Strong answers give a concrete, customer-validated timeline; a 'no-code' claim doesn't guarantee genuine ease of use for complex real-world playbook logic.
RFPIntegrate with key components of the existing IT ecosystem, including Heimdal EDR, Cisco Meraki, Microsoft 365, Azure AD, and on-prem Active Directory.
RFPProvide robust correlation, investigation, and automation workflows to reduce response time and improve analyst efficiency.
RFPDevelop and deploy automated response playbooks using Logic Apps or other Microsoft-native tools, aligned with pre-approved containment actions and escalation paths (Playbook and Automation Setup).
RFPShould support automated response actions (e.g., isolating compromised endpoints, triggering firewall rules, revoking user access).
RFPVendors should provide pricing for different deployment models and include cost breakdowns for licensing (per user, per endpoint, per GB, etc.), support and maintenance, and additional features (e.g., SOAR, extended retention, custom integrations).
RFPAlerting and Incident Integration: configure alert notifications, integrate with ITSM/SOAR platforms (e.g., in-house ticketing, Jira, Zoho), and configure incident workflows and escalation.
RFPBidder to provide workflow automation so that applications and infrastructure are integrated automatically, with minimal manual intervention.
RFPThe vendor should bring workflows and solutions that can automate the majority of incident response activities such as false positive management, managing whitelists, escalation workflow, and SLA management; alerts should be notified to the financial institution only after a proper triage process and enriched with context data, environmental data, vulnerability data, historical data, and threat intelligence.
RFPSolution should support centralized incident management and triaging of alerts from multiple security products (SIEM, DLP, IPS, WAF, Anti-APT, ETDR); investigation module should integrate with log sources (SIEM, ETDR, EPP, Data Lake) on demand to pull data related to the investigated alert with charting and graphing to analyse data.
RFPSolution should support full workflow for incident classification, coordination, escalation, exception approvals, and tracking of security exception approvals; SP to provide a ticketing tool with SOC operations access to the financial institution.
RFPProposal must clearly describe authority boundaries for three tiers of response action: (1) non-disruptive playbook actions authorized without prior agency approval, (2) actions requiring standing agency approval (e.g. account disable/reset, endpoint isolation, blocking IOCs at firewall/proxy/email), and (3) actions requiring explicit agency authorization (changes outside Sentinel, destructive actions, external communications) — including an emergency exception process for confirmed active incidents threatening public safety or critical agency services.
RFPZero Trust Phase 1 planning: Contractor shall develop technical requirements/implementation plans for Defender for Servers & Cloud Apps (CASB) integration, migration from traditional VPNs to a SASE model, and Security Orchestration, Automation and Response (SOAR) with Conditional Access — engineering automated response playbooks and identity-centric access policies.
