Start from a neutral baseline and add what matters to you. Criteria are labeled by source — the platform baseline is architecture-neutral; buyer-contributed criteria are shown separately.
Build your evaluation
no account needed
Match on your requirements
no account needed
This evaluation is stored in your browser only. We cannot see it, and it is not tied to any account. Save it to a link or create an account to keep it across devices — you can export it at any time either way.
1. Weight what matters
21 criteria
baselineWhat categories of AI/LLM risk does the product address — prompt injection, data/PII leakage via prompts or outputs, model supply-chain risk (weights/fine-tunes), or agentic/tool-use misuse — and which are fully covered versus roadmap?
baselineDescribe how the product discovers 'shadow AI' usage across the organization (unsanctioned LLM apps, browser extensions, API calls to external model providers) and quantify detection coverage with a customer reference.
baselineFor prompt injection and jailbreak defense, is detection static (pattern/signature matching) or does it use an evaluator model, and what is the measured false-positive rate on legitimate prompts?
baselineHow does the product prevent sensitive data (PII, secrets, source code) from being sent to third-party LLM APIs — inline redaction/masking, blocking, or logging-only?
baselineDetail support for securing self-hosted/fine-tuned models, including model provenance/lineage tracking, vulnerability scanning of model artifacts, and detection of data poisoning in training pipelines.
baselineDoes the platform inspect and control agentic AI workflows (autonomous tool-calling, multi-step agents), and if so, how are excessive-permission or unintended-action risks flagged or blocked in real time?
baselineExplain how the product maps findings to a recognized framework (OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF) and how coverage against that framework is reported to auditors or leadership.
baselineWhat is the deployment model for inline traffic inspection (proxy, SDK/API wrapper, browser extension) and what latency overhead does it add to LLM calls under production load?
baselineDoes the platform address model output risk — hallucination detection, harmful/toxic content moderation, and brand-safety filtering on generated responses — or only input-side threats like prompt injection?
baselineWhich LLM providers and model types are natively supported (OpenAI, Anthropic, Google, open-weight/self-hosted models), and does coverage or feature depth differ meaningfully by provider?
baselineDescribe RAG-specific security: how are vector database contents and embeddings protected from unauthorized retrieval, and can the platform detect embedding-based data exfiltration or poisoning of a retrieval corpus?
baselineDoes the platform offer or integrate AI red-teaming / adversarial testing services to proactively probe a customer's own deployed models, and how often are new jailbreak techniques incorporated into that testing?
baselineExplain the incident response process specific to an AI-security event (e.g., a jailbreak that caused a real data leak) — what forensic data is captured, and is there a named IR/breach-coach service included or available as an add-on?
baselineHow does the platform track and report compliance readiness for AI-specific regulation (EU AI Act risk-tiering, sector-specific AI rules), and is this a live-maintained mapping or a static one-time assessment?
baselineDetail the pricing model — per seat, per API call/token volume, per protected model — and how cost scales as an organization's LLM usage grows 10x; provide a customer reference at meaningful production scale.
baselineWhat is measured false-positive rate on legitimate AI-assisted workflows (e.g., a developer using an AI coding assistant), and how much manual tuning is required before the product is usable without alert fatigue?
baselineProvide an in-depth overview of the proposed use of Artificial Intelligence in the technical approach, describing how and in what circumstances AI will be used to perform the services, with separate pricing for scenarios with and without AI use.
baselineDemonstrate how the technology utilizes artificial intelligence to identify anomalies, reduce false positives, and update OT asset information.
baselineShould offer machine learning (ML) or AI-based behavioral analytics to detect anomalies.
baselineContractor shall support Supply Chain Risk Management (SCRM) activities, including coordination, record maintenance, and risk mitigation recommendations for third-party technologies, and support AI security-related compliance, vulnerability, and risk activities for systems/tools that incorporate AI-enabled functions or emerging technologies.
baselineVendor shall ensure security controls are enabled to prevent agency data leakage to AI or ML systems and prevent impacts from adversarial machine learning; Vendor staff must undergo annual training and the Vendor shall adhere to a framework such as ISO/IEC 42001:2023 or the NIST AI Risk Management Framework.
Signing up adds sharing with your team, sending this as an RFP to vendors, and private document sharing. Nothing above is taken away, and nothing here is sent anywhere until you choose to.
Platform baseline
neutral · staff-reviewed
RFIWhat categories of AI/LLM risk does the product address — prompt injection, data/PII leakage via prompts or outputs, model supply-chain risk (weights/fine-tunes), or agentic/tool-use misuse — and which are fully covered versus roadmap?Answer key — what a strong answer shows
Strong answers give a category-by-category breakdown of what's shipped today versus planned, rather than a blanket 'we cover AI security' claim.
RFPDescribe how the product discovers 'shadow AI' usage across the organization (unsanctioned LLM apps, browser extensions, API calls to external model providers) and quantify detection coverage with a customer reference.Answer key — what a strong answer shows
Look for a concrete discovery mechanism (network/proxy telemetry, endpoint agent, CASB-style API discovery) and a real coverage figure, not a marketing claim of 'full visibility.'
RFIFor prompt injection and jailbreak defense, is detection static (pattern/signature matching) or does it use an evaluator model, and what is the measured false-positive rate on legitimate prompts?Answer key — what a strong answer shows
Evaluator-model-based detection generalizes better than static patterns; a vendor that can't state a measured false-positive rate likely hasn't tested against real production traffic.
RFIHow does the product prevent sensitive data (PII, secrets, source code) from being sent to third-party LLM APIs — inline redaction/masking, blocking, or logging-only?Answer key — what a strong answer shows
Inline redaction or blocking is materially stronger than logging-only, which only provides after-the-fact visibility into a leak that already happened.
RFPDetail support for securing self-hosted/fine-tuned models, including model provenance/lineage tracking, vulnerability scanning of model artifacts, and detection of data poisoning in training pipelines.Answer key — what a strong answer shows
From other buyers
crowdsourced · anonymized
💬
No buyer-contributed criteria yet
Verified buyers can suggest criteria (anonymized before pooling).
Strong answers address all three sub-questions distinctly; watch for answers that only cover inference-time protection while ignoring the training/supply-chain surface.
RFIDoes the platform inspect and control agentic AI workflows (autonomous tool-calling, multi-step agents), and if so, how are excessive-permission or unintended-action risks flagged or blocked in real time?Answer key — what a strong answer shows
Look for real-time policy enforcement on tool calls (not just post-hoc logging) and a clear permission/scoping model for what an agent is allowed to do.
RFPExplain how the product maps findings to a recognized framework (OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF) and how coverage against that framework is reported to auditors or leadership.Answer key — what a strong answer shows
A vendor that can produce a real framework-coverage report is further along than one offering only marketing-deck framework alignment claims.
RFIWhat is the deployment model for inline traffic inspection (proxy, SDK/API wrapper, browser extension) and what latency overhead does it add to LLM calls under production load?Answer key — what a strong answer shows
Strong answers state a measured latency figure under realistic load, not just 'minimal overhead.' Probe whether the figure is customer-validated or lab-only.
RFIDoes the platform address model output risk — hallucination detection, harmful/toxic content moderation, and brand-safety filtering on generated responses — or only input-side threats like prompt injection?Answer key — what a strong answer shows
Strong answers explicitly cover output-side risk as a distinct capability; a vendor that only discusses input filtering is missing half the real-world incident surface.
RFIWhich LLM providers and model types are natively supported (OpenAI, Anthropic, Google, open-weight/self-hosted models), and does coverage or feature depth differ meaningfully by provider?Answer key — what a strong answer shows
Look for an honest per-provider breakdown; a vendor claiming uniform 'all models supported' without specifics likely has shallower coverage for less-common providers.
RFPDescribe RAG-specific security: how are vector database contents and embeddings protected from unauthorized retrieval, and can the platform detect embedding-based data exfiltration or poisoning of a retrieval corpus?Answer key — what a strong answer shows
Look for a concrete answer on vector-store access control and poisoning detection; 'we secure the LLM layer' without addressing the retrieval corpus is an incomplete answer for RAG deployments.
RFIDoes the platform offer or integrate AI red-teaming / adversarial testing services to proactively probe a customer's own deployed models, and how often are new jailbreak techniques incorporated into that testing?Answer key — what a strong answer shows
Strong answers describe an ongoing (not one-time) red-teaming capability with a stated update cadence for new attack techniques.
RFPExplain the incident response process specific to an AI-security event (e.g., a jailbreak that caused a real data leak) — what forensic data is captured, and is there a named IR/breach-coach service included or available as an add-on?Answer key — what a strong answer shows
Strong answers describe AI-specific forensic capture (prompt/response logs, model version, retrieval context) rather than generic security-incident logging.
RFIHow does the platform track and report compliance readiness for AI-specific regulation (EU AI Act risk-tiering, sector-specific AI rules), and is this a live-maintained mapping or a static one-time assessment?Answer key — what a strong answer shows
Look for a maintained, updating regulatory mapping; a one-time PDF assessment goes stale as regulation evolves.
RFPDetail the pricing model — per seat, per API call/token volume, per protected model — and how cost scales as an organization's LLM usage grows 10x; provide a customer reference at meaningful production scale.Answer key — what a strong answer shows
Strong answers give real unit economics and a genuine production-scale reference, not just a pilot-scale pricing example.
RFIWhat is measured false-positive rate on legitimate AI-assisted workflows (e.g., a developer using an AI coding assistant), and how much manual tuning is required before the product is usable without alert fatigue?Answer key — what a strong answer shows
A vendor with a real, customer-validated false-positive figure and a described tuning process is stronger than one claiming 'accurate out of the box' with no numbers.
RFPProvide an in-depth overview of the proposed use of Artificial Intelligence in the technical approach, describing how and in what circumstances AI will be used to perform the services, with separate pricing for scenarios with and without AI use.
RFPDemonstrate how the technology utilizes artificial intelligence to identify anomalies, reduce false positives, and update OT asset information.
RFPShould offer machine learning (ML) or AI-based behavioral analytics to detect anomalies.
RFPContractor shall support Supply Chain Risk Management (SCRM) activities, including coordination, record maintenance, and risk mitigation recommendations for third-party technologies, and support AI security-related compliance, vulnerability, and risk activities for systems/tools that incorporate AI-enabled functions or emerging technologies.
RFIVendor shall ensure security controls are enabled to prevent agency data leakage to AI or ML systems and prevent impacts from adversarial machine learning; Vendor staff must undergo annual training and the Vendor shall adhere to a framework such as ISO/IEC 42001:2023 or the NIST AI Risk Management Framework.
