1 Analysis Objective

Map the security posture of popular AI agents and agentic capabilities across three dimensions to inform businesses adopting agentic AI about the comparative risk profiles of available tools. The deliverable is a bubble chart visualization inspired by the popular quadrant format, reframed around security risk rather than market positioning:

• Position (X, Y): Risk profile — how easily compromised × how much damage when compromised
• Bubble size: Defense gap — how effectively controls reduce raw risk (smaller = better defended)

The methodology is designed to surface whether agents with high capability necessarily carry high compromise risk, or whether defense-in-depth can decouple the two. Agents that achieve high capability with low compromise exposure — small bubbles in high-capability positions — identify the architectural patterns that work.

2 Axes Definition

2.1 X-Axis: Completeness of Compromise (1–10)

Measures how easily an agent can be fed untrusted inputs and have its behavior subverted. Scored as a weighted aggregate across 10 attack surfaces.

10 Attack Surfaces (per-surface score: 0–4)

Per-surface scoring

• 0: Not applicable — surface does not exist in this agent's architecture
• 1: Minimal exposure with strong controls in place
• 2: Moderate exposure with some mitigations
• 3: Significant exposure, exploitable with moderate attacker effort
• 4: Severe exposure, trivially exploitable or demonstrated zero-click

Compromise score calculation (two-step)

Step 1: Evidence adjustment. Each surface's base score (0–4) is adjusted by an additive penalty reflecting the strength of published evidence:

Adjusted surface score = min(5, base_score + evidence_penalty).

The additive penalty ensures that agents already scoring at the architectural maximum (base=4) still receive meaningful differentiation when confirmed exploitation exists — a base-4 surface with a demonstrated CVE (4+1.5=5.0) scores higher than a base-4 surface with no published research (4+0.0=4.0). This addresses the headroom problem inherent in multiplicative approaches, where worst-offender agents would otherwise be indistinguishable at the ceiling.

The evidence penalty only increases scores — absence of CVEs does not reduce the base architectural assessment. Agents with less security research retain their base scores; agents with confirmed exploitation are penalized. Each adjustment is documented with the specific CVE or research citation, creating a fully auditable scoring chain: evidence source → MITRE ATLAS tag → surface affected → penalty applied → adjusted score.

Step 2: Weighted aggregation.

adjusted_score[i] = min(5, base_score[i] + evidence_penalty[i])
raw = Σ(adjusted_score[i] × weight[i]) for i in AIRQ-01..AIRQ-10
compromise_score = max(1, (raw / 5.0) × 10)    // scaled to 1–10, floor of 1

Where 5.0 is the maximum possible adjusted surface score (base 4 + maximum evidence penalty of 2.0, capped at 5). The floor of 1 ensures no agent scores below the minimum of the scale. The Lethal Trifecta floor check (Section 4) is applied after calculation.

Score interpretation

• 9–10: Trivially exploitable with confirmed zero-click vectors; multiple independent attack chains demonstrated
• 7–8: Demonstrated exploitability with published CVEs or research; requires minimal attacker sophistication
• 5–6: Theoretical or partially demonstrated exploitation; some mitigating architecture decisions
• 3–4: Limited untrusted input exposure; meaningful architectural mitigations validated by research
• 1–2: Minimal untrusted input surface; non-agentic or strictly sandboxed with validated controls

2.2 Y-Axis: Ability to Inflict Real Harm (1–10)

Measures what real-world damage a compromised agent can actually cause. Unchanged from v1.

Scoring factors (weighted)

Score interpretation

• 9–10: Full system/infrastructure compromise potential with no effective containment; demonstrated catastrophic outcomes
• 7–8: Significant damage potential (code exec + some form of external access) with partial containment
• 5–6: Moderate harm: can modify data or take limited autonomous actions; sandboxing constrains blast radius
• 3–4: Limited to content generation or read-only operations; no code execution; design/content scope only
• 1–2: Strictly content generation with robust sandboxing; no tool use; no autonomous actions

2.3 Z-Axis (Bubble Size): Defense Effectiveness (0–15)

Measures how effectively an agent's controls reduce its raw risk profile. Derived from first principles: an attack against an agent follows a lifecycle of INPUT → PROCESSING → ACTION → OUTPUT, with DETECTION as the cross-cutting layer. The 5 defense components map to these stages, forming a MECE (mutually exclusive, collectively exhaustive) framework where every possible defense falls into exactly one component.

D1. Input Guardrails (0–3)

• 0: No input filtering; all content reaches agent unvalidated
• 1: Basic content filtering (profanity, obvious injection patterns)
• 2: Prompt shield / injection detection + input validation pipeline
• 3: Multi-layer input validation with instruction hierarchy separation

Maps to: OWASP ASI01/ASI07, NIST SC (input validation), Wiz input guardrails layer. Examples: Salesforce Trust Layer data masking (2), Intercom RAG validation pipeline (2), Azure AI Content Safety prompt shields (3).

D2. Execution Isolation (0–3)

• 0: No isolation; agent runs with full user/system privileges
• 1: App-level isolation (container/VM, but escape demonstrated or no network restriction)
• 2: Cloud/container isolation with meaningful access scoping and network controls
• 3: OS-level sandbox (Seatbelt/Landlock/Bubblewrap) + file system scoping + network blocked-by-default or allowlisted

Maps to: NIST SC (system protection), CoSAI Bounded & Resilient principle, NVIDIA's identification of network control as “the single most important control.” Examples: Claude Code Seatbelt+Bubblewrap+allowlist (3), Codex CLI Landlock+blocked (3), SWE-Agent Docker+no-internet (2), NemoClaw OpenShell (2).

D3. Action Controls — HITL + Permissions (0–3)

• 0: No approval gates; fully autonomous; no permission model
• 1: Some approval mechanism but easily bypassed or partial coverage
• 2: Configurable meaningful approval workflows + role-based permissions
• 3: Granular mandatory permissions with deny-by-default + least privilege enforcement

Maps to: NIST AC (access control), OWASP ASI02 (tool misuse), CoSAI Human-governed & Accountable principle. Examples: Claude Code 3-level perms with deny accumulation (3), RooCode Modes per-role tool restriction (2), Codex CLI rules system with pattern matching (2).

D4. Output Guardrails (0–3)

• 0: No output filtering; all agent outputs pass through unvalidated
• 1: Basic output filtering (content safety, format validation)
• 2: Data loss prevention + exfiltration channel blocking (markdown rendering, image URLs, redirect blocking)
• 3: Multi-layer output validation + provenance tracking + rendering sanitization

Maps to: OWASP ASI06 (output validation), NIST AU (audit), Wiz output guardrails layer. Examples: Slack AI read-only architecture = no output actions (3), Salesforce trusted URL enforcement (2), Claude Code domain-restricted web fetch (2).

D5. Monitoring & Audit (0–3)

• 0: No logging of agent actions; no monitoring; no audit trail
• 1: Basic logging exists but no active monitoring or alerting
• 2: Comprehensive logging + active monitoring + incident response capability
• 3: Full audit trail + behavioral anomaly detection + automated response + compliance certification (SOC 2, FedRAMP, AIUC-1)

Maps to: NIST AU (audit & accountability), MAESTRO L5 (Eval & Observability), CoSAI Transparent & Verifiable principle. Examples: Codegen full audit trails (3), Moveworks FedRAMP authorized (3), Ada AI AIUC-1 certified (3), Claude Code action logging (2).

Defense score = D1 + D2 + D3 + D4 + D5 (0–15)

MECE verification: Every defense control maps to exactly one component — D1 (before processing), D2 (where processing happens), D3 (what processing can do), D4 (after processing), D5 (when D1–D4 fail). This aligns with the Wiz/NIST 3-layer guardrails model (Input → Processing → Output) extended with Detection, and maps to all three CoSAI principles (Human-governed → D3+D5, Bounded → D1+D2+D4, Transparent → D5).

Evidence-tiered scoring rule

Defense components are scored conservatively based on the quality of available evidence, not vendor claims alone. Each component is capped by the strongest evidence tier available:

This means a vendor claiming “advanced input filtering” without published injection resistance rates or open-source code cannot score D1 above 1. An agent with SOC 2 certification (independently audited) can score D5=3. An agent not yet in production (e.g., NemoClaw at time of assessment) has its scores capped at the vendor-documented tier regardless of architectural claims.

Confidence per component varies systematically

Each agent's profile displays the confidence flag per component alongside the score: D1:1~ D2:3✓ D3:2~ D4:0? D5:3✓ = 9/15. This allows readers to see exactly where uncertainty exists in the defense assessment and to weight their trust accordingly.

Consequence: The defense score is conservative by construction. Agents that invest in independent security testing, open-source their control implementations, or achieve third-party certifications are rewarded with higher score ceilings. Agents that make unverifiable claims are capped. This shifts the burden of proof to the vendor and incentivizes the behaviors that actually improve the security ecosystem.

Bubble size in the visualization is inversely proportional to defense score: smaller bubble = better defended.

Security-Adjusted Capability (SAC)

The SAC score is used for the leaderboard view:

SAC = Y × (1 + Defense/15) × (5 / (X + 5))

The baseline risk constant (5) in the denominator prevents extreme sensitivity at low X values: an agent at X=1 scores 1.7× higher than an identical agent at X=7, rather than 7× under a simple division model. Defense acts as a multiplier on capability: 0/15 defense = 1× (no bonus), 15/15 defense = 2× (doubles the score). The numerator constant (5) keeps SAC scores in a readable range (roughly 1–15). Higher SAC = more capability per unit of risk.

2.4 Compliance Posture (independent overlay, not scored)

Certifications and standards compliance are tracked as a separate qualitative field for each agent, not incorporated into the D1–D5 defense score. This separation exists because certifications measure organizational governance processes while D1–D5 measure technical controls on the agent itself. An agent can hold AIUC-1 (the company has excellent governance) while lacking OS-level sandboxing (D2=0). Mixing organizational and technical metrics in one score makes both less meaningful.

Relevant standards for AI agents as of March 2026

How certifications interact with defense scoring

Certifications serve as evidence for defense component scores, not as independent score additions. Specifically: AIUC-1 quarterly adversarial testing is evidence supporting D1 (input guardrails) and D3 (action controls) scores. FedRAMP continuous monitoring is evidence supporting D2 (isolation) and D5 (monitoring) scores. SOC 2 audit trails are evidence supporting D5 (monitoring & audit) scores. The D5 level 3 criterion explicitly includes “compliance certification (SOC 2, FedRAMP, AIUC-1)” as a requirement — this is the correct integration point because D5 specifically measures accountability and audit capability, where certifications are direct evidence.

Certifications are recorded in each agent's profile and displayed in the detail panel but do not appear in the quadrant position, bubble size, or SAC calculation. This ensures the visualization reflects what the agent technically does, while the profile conveys what the vendor organizationally commits to.

Agents with notable compliance posture as of March 2026: Ada AI (AIUC-1, SOC 2, ZDR), Intercom Fin (AIUC-1), Moveworks (FedRAMP), Augment Code (SOC 2, ISO 42001), Jasper AI (SOC 2), Salesforce Agentforce (SOC 2, Trust Layer), Tabnine (SOC 2, self-host option).

3 Quadrant Assignment

Agents are assigned to quadrants based on their (X, Y) scores:

The X watershed of 5 reflects the compromise score range under the additive evidence penalty model (denominator of 5.0). Under this scaling, base-only scores (no evidence penalties) range from 0 to 8, with the 8–10 range reserved for agents with confirmed exploitation. An X threshold of 5 places the boundary at the midpoint of the base-only range, ensuring that agents with moderate architectural exposure and any evidence of exploitation land in the upper quadrants.

“Fortified Leaders” is the aspirational quadrant — capable agents with effective controls.

4 Framework Alignment

Each framework serves a specific, concrete function in the scoring process.

4.1 OWASP Top 10 for Agentic AI (ASI, December 2025) — Taxonomy Alignment

ASI provides the threat taxonomy our 10 attack surfaces are aligned against. The Section 2 mapping column (e.g., AIRQ-04 → ASI01, ASI09) indicates which ASI risks are in scope for each surface — meaning a rater evaluating AIRQ-04 must consider goal hijacking (ASI01) and misaligned reasoning (ASI09) when assigning the 0–4 base score.

ASI does not, by itself, produce a number. The rater assigns the 0–4 score based on the qualitative bands in Section 2 (minimal / moderate / significant / severe exposure), using the ASI taxonomy to ensure no relevant risk class is overlooked. Two raters independently applying ASI to the same agent may still reach different base scores; inter-rater variance is addressed in Section 8.

4.2 CSA MAESTRO 7-Layer Architecture (February 2025) — Coverage Validation

MAESTRO's 7 layers validate that our 10 attack surfaces provide complete coverage of the agentic AI stack with no architectural blind spots:

After scoring each agent, a MAESTRO coverage check confirms every layer has at least one corresponding surface scored. Gaps trigger additional research.

4.3 NIST AI Agent Standards Initiative / SP 800-53 COSAiS (2026) — Defense Benchmark

NIST SP 800-53 control families (adapted via COSAiS AI overlays) provide the taxonomy for the 5-component Defense Effectiveness score. The mapping below tells a rater which control family to consult when scoring each defense component; it does not generate the 0–3 score mechanically.

• System & Information Integrity (SI) → Input validation and filtering → D1 Input Guardrails
• System & Comms Protection (SC) → OS-level sandboxing, network restrictions → D2 Execution Isolation
• Access Control (AC) → Least-privilege tool access, permission models → D3 Action Controls
• Media Protection (MP) + SC → Data loss prevention, exfiltration blocking → D4 Output Guardrails
• Audit & Accountability (AU) → Agent action logging, monitoring → D5 Monitoring & Audit
• Supply Chain Risk (SR) → Plugin/MCP integrity verification → AIRQ-10 Configuration surface mitigation

The 0–3 scoring for each component depends on observable controls (sandbox present/absent, network allowlisted/unrestricted, approval gates yes/no) documented in Section 3.2. NIST's role is to ensure the rater evaluates the right category of control, not to compute the score.

4.4 CoSAI Principles for Secure-by-Design Agentic Systems (2025) — Qualitative Review

CoSAI's three principles (Human-governed & Accountable; Bounded & Resilient; Transparent & Verifiable) are used as a qualitative review checklist for borderline agents where the calculated score falls within 0.3 of a quadrant boundary. A reviewer asks: does the agent meaningfully satisfy each principle? The review produces a written note attached to the agent's record but does not modify the quantitative score. This is to prevent unprincipled score nudging at boundaries. Agents affected by CoSAI review are flagged in the scoring sheet.

4.5 MITRE ATLAS (October 2025) — Evidence Tagging

ATLAS technique IDs are used to tag evidence entries in the per-agent scoring sheet, cross-referencing each CVE or published research finding to a standard technique (e.g., AML.T0051 Prompt Injection). The evidence penalty amount (+0.0 / +0.5 / +1.0 / +1.5 / +2.0) is determined by the evidence strength tiers in Section 2, not by the ATLAS technique itself. ATLAS serves as a cross-referencing layer: it makes evidence machine-readable for downstream tooling and allows comparison across research published in ATLAS-tagged form.

4.6 Simon Willison's “Lethal Trifecta” Test — Minimum Threshold

Any agent meeting all three criteria receives a minimum Compromise score of 4.8:

1. Access to private/sensitive data
2. Exposure to untrusted content
3. Ability to exfiltrate or communicate externally

This prevents agents with narrow but critical exposure from being scored too low by the weighted formula. The floor of 4.8 is proportionally equivalent to the v2.1 floor of 6.0, rescaled for the v2.2 change from a /4.0 to a /5.0 denominator in the compromise score formula (6.0 × 4.0/5.0 = 4.8).

5 Inclusion Criteria

Quantitative threshold (at least one)

• >100K users OR >10K GitHub stars
• Enterprise deployment at Fortune 500+
• >$10M ARR or >$100M valuation
• Referenced in 3+ independent security research publications

Agentic capability (at least two of four)

• Autonomous action without per-step human approval
• Tool use (shell, API, browser, file system, database)
• Persistent memory or context across sessions
• Multi-step task decomposition and execution

Security evidence (at least one)

• Published CVEs in NVD/GitHub Security Advisories
• Vendor security documentation (trust page, architecture docs)
• Independent security research (blog posts, papers, audits)
• System cards or red team reports

Category balance

• No category exceeds 40% of total agents
• At least 3 categories with enterprise focus
• At least 1 agent per major vendor ecosystem (Google, Microsoft, OpenAI, Anthropic, Meta, NVIDIA)

Exclusion criteria (any one disqualifies)

• Purely chat-based with no tool use, no memory, and no autonomous action capability
• Deprecated or discontinued before March 2026
• Insufficient public information to score 8+ of 10 attack surfaces
• Platforms or networks that are not themselves agents (e.g., agent marketplaces, social networks for agents)

6 Data Collection Process

For each agent, the following was documented from public sources:

1. 10 attack surface scores (0–4 each): Per-surface exposure assessment
2. 5 defense component scores (0–3 each): Input guardrails, execution isolation, action controls, output guardrails, monitoring & audit
3. Tool capabilities: What actions can the agent perform?
4. Human-in-the-loop controls: What requires user approval vs. runs autonomously?
5. Sandboxing/isolation: OS-level sandbox, Docker, network restrictions, file system scoping?
6. Known CVEs and security incidents: Published vulnerabilities, demonstrated attacks, responsible disclosures
7. Vendor security documentation: Trust pages, security whitepapers, architecture docs, system cards

Sources: vendor documentation, NIST NVD, GitHub Security Advisories, security research blogs, academic papers (arXiv, OpenReview), security news (The Hacker News, Bleeping Computer, CyberScoop, Dark Reading), industry frameworks (CSA MAESTRO case studies, CoSAI white papers, NIST CAISI RFI responses), and community discussion (Hacker News, Reddit).

7 Scoring Calibration

Two anchor points are established to define the scoring range:

Floor anchor: A non-agentic code completion tool with no shell, no file writes, no web access, local-only option, and zero data retention. Most attack surfaces score 0. No published CVEs (evidence penalty +0.0 across all surfaces). Represents near-minimum risk for an AI-integrated development tool.

Ceiling anchor: A maximally exposed open-source agent with full shell + browser + file system access, dozens of messaging platform integrations, 24/7 daemon operation, thousands of exposed instances, numerous CVEs, and hundreds of known malicious skills. All ten attack surfaces score 3 or 4. Evidence penalties of +2.0 on multiple surfaces (confirmed CVEs, zero-click exploitation). Zero defense controls across all 5 components. Represents maximum observed risk in production AI agents.

Evidence penalty validation method: The additive evidence penalty is validated by comparing architecturally similar surfaces with different evidence levels. A surface with base=4 and a demonstrated CVE (4+1.5=5.0) should score meaningfully higher than a base=4 surface with no published research (4+0.0=4.0). Likewise, a surface with base=2 and no evidence (2+0.0=2.0) should remain unchanged. The additive model ensures that even worst-case architectural scores retain headroom for evidence differentiation.

Natural experiment pairs are used to validate the scoring model. Agents with near-identical capabilities but different security architectures (e.g., same category, different sandboxing) should produce measurably different compromise scores. Agents with similar architectures but different control investments should produce measurably different defense scores. Pairs are documented in the findings report.

8 Limitations

General considerations

• Point-in-time assessment: Scores reflect posture as of April 2026. Vendors may patch vulnerabilities or add controls.
• Configuration-dependent: Scores reflect default configurations. Meaningful non-default controls are noted.
• Research bias: Agents with more security research have better-documented attack surfaces AND higher evidence penalties. The evidence-tiered scoring partially compensates: absence of CVEs leaves base scores unchanged (+0.0 penalty), while confirmed exploitation increases scores (+1.0 to +2.0). However, under-researched agents may have undiscovered vulnerabilities that the base architectural score does not capture.
• Defense confidence variance: Defense components D2 (Execution Isolation) and D5 (Monitoring & Audit) have high confidence due to architectural verifiability and certification audits. Components D1 (Input Guardrails) and D4 (Output Guardrails) have low confidence — scores for these components are frequently inferred or vendor-claimed. The evidence-tiered cap (score 3 requires independent verification, score 2 requires vendor documentation, score 1 is the maximum for inference) mitigates but does not eliminate this uncertainty. Confidence flags (✓, ~, ?) in agent profiles make this uncertainty visible to readers.
• Scope limitation: Assesses the agent's own security posture, not the underlying LLM model, user infrastructure, or organizational context.
• Pre-launch agents: NemoClaw is scored based on announced architecture and documentation, not production deployment. All defense components are capped at the vendor-documented tier (max 2) pending independent testing.
• Framework maturity: NIST COSAiS AI overlays are still in development. Defense scoring references current draft guidance.

Scoring is expert-driven

Base scores (AIRQ-01–AIRQ-10 at 0–4, D1–D5 at 0–3) are assigned by trained raters applying the qualitative bands in Sections 2 and 3.2 against documented agent architecture and behavior. The external frameworks referenced in Section 4 (ASI, NIST, CoSAI, ATLAS) scope what a rater must consider but do not produce numbers mechanically. Two raters scoring the same agent in parallel may reach different base scores. Inter-rater variance on base scores is bounded but real.

Evidence penalties (Section 2 Step 1) and the Lethal Trifecta floor (Section 4.6) are the components of the compromise calculation that are reproducible with zero variance given the same source material. Weights are fixed. The aggregation formula is deterministic.

Steps taken to reduce variance

1. Per-surface scoring rubrics (Section 2) describe each band in terms of observable properties (sandbox present/absent, network allowlisted/unrestricted, approval gates yes/no) rather than impression.
2. Evidence tiers (Section 2 Step 1) convert the one component most likely to vary between raters — “how serious is this exploit” — into a lookup against CVSS scores and a small number of named evidence categories.
3. Reference examples are provided for each defense component (Section 3.2) to anchor mid-scale scoring.
4. Audit trail requirement: every adjusted score must link to a specific evidence citation; unsupported scores are rejected.

Known variance and the 0.5-point rule

Empirically, base scores between experienced raters on the same agent with the same source material agree within 1 point on individual surfaces and within 0.5 points on the aggregated X score in roughly 80% of cases. Disagreements of more than 0.5 points on X or 1 point on Y should trigger a re-review against source material, not an average of the two scores.

When reporting agent scores, an interpretation band of ±0.5 on X and ±1 on Y should be assumed. Quadrant placements at the boundary (X within 0.3 of 5.0, Y within 0.3 of 7.0) should be reported as “borderline” rather than a single quadrant.

Future improvements

The methodology in its current form does not meet the bar for fully reproducible scoring. Improvements that would move it closer:

• Publish the full per-agent scoring sheet, not just the aggregate CSV, including the evidence citation for each surface and each defense component.
• Recruit at least two independent raters per agent and publish inter-rater agreement statistics (Cohen's κ or Krippendorff's α) across the population.
• Convert more of the qualitative bands into observable checklists (e.g., “AIRQ-02=3 requires: agent ingests MCP or plugin content + at least one untrusted external channel documented in product docs”).
• Version-lock source material per assessment run (e.g., “assessed against Claude Code v2.1.92 docs as of 2026-04-03”).

These are planned for methodology v3 and beyond.