Nine Questions That Tell You Whether Your AI Security Vendor Is Selling Theater

A guide for security architects running an agentic AI security evaluation

AI security theater is a security control that creates the appearance of protection without materially reducing an organization's real-world AI risk. In agentic AI, common examples include prompt-only filtering, model-provider guardrails with no visibility into tools or business context, alert-only monitoring for autonomous agents, and content moderation that leaves high-risk agent actions unprotected.

In a recent Straiker webinar, Debunking Agentic AI: The Truth About Identity, Agency, and Agentic Security Theater, Rob van der Veer—Founder of the OWASP AI Exchange—and Sreenath Kurupati, Straiker’s CTO, examined the risks enterprises face when relying on incomplete protection for their AI systems.

They focused on distinguishing truly load-bearing security controls from superficial measures designed merely to check a compliance box, leaving critical data exposed as enterprise deployment scales.

Sreenath was direct about the prevalence of superficial controls: “Very often you have incomplete solutions or well-intentioned solutions not tackling everything, and it often becomes security theater from that perspective. It's certainly a really hard problem, by the way, and the industry's gonna have to work hard to get this done right.”

For security architects evaluating agentic security solutions and vendors, identifying this difference is essential. 

The strongest agentic AI security vendors should provide full-trace visibility, inspect tool and agent activity, detect semantic threats inline, enforce policies rather than merely alert, and contain compromised agents. Here are nine critical questions to ask during your next POC to true agentic security from security theater.

1. Does this see tool calls, or only prompts?

An agent's risk lives in what it does, not in what was said to it. The prompt is one input among many. The reasoning step, the tool the agent selects, the arguments it passes, the response it receives, and the next action it plans are where the security consequences emerge.

Sreenath made this point about inherited model guardrails, but the same limitation applies to any vendor whose inspection stops at the model boundary:

“Model guardrails don't have the context fully on what the tool actions are happening behind the scene, the entire system you have. They don't have the business context. So if you go out saying, I've enabled the frontier model guardrails, you are potentially convincing yourself that you've done your job as a security practitioner, but really what you've done is just secured a small surface area of your application.”

If a vendor tells you it "inspects inputs and outputs," ask what it considers an input. If the answer is the user's message and the model's completion, you are looking at a chatbot filter.

During the POC, ask the vendor to show you a complete incident trace. You should be able to see the user prompt, agent reasoning, tool invocation, arguments, tool response, and relevant business context together. More importantly, you should be able to identify where the attack entered and what the agent subsequently touched.

How Straiker answers this question

Defend AI inspects every request, tool call, and agent-to-agent handoff, including MCP traffic, and renders the agentic chain of threats as a visual trace from user to model to tool so you can see exactly where an attack entered and what it touched. 

Sreenath described the underlying principle during the webinar: "The only way to really figure out that there's actually a prompt injection happening is by combining multiple levels of information. You take the user prompt, you take the agent's reasoning, you take the tool being run, you take the output that's coming in."

2. What happens after a detection: does it block the action, or contain the agent?

Blocking an action and containing an agent are different controls.

A runtime guardrail can stop a dangerous tool invocation while leaving the agent running with the same compromised context. If the agent is still carrying a poisoned instruction and still planning, it can try another route to the same objective.

Straiker's STAR Labs research found that in 85% of successful attacks, the agent did something it was never authorized to do. That is an important distinction when evaluating runtime security. The failure is often not one isolated tool call. The agent itself has crossed an authority boundary.

So if a vendor says, "We block the malicious action," ask what happens to the agent afterward. If it continues its session, you have a control that addresses the symptom without necessarily addressing the compromised agent.

The stronger answer is both capabilities, with a clear escalation path between them.

These are different controls, and vendors blur them constantly. Blocking stops one action. Containment stops the agent.

How Straiker answers this question

Defend AI blocks at the action level with fine-grained controls for the moves that matter, including remote code execution, data exfiltration, destructive commands, system file access, and connections to high-risk MCP servers. When the agent itself is the problem, the Agentic Kill Switch contains it in seconds (more on the kill switch in question 6, too).

3. Is there an LLM in the detection path, and who pays for the inference?

If a vendor claims to understand whether an instruction is malicious, ask how that understanding actually happens.

Imagine there’s an instruction buried in a support ticket. Determining whether it is malicious may require understanding the language in the ticket, the agent's intended action, the tool it is about to use, and whether that combination is appropriate for the business. Simple pattern matching cannot answer that semantic question.

"Especially in the world of LLMs, LLMs who speak so many languages, you would at least need an LLM. But in practice, companies decide not to use an LLM for this type of detection because it's simply too expensive," Rob said. 

That cost creates an important tradeoff. A vendor that calls a frontier model on every step of an agent loop faces both inference cost and latency. Ask how the vendor solves that problem rather than accepting "AI-powered detection" as an answer.

You should know what kind of model runs in the detection path, what it was trained on, whether it operates inline or asynchronously, what latency it adds, and what the economics look like at your production volume. Ask the vendor to test it against your traffic profile.

How Straiker answers this question

Defend AI runs on purpose-built detection models trained on millions of real-world agent traces rather than general-purpose frontier models called on every request. That is what makes inline evaluation economically viable. The measured result is 98.1% detection accuracy at under 300ms latency, with 6 to 21 times lower false positive rates than frontier model judges. 

The cost problem Rob identified requires building detection models specifically for this job rather than renting a general one.

4. Or is it rules underneath?

(This is a continuation or follow up question to the previous one) 

Some products that present as AI-native are running regex, keyword lists, and signature matching with a classifier bolted on for the demo. As Sreenath said, "You also have solutions where, which are supposedly looking at all the context, but behind the scenes they actually don't work. They could be relying on very old security rules and stuff like that. It goes back to my comment that the newer systems require understanding natural language and tools, and you almost need an AI model to operate this on."

The easiest way to test the difference is to change the form of an attack without changing its meaning. Take an injection that works, translate it into Spanish, embed it in an image, place it in a PDF, and run the variations through the product. If detection changes dramatically based on wording, language, or document format, the underlying system may be relying more heavily on pattern matching than its positioning suggests.

How Straiker answers this question

Defend AI's detection is multimodal by design, covering threats hidden in text, code, images, audio, and file uploads, with multi-language support. Run the paraphrase test against it. That is what it was built for.

5. What is the true-positive rate, and what is the denominator?

Every vendor in this market will quote you a detection number. Very few will tell you what it was measured against. A detection percentage is only meaningful when you know what it was measured against.

A 99% detection rate on a public benchmark of known jailbreak strings says little about performance against an indirect prompt injection buried in a ticketing system. Likewise, a high true-positive rate without a false-positive rate can describe a product that fires frequently enough that your team eventually disables it.

Ask for four things together:

• True-positive rate and the dataset used to measure it
• False-positive rate on realistic benign traffic
• Latency added at your p95
• A clear definition of what the vendor considers "realistic benign traffic"

A single detection percentage without its dataset, false-positive rate, latency, or comparison baseline is difficult to evaluate. Public benchmark results also deserve scrutiny if the benchmark data may have been part of model training.

How Straiker answers this question

Defend AI's reported 98.1% detection accuracy on agent traces was measured at under 300ms latency, with false positives 6 to 21 times lower than frontier model judges evaluated on the same task.

The comparison baseline matters because a detection number without a baseline is difficult to interpret. During an evaluation, including an evaluation of Straiker, the strongest test is to measure performance against your own traffic and attack scenarios.

6. Can it stop inference mid-flight, or does it only alert?

Traditional security operations often assume that a human analyst has time to investigate an alert, understand the context, and execute a response. Autonomous agents can operate much faster. 

“We tend to implement detection in the form of incident response,” Rob explained. In a traditional Security Operations Center, analysts have time to review patterns and follow playbooks. But with agentic systems capable of executing actions or exfiltrating data, “the harm is already done.” Security must be able to stop the inference immediately upon a high-certainty detection, yet vendors remain “lagging when it comes to implementing such features.”

For an agent capable of exfiltrating data or executing code, an alert that waits in a SIEM queue may arrive after the damage is complete. Ask what happens between detection and human response. If the agent continues operating while the security team investigates, the product is providing visibility rather than containment.

A useful architecture should be able to halt execution automatically when confidence is high, escalate to a human when confidence is lower, and contain the agent when its context or behavior can no longer be trusted.

Straiker CEO and co-founder Ankur Shah described the product rationale this way: “We built the kill switch because 'open a ticket' is not an incident response plan for software that doesn't wait for business hours.”

How Straiker answers this question

The Agentic Kill Switch was built to close this very gap, and it is the industry's first control of its kind.

A runtime guardrail stops a dangerous tool call while leaving the compromised agent running. The Agentic Kill Switch takes the agent offline in seconds: it revokes the agent's tools, freezes its memory, suspends its session, or halts an entire fleet of agents at once, with audit-grade evidence attached to every containment action and a human security practitioner making the call.

Two architectural pillars distinguish this approach:

First, it operates at the agent level, not the action level. When an agent’s context is poisoned, blocking a single call fails; the agent’s compromised plan and memory persist. Containing the agent is the only way to end the incident.

Second, it provides fleet-level containment. As agents share tasks and trust peer output, compromised contexts propagate rapidly. Killing agents individually is ineffective; the Agentic Kill Switch halts the entire fleet instantly.

Finally, control remains with you. Shutdown controls at the frontier lab level cannot protect enterprises running agents on open-weights, custom frameworks, or connected MCP servers. In an "extremely fragmented and democratized" AI landscape, security must reside where your agents actually operate.

7. Does it distinguish reputational harm from actual harm?

Not every undesirable AI output represents the same security risk.

Rob challenged the amount of attention organizations sometimes devote to preventing chatbots from saying something offensive:

"Some of my clients spend a lot of effort on making their chatbots not offensive. What I mean by that is that users, if they want to manipulate a chatbot, they can use all kinds of tricks and then make it offend them. There's no real harm involved, right? Because they ask for it, and they'll get it."

He also noted that the public's reaction to offensive chatbot output is changing:

"The public is starting to realize this," Rob noted, "and I think that the newsworthiness of an offensive chatbot is starting to erode."

But he made an important distinction:

"If you're working with a chatbot, you definitely need to make sure that it doesn't lead people along a negative path... Depression, mental health, you know, that needs to be really taken care of."

That distinction matters when evaluating AI guardrails. A chatbot producing an embarrassing response creates reputational risk. An AI system leading a vulnerable person toward harmful behavior creates a very different kind of safety obligation.

The question for a vendor is whether its controls can distinguish between those risks and whether you can weigh them independently.

A demo focused entirely on blocking rude outputs, while the agent's ability to access systems and take actions remains largely unaddressed, is not a complete agentic security strategy. It is content moderation positioned as security.

How Straiker answers this question

Defend AI separates output safety concerns, including toxicity, hallucination, and application drift, from agentic harm controls covering remote code execution, exfiltration, destructive commands, and tool misuse. Both exist, they are configurable independently, and the guardrails are customizable to your policy and use case. You decide where the effort goes rather than inheriting a vendor's weighting.

8. Are you guarding content, or net agentic harm?

When asked which controls organizations deploying AI agents should prioritize, Sreenath focused on what he calls "agentic harms."

“It goes back to, ultimately, I'll use an overall catchword, like agentic harms. This includes things like remote code execution, data exfiltration, resource exhaustion—all caused very often by either indirect prompt injection or the agent itself simply misperforming. These are the threats you want guardrails against, focusing on net agentic harmful actions and the input stimuli that cause them.”

That means a security architecture needs visibility into both the inputs that cause an attack and the actions that result from it. Monitoring only inputs can miss an agent that behaves dangerously without an obvious injection. Monitoring only outputs can make it difficult to understand how the compromise occurred or intervene before the harmful action takes place.

Straiker's STAR Labs threat research illustrates the potential consequences. Its research found that 36% of successful attacks on AI coding agents ended in remote code execution, while 91% of successful attacks on productivity agents left no trace at all.

Those outcomes require a different security model from a policy catalog focused primarily on toxicity, PII, profanity, or off-topic responses. Those categories can be useful, but they do not describe an agent deleting a repository, exfiltrating data, misusing a tool, or moving laterally through connected systems.

When evaluating an agentic AI security product, look for explicit coverage of remote code execution, data exfiltration, destructive actions, unauthorized tool use, malicious MCP interaction, lateral movement, and resource exhaustion, with the input stimulus connected to the resulting action.

How Straiker answers this question

Ascend AI runs autonomous, multi-turn adversarial campaigns against agents targeting remote code execution, tool manipulation, data exfiltration, prompt injection, and lateral movement through MCP and connected tools. Findings are mapped to OWASP, MITRE ATLAS, and NIST.

Defend AI enforces against the same techniques at runtime. The two products are connected so that attack paths identified through Ascend AI can become runtime policies in Defend AI. 

9. Would this have caught the last three published incidents?

Save this for the end of the call, and make it concrete. Pick three recent, documented agentic attacks. Ask the vendor to walk you through each one against their product, step by step, out loud.

The value is in watching how they handle it. A team that has done the work will tell you which parts they would catch, which parts they would miss, and where the detection would have fired in the chain. A team selling theater will assert coverage without walking the steps, or will redirect to a different scenario they have rehearsed.

Ask specifically: at which step does your product fire, what does it do at that moment, and what damage has already occurred by then? 

Rob's underlying observation about where detection logic belongs is what makes this test work:

"The place where the application is being built, that's the place where you can really specify what is malicious and what is benign."

A vendor who understands that will ask you questions back about your architecture before answering. That is a good sign.

How Straiker answers this question

Straiker's STAR Labs team publishes detailed research into real agentic attacks, including an AI SRE agent manipulated through poisoned OpenTelemetry data into deploying a privileged Kubernetes container and executing ransomware, a coding agent tricked into exfiltrating data over VXLAN, and an agent that hid its own commands to conceal a credential theft. Bring those to the call and make us walk them, we can happily show them to you in action. 

The closing question most accidentally gloss over: what did the guardrail actually buy?

There’s one final takeaway from the webinar that’s worth keeping top of mind during any evaluation. Rob brought up the dilemma around open-weights models when talking about harmful content controls: “You read a lot about people being able to get recipes for poison or nuclear bombs, and building in a lot of guardrails, even the open source model providers. But the thing is, the moment you release such an open source model, it's relatively easy to fine-tune them, to sort of pervert those guardrails. And make the model give an answer to any question that you want.”

He concluded that, if a determined attacker can get the exact same answer in thirty seconds using a fine-tuned open model or a basic web search, the guardrail you spent months building didn't actually lower your real-world risk. It just protected you from a bad screenshot on social media.

He pointed to a piece of European legislation he has been working on that encodes this reasoning, requiring harmful content guardrails "unless people with bad intent can get that information easily elsewhere." He called it a useful rule precisely because it "prevents unnecessary security theater."

Apply the same logic to every control a vendor pitches you. Ask what an attacker's alternative path costs them if this control works perfectly. If the answer is thirty seconds and a swap to another model, you’re just paying for brand PR protection.

The controls that actually matter are the ones without a shortcut. An attacker can’t bypass your agent's strict permission boundaries. They can't fine-tune away your kill switch when an agent goes rogue. And they certainly can't download your internal customer database from anywhere else. Hard controls that limit what your agents are allowed to do and give you the power to pull the plug will retain their value no matter how fast open-source AI evolves.

Straiker answers all these questions

The nine questions above are ultimately asking the same thing: does a security control meaningfully reduce the risk created by an autonomous system, or does it simply make the environment look safer?

That is the standard we use at Straiker.

Our approach starts with the assumption that you cannot defend what you cannot see, test what you have not discovered, or contain an agent you cannot control. That is why our platform connects three parts of the security lifecycle: discover the agents in your environment, attack them the way a real adversary would, and enforce controls at runtime when something goes wrong.

Discover AI finds every agent, model, tool, MCP server, and coding agent across your environment, from AWS Bedrock and Azure AI Foundry to Claude Code, Cursor, and Codex on developer laptops. Within minutes of onboarding you get a map of every user, every agent, the models behind them, and the tools those agents can reach. That map is your blast radius, drawn.

Ascend AI attacks that inventory the way a real adversary would, running autonomous multi-turn campaigns for remote code execution, tool manipulation, data exfiltration, prompt injection, and lateral movement through MCP and connected tools. You watch each attack unfold live. Every run comes back mapped to OWASP, MITRE ATLAS, and NIST, and every finding becomes a runtime policy.

‍Defend AI enforces at runtime on every request, tool call, and agent-to-agent handoff, at 98.1% detection accuracy and under 300ms latency, with 6 to 21 times fewer false positives than frontier model judges. It deploys in minutes through a single hook-based integration by API, SDK, webhook, or AI sensor, with no proxies, thick clients, or infrastructure changes. And when an agent crosses the line, the Agentic Kill Switch takes it offline in seconds, with your security team making the call and audit-grade evidence on every action.

Sreenath's closing image from the webinar is the right way to think about the whole category:

"The AI genie is out in the open. I don't think we can put it back in. The genie's gonna get bigger and bigger. What we're gonna try to do here is maybe hold on to the genie's legs and hands and limit what they can do. And as long as we have some reasonable level of control there, we have a reasonable chance, but it's gonna be a good challenge."

Nobody is putting agents back in the bottle, and slowing your AI adoption to feel safer is a losing trade. What works is knowing every agent you run, testing each one the way an adversary would, and holding the power to stop any of them the instant it turns. When you can stop one agent in seconds, you never have to stop the business.

Ready to pressure-test this?

Book a demo and bring these nine questions with you. We will walk every one of them, including the ones where the honest answer is complicated. If you would rather start with your own environment, request a free AI risk assessment and we will show you what is actually running.

And if you want the full conversation that produced this list, watch the webinar.

Frequently Asked Questions About Agentic Security

How do you evaluate an AI security vendor during a POC?

Evaluate an AI security vendor against your own agent traffic, not only its demo environment. Verify that it can inspect prompts, reasoning, tool calls, tool responses, MCP traffic, and agent-to-agent handoffs; ask what happens after detection; and request true-positive rate, false-positive rate, latency, and evaluation methodology. Then test the product against documented agentic attacks.

What should an agentic AI security platform monitor?

An agentic AI security platform should monitor the full agentic trace, including the user prompt, agent reasoning, selected tool, tool arguments, tool response, and subsequent actions. It should also cover MCP traffic and agent-to-agent handoffs, because prompt and output monitoring alone cannot show how an agent reached a harmful action or what systems it can affect.

Are AI model guardrails enough for agentic AI security?

No. Model guardrails provide an important layer of protection for model inputs and outputs, but they generally lack the application's business context, including which tools an agent can access, what data it can reach, and which actions it is authorized to perform. Agentic security requires controls at the application and runtime layers as well.

What is an AI agent kill switch?

An AI agent kill switch is a runtime security control that lets an organization stop a compromised or misbehaving AI agent rather than simply blocking an individual action. Depending on the implementation, containment can include revoking tools, suspending the agent's session, freezing its memory, or stopping multiple affected agents at once.

Why isn't blocking a malicious AI agent action enough?

Blocking one malicious tool call does not necessarily remove the compromised context that caused the action. If an agent has been manipulated through prompt injection or another attack, it can continue operating and attempt a different path. Agent-level containment addresses that broader failure by stopping the compromised agent itself.

How do you measure AI security detection accuracy?

Detection accuracy should be evaluated alongside the dataset, false-positive rate, and latency used to produce it. A 99% detection rate on a narrow public benchmark does not establish how a system will perform against indirect prompt injection or real agent traffic. During a POC, vendors should be willing to measure performance against your environment and representative benign traffic.

What is the difference between AI security monitoring and AI runtime protection?

AI security monitoring identifies and records suspicious behavior, while runtime protection can intervene while the agent is operating. For autonomous agents, that distinction matters because an alert sent to a SOC may arrive after an agent has already accessed data, executed code, or taken another harmful action. Effective runtime protection can block the action or contain the agent.

What threats should agentic AI security protect against?

Agentic AI security should address threats tied to what an agent can actually do, including indirect prompt injection, remote code execution, data exfiltration, destructive commands, unauthorized tool use, malicious MCP interactions, lateral movement, and resource exhaustion. Content risks such as toxicity and policy violations can also matter, but they represent a different class of control.

What is the difference between AI guardrails and agentic security?

AI guardrails typically focus on controlling model inputs and outputs, such as toxicity, policy violations, hallucination, or sensitive information in responses. Agentic security extends that protection to the agent's behavior, tools, permissions, and actions, including the ability to block harmful operations and contain an agent when its context can no longer be trusted.

What should I look for in agentic AI runtime security specifically?

Look for these six essential capabilities in agentic AI runtime security:

• Full-trace visibility: Monitors prompts, reasoning, tools, and business context.
• Broad coverage: Inspects MCP and agent-to-agent traffic.
• Semantic detection: Understands meaning across formats rather than relying on pattern matching.
• Performance: Operates inline at production speed with verifiable false-positive rates.
• Active enforcement: Blocks malicious actions rather than just alerting.
• Agent containment: Can neutralize compromised agents directly.

Straiker’s Defend AI provides these capabilities, integrating with Discover AI for inventory and Ascend AI for continuous adversarial testing.