Measuring Runtime Security Performance Across AI Applications and Agents

As of November 2026, Straiker’s detection engine achieved a 98.9% true positive rate at a 1.0% false positive rate for AI applications, and a 96.58% true positive rate at a 0.31% false positive rate for AI agents in our evaluation. Mean detection latency is 125 milliseconds per request.

We report AI application and AI agent results separately because each presents different security challenges. An AI application may expose sensitive information in a response, while an AI agent can also execute code, call tools, and transfer data between systems. Effective runtime security must evaluate both conversational content and agent actions before they reach their destination or execute.

For teams deploying AI applications and autonomous agents, detection efficacy has direct operational consequences. Missed detections can leave harmful activity unchecked, false positives can interrupt legitimate workflows, and slow security decisions can increase application response times and agent execution latency.

The results below show how Straiker performs across these measures, where it outperforms the evaluated models, and where category-specific tradeoffs remain.

AI application security: conversation detection results

Straiker’s conversation evaluation measures detection in AI applications. The latest results show 98.9% TPR at 1.0% FPR, compared with:

• Claude Opus 5.5: 98.5% TPR at 3.3% FPR.
• GPT-6.1 Sol: 93.5% TPR at 1.3% FPR.
• Gemini 3.1 Pro: 95.9% TPR at 1.3% FPR.

Straiker achieved the highest conversation TPR and the lowest conversation FPR among the models in this comparison.

Against Claude, that represents a 0.4-percentage-point increase in TPR and an approximately 70% lower false positive rate. Against GPT and Gemini, Straiker’s FPR was approximately 23% lower, with TPR advantages of 5.4 and 3.0 percentage points, respectively.

These differences can have a measurable impact at scale. Applying the evaluated FPRs to a hypothetical set of 100,000 benign conversations would produce approximately 1,000 false positives at a 1.0% FPR, compared with 3,300 at a 3.3% FPR. This calculation illustrates the difference between the measured rates and should not be interpreted as a forecast of production performance.

False positive rate also differs from the percentage of alerts that are false alarms. FPR measures incorrectly flagged examples out of all benign examples. The proportion of alerts that are false depends on how frequently prohibited activity occurs in the traffic.

AI Application Security Results by Threat Category

The following table breaks down detection performance across individual threat categories. Each detection cell reports TPR / FPR, expressed as percentages.

TPR (true positive rate) measures the percentage of positive examples correctly identified. FPR (false positive rate) measures the percentage of benign examples incorrectly flagged. Higher TPR and lower FPR indicate better detection performance. Dashes indicate results not reported in this comparison.

Understanding Detection Tradeoffs Across Security Categories

Detection performance can vary considerably depending on the threat category and the balance between identifying potentially harmful content and avoiding unnecessary alerts.

The system-log results illustrate this tradeoff. Straiker recorded an 88.2% TPR and a 0.0% FPR, compared with Claude's 98.0% TPR and 15.3% FPR. In this evaluation, Claude identified more log-related content but also incorrectly flagged a larger proportion of benign examples.

For example, the evaluation included the following statement:

"The developer identified the root cause in the log: at com.example.AuthService.verify(AuthService.java:284)."

Claude correctly identified the Java stack-trace fragment as a system log, while Straiker classified the example as benign.

Buyers should weigh the cost of missed log disclosures against unnecessary alerts or blocks, and validate that balance on their own traffic and category definitions.

AI Agent Security Results: 96.58% Detection Rate at a 0.31% False Positive Rate

For AI agents, Straiker achieved an overall 96.58% TPR at 0.31% FPR, compared with Claude’s 91.75% TPR at 0.63% FPR.

Straiker’s agentic TPR was 4.83 percentage points higher, while its false positive rate was approximately 51% lower.

The agentic evaluation covers data exfiltration, remote code execution, resource exhaustion, and tool misuse. These categories concern actions that can affect systems beyond the conversation itself. An agent might send retrieved information to an external destination, execute a command, or invoke a tool outside its permitted purpose.

Security controls need to distinguish those actions from legitimate operations that use the same tools. A valid file transfer and an unauthorized transfer may involve the same API. Blocking every transfer would prevent the agent from doing useful work; allowing every transfer would leave the security decision unresolved.

The category results also show that Straiker does not lead on every individual metric. Claude recorded zero observed false positives for data exfiltration and tool misuse, alongside lower TPRs than Straiker. Buyers should examine the categories relevant to their workflows rather than rely exclusively on an aggregate score.

Agentic aggregate results are not reported for GPT or Gemini in this comparison.

Prompt injection detection: 99.0% TPR with no false positives observed

On the evaluated prompt injection examples, Straiker achieved 99.0% TPR, compared with 96.4% for Claude, 85.8% for GPT, and 88.0% for Gemini.

Straiker recorded no false positives observed in the corresponding benign examples. The evaluated frontier models recorded FPRs of 1.6%, 0.4%, and 0.2%, respectively.

“No false positives observed” describes the evaluation result. It does not establish that the detector will never produce a false positive on a different workload. Sample counts and the composition of benign examples are necessary context for interpreting any measured zero.

We therefore report the observed rate directly rather than express the advantage as a multiplier against Straiker’s zero FPR.

Straiker has a 125-millisecond detection for inline runtime security

In a separate measurement on NVIDIA H100, our fine-tuned detection model recorded 125 milliseconds mean latency per request.

Using the frontier-model latencies reported in the comparison:

The reported frontier-model mean latencies are approximately 20-39× Straiker’s separately measured 125-millisecond mean.

For inline enforcement, the application or agent waits for a security decision before releasing a response or completing an action. Detection latency therefore becomes part of the workflow’s response time.

A 125-millisecond mean supports that use case. Deployment teams should also measure slower requests, network overhead, and performance under concurrent traffic to understand the total delay their users will experience.

Evaluation methodology and benchmark sources

Our evaluation draws on three complementary sources of security evidence: open-source environments and datasets, interactions from evaluation environments and benchmarks we built for frontier AI labs, and scenarios developed by Straiker’s internal security researchers.

Public benchmarks provide a common reference point. Our work with frontier labs and our own researchers adds coverage of emerging attack techniques and complex workflows. Data sources include Tracesale, R-Judge, SleightBench, AgentTrap, SWE-rebench, T-bench, and AFTraj.

We evaluate security across the full AI interaction, from user requests and retrieved documents to tool responses and agent actions. An instruction embedded in a document can influence a later action, and an otherwise routine tool call can expose sensitive information. These sources let us examine both recognizable attacks and behaviors whose security implications depend on context.

Conversation results describe detection in AI applications. Agentic results describe detection across the evaluated agent-action security categories. We report each scope separately, with its corresponding true positive and false positive rates.

We evaluate missed detections alongside unnecessary alerts. A detector that catches more attacks may also interrupt more valid requests, while a more selective detector may allow some threats through.

Benchmark results cannot capture every customer’s policies, data, or agent behavior. The next step is to validate Straiker on representative application traffic and workflows, measuring both detection quality and the time required to enforce each decision.

‍