Runtime Security for AI Applications and AI Agents: Straiker’s Detection Results

As of November 2026, Straiker’s detection engine achieved a 98.9% true positive rate at a 1.0% false positive rate for AI applications, and a 96.58% true positive rate at a 0.31% false positive rate for AI agents in our evaluation. Mean detection latency is 125 milliseconds per request.

We report these results separately because conversations and agent actions present different security problems. An AI application may expose sensitive information in a response. An AI agent can also execute code, call tools, or transfer data to another system. Its security controls must evaluate those actions before they proceed.

For teams deploying runtime security, detection efficacy has direct operational consequences. Missed detections leave harmful activity unchecked. False positives interrupt legitimate work. Slow decisions add time to application responses and agent workflows.

The results below show how Straiker performs across these measures, where it outperforms the evaluated models, and where category-specific tradeoffs remain.

AI application security: conversation detection results

Straiker’s conversation evaluation measures detection in AI applications. The latest results show 98.9% TPR at 1.0% FPR, compared with:

• Claude Opus 5.5: 98.5% TPR at 3.3% FPR.
• GPT-6.1 Sol: 93.5% TPR at 1.3% FPR.
• Gemini 3.1 Pro: 95.9% TPR at 1.3% FPR.

Straiker achieved the highest conversation TPR and the lowest conversation FPR among the models in this comparison.

Against Claude, that represents a 0.4-percentage-point increase in TPR and an approximately 70% lower false positive rate. Against GPT and Gemini, Straiker’s FPR was approximately 23% lower, with TPR advantages of 5.4 and 3.0 percentage points, respectively.

The operational difference becomes clearer at volume. Applying the measured rates to a hypothetical set of 100,000 benign conversations would produce approximately 1,000 false positives at a 1.0% FPR, compared with 3,300 at a 3.3% FPR. This illustrates the rates; it is not a forecast of production performance.

False positive rate also differs from the percentage of alerts that are false alarms. FPR measures incorrectly flagged examples out of all benign examples. The proportion of alerts that are false depends on how frequently prohibited activity occurs in the traffic.

The full category results are below. Each detection cell reports TPR / FPR, in percent.

TPR = true positive rate: the percentage of positive examples correctly identified. FPR = false positive rate: the percentage of benign examples incorrectly flagged. Higher TPR and lower FPR are better. Dashes indicate results not reported in this comparison.

The system-log results illustrate a tradeoff between broader detection and fewer false alerts. Straiker flagged fewer examples as system logs, while Claude recognized more log-related content.

For example, the evaluation included: “The developer identified the root cause in the log: at com.example.AuthService.verify(AuthService.java:284).” Claude correctly identified the Java stack-trace fragment as a system log, while Straiker returned benign.

Buyers should weigh the cost of missed log disclosures against unnecessary alerts or blocks, and validate that balance on their own traffic and category definitions.

AI agent security: detecting exfiltration, code execution, and tool misuse

For AI agents, Straiker achieved an overall 96.58% TPR at 0.31% FPR, compared with Claude’s 91.75% TPR at 0.63% FPR.

Straiker’s agentic TPR was 4.83 percentage points higher, while its false positive rate was approximately 51% lower.

The agentic evaluation covers data exfiltration, remote code execution, resource exhaustion, and tool misuse. These categories concern actions that can affect systems beyond the conversation itself. An agent might send retrieved information to an external destination, execute a command, or invoke a tool outside its permitted purpose.

Security controls need to distinguish those actions from legitimate operations that use the same tools. A valid file transfer and an unauthorized transfer may involve the same API. Blocking every transfer would prevent the agent from doing useful work; allowing every transfer would leave the security decision unresolved.

The category results also show that Straiker does not lead on every individual metric. Claude recorded zero observed false positives for data exfiltration and tool misuse, alongside lower TPRs than Straiker. Buyers should examine the categories relevant to their workflows rather than rely exclusively on an aggregate score.

Agentic aggregate results are not reported for GPT or Gemini in this comparison.

Prompt injection detection: 99.0% TPR with no false positives observed

On the evaluated prompt injection examples, Straiker achieved 99.0% TPR, compared with 96.4% for Claude, 85.8% for GPT, and 88.0% for Gemini.

Straiker recorded no false positives observed in the corresponding benign examples. The evaluated frontier models recorded FPRs of 1.6%, 0.4%, and 0.2%, respectively.

“No false positives observed” describes the evaluation result. It does not establish that the detector will never produce a false positive on a different workload. Sample counts and the composition of benign examples are necessary context for interpreting any measured zero.

We therefore report the observed rate directly rather than express the advantage as a multiplier against Straiker’s zero FPR.

125-millisecond detection for inline runtime security

In a separate measurement on NVIDIA H100, our fine-tuned detection model recorded 125 milliseconds mean latency per request.

Using the frontier-model latencies reported in the comparison:

Measurement note: Straiker’s latency was measured separately on NVIDIA H100. Ratios are calculated from the reported mean timings; they should not be interpreted as a controlled, like-for-like performance comparison. Observed latency depends on workload, serving configuration, and what the measurement includes.

The reported frontier-model mean latencies are approximately 20-39× Straiker’s separately measured 125-millisecond mean.

For inline enforcement, the application or agent waits for a security decision before releasing a response or completing an action. Detection latency therefore becomes part of the workflow’s response time.

A 125-millisecond mean supports that use case. Deployment teams should also measure slower requests, network overhead, and performance under concurrent traffic to understand the total delay their users will experience.

Evaluation methodology and benchmark sources

Our evaluation draws on three complementary sources of security evidence: open-source environments and datasets, interactions from evaluation environments and benchmarks we built for frontier AI labs, and scenarios developed by Straiker’s internal security researchers.

Public benchmarks provide a common reference point. Our work with frontier labs and our own researchers adds coverage of emerging attack techniques and complex workflows. Data sources include Tracesale, R-Judge, SleightBench, AgentTrap, SWE-rebench, T-bench, and AFTraj.

We evaluate security across the full AI interaction, from user requests and retrieved documents to tool responses and agent actions. An instruction embedded in a document can influence a later action, and an otherwise routine tool call can expose sensitive information. These sources let us examine both recognizable attacks and behaviors whose security implications depend on context.

Conversation results describe detection in AI applications. Agentic results describe detection across the evaluated agent-action security categories. We report each scope separately, with its corresponding true positive and false positive rates.

We evaluate missed detections alongside unnecessary alerts. A detector that catches more attacks may also interrupt more valid requests, while a more selective detector may allow some threats through.

Benchmark results cannot capture every customer’s policies, data, or agent behavior. The next step is to validate Straiker on representative application traffic and workflows, measuring both detection quality and the time required to enforce each decision.