From Smart to Secure: Why AI Applications Need to Be Continuously Tested

Red teaming exercises, particularly penetration testing in application security (AppSec), have long been the gold standard for identifying vulnerabilities. Security professionals craft manual and automated attacks on applications to uncover known and unknown issues. These applications were typically deterministic systems with well-defined interfaces (though occasionally undocumented), allowing for a rules-based approach to security, guided by human creativity and experience. For most organizations, regular penetration testing serves as a checkbox for compliance and is often metaphorically pinned on the office fridge alongside family photos.

However, the advent of artificial intelligence (AI) has dramatically transformed the landscape of red teaming and the assessment of risk and, in the future, could greatly complicate how we define compliance.

It’s worth stepping back and considering how we need to rethink our approach to penetration testing. AI applications don’t just expand the attack surface, they change how that surface behaves, evolves, and hides vulnerabilities. As a result, the future of red teaming isn’t about frequency or scope, it’s about continuity.

TL;DR:  5 reasons AI apps need continuous risk assessments 

1. AI apps behave probabilistically, which can make the same inputs yield different outputs over time.
2. Retrieval-Augmented Generation (RAG) and agentic architectures introduce constantly shifting data and behavior.
3. Continuous red teaming with AI agents is now essential to stay ahead of threats.
4. Security in the AI age must evolve from static testing to real-time, adaptive risk assessments.
5. Future-proof defenses will blur attack and defense, learning from every simulated attack.

The Dynamic Nature of AI Applications

AI applications don’t just follow a script; they respond, adapt, and even learn in real time. They’re built on probabilistic reasoning, not fixed logic. This means that running the same red team assessment twice, even minutes apart, can result in different behaviors. Even if you haven’t changed the code or model version, the AI might still behave differently depending on user interactions, context windows, and the data it’s drawing from.

Take RAG, for example. It’s one of the most common architectures for production AI applications today, where the model references an external dataset to ground its responses in accurate, often domain-specific knowledge. In such instances, the data may be constantly changing. In many real-world scenarios, the contents of the RAG can shift hourly or even more frequently.  In some cases, there’s no clear visibility into what’s going into the RAG at any given time.

RAG Poisoning: A Simple Case Study

Let’s take a look at a basic example of RAG poisoning, a technique to manipulate AI responses by misleading or maliciously corrupting the knowledge database that the RAG pulls from.

The RAG we tested has sensitive information about individuals, including names, addresses, and credit card data. Attempts to extract information through role-play or convincing the large language model (LLM) that the data is fake fail as expected due to the inbuilt model guardrails.

However, some aspects of the data within our RAG are within the control of users, like the “address” field, and are typically unchecked by filters such as regular expressions. A potential attacker could use this as an opportunity to poison the RAG.

An attacker, or in our case, our research team, issues the identical prompt, but this time, the data extracted from the RAG includes a reinforcing prompt, which is enough in this instance to convince the LLM that the prompt is correct in suggesting that the data is fake. 

RAG is just one example. As AI matures and applications become more autonomous, things get more unpredictable and complex. We’re already seeing the early stages of this with Model Context Protocol (MCP), which aims to provide a standardized interface for applications to control and coordinate AI models, including setting goals, managing context, and invoking external tools. 

This is where traditional, point-in-time security assessments fall flat. The visible threat landscape around LLM-powered AI apps is shifting at the same breakneck speed as the technology. From basic jailbreaks to indirect prompt injection and now multi-modal and agentic attacks, defending against these requires a dedicated solution that must continuously learn, simulate, and assess risks as they evolve.

AI security needs its own real-time immune system trained on the latest attacks, infused with insights from continuous learning, and embedded directly into your defenses.

Evolving Red Teaming Strategies with AI

Today, we can use AI to test AI. Assessing AI applications isn’t just about throwing known bad prompts at systems. Employing AI agents, we can carry out multi-turn conversations and adjust tactics and strategies in real time to replicate real attackers. These “white-hat” agents learn from every attempt, making each test more effective than the last, testing the application across all layers beyond the model itself, exploring agent orchestration, web, identity, data, and user behavior.

Navigating Agentic Complexity: A New Security Wildcard

Agentic apps don’t just work on their own; they are multimodal, pulling data from everywhere, using text, images, and video, and even talking to other agents. This level of autonomy creates something new: a system that constantly shifts, adapts, and changes direction in unpredictable ways.

Imagine an agentic AI system designed to automate research, aggregation, and content generation. It pulls from proprietary databases, external APIs, real-time web scraping, and other AI models. Now, consider another AI agent designed for summarization, feeding processed data back into the system. Suddenly, not only are you dealing with the inherent unpredictability of a single AI model but also the emergent properties of multiple interconnected agents interacting in ways no developer could fully anticipate.  

With the emergence of the MCP, this complexity compounds.  Standardized interfaces bring huge advantages to agentic applications and make it easier for agents to connect to tools and each other dynamically.  However, this innovation is a double-edged sword, enabling rapid expansion of both the attack surface and the blast radius: a single compromised tool or crafted indirect prompt can cascade through interconnected agents, triggering actions, spreading misinformation, and corrupting behavior at scale.

Conclusion

AI security is entering a new era, where static defenses and point-in-time penetration tests are no longer sufficient. Protecting against AI threats requires us to rethink security, one where change and threats are constant, and defenses that adapt in real time.

Just as modern cloud security embraces continuous monitoring and automated remediation, AI security will need to blur the line between attack and defense, leveraging continuous risk assessment insights to shape how AI applications protect themselves.

The arms race is already well underway. Those who evolve now, embedding continuous assessment and dynamic guardrails into their infrastructure, won’t just survive; they’ll lead. Get your complementary AI assessment with Straiker now.