This is where the Cloud Security Alliance (CSA) has stepped in to fill a critical gap with their groundbreaking "Agentic AI Red Teaming Guide", released on May 28, 2025. This guide provides a comprehensive framework for red teaming agentic AI, offering a structured approach to testing for vulnerabilities that are unique to these advanced systems.
In parallel with the development of new red teaming methodologies, the industry is also focusing on building secure-by-design, agent-based systems. The Coalition for Secure AI (CoSAI) has been at the forefront of this effort, releasing its "Principles for Secure-by-Design Agentic Systems" in July 2025. These principles provide a foundational framework for ensuring that agentic AI is developed and deployed in a manner that is human-governed, resilient, transparent, and auditable. The CoSAI frameworks emphasize the importance of continuous validation and monitoring, ensuring that these systems remain secure and aligned with human values throughout their lifecycle.
This article will explore agentic AI red teaming, discussing the unique challenges these systems present and how the CSA's guide provides a roadmap for securing them. We will examine the key principles of secure-by-design agentic systems as laid out by CoSAI, and how these principles can be integrated into the development lifecycle.
Section 1: What makes agentic AI different
Agentic AI systems represent a fundamental departure from traditional AI models. While both are built on machine learning and data analysis, the key differentiator lies in their autonomy and ability to interact with the world. Traditional AI is largely reactive, responding to specific prompts and commands from users. Agentic AI, on the other hand, is proactive and goal-oriented. It can independently plan and execute a series of actions to achieve a high-level objective, even in the face of unforeseen circumstances. This autonomy is made possible by a unique set of capabilities that also introduces a new class of security vulnerabilities.
One of the most significant capability differences is the ability of agentic AI to use tools. These tools can range from simple APIs to complex software systems, enabling the agent to interact with both the digital and physical worlds. For example, an agent could use a web browser to research a topic, a code interpreter to execute a script, or a robotic arm to manipulate an object. This tool-use capability, while powerful, also creates a new attack surface. An attacker could potentially trick an agent into using a malicious tool or exploit a vulnerability in a legitimate tool to gain control of the agent.
Another key difference is the agent's memory. Unlike traditional AI models that have a limited context window, agentic AI systems can potentially maintain a persistent memory of their past interactions and experiences. This allows them to learn and adapt over time, but it also creates a new target for attackers. An attacker could potentially manipulate the agent's memory to alter its behavior or extract sensitive information that the agent has stored. The CSA's "Agentic AI Red Teaming Guide" highlights memory manipulation as one of the key threat categories for agentic AI.
Orchestration flaws are another critical vulnerability in agentic AI systems. These systems are often composed of multiple agents, each with its own set of capabilities and objectives. The orchestration layer is responsible for coordinating the actions of these agents to achieve a common goal. However, a flaw in the orchestration layer could allow an attacker to disrupt the system, or even turn the agents against each other. The CSA guide emphasizes the importance of testing for orchestration flaws, as they can have a cascading effect on the entire system.
In addition to these capability differences, the CSA has identified twelve threat categories of Agentic AI Systems:
Agent authorization & control hijacking: The threat of attackers issuing unauthorized commands or hijacking the agent’s decision-making process.
Checker-out-of-the-loop: The weakness where safety or oversight mechanisms fail to detect or stop unsafe agent behaviors.
Agent critical system interaction: The risks from unauthorized or harmful operations when the agent interacts with critical external systems.
Goal and instruction manipulation: The threat of attackers altering the agent’s goals or instructions to cause harmful or unintended actions.
Agent hallucination exploitation: The risk that the agent can be misled by false or fabricated information, causing incorrect actions.
Agent impact chain & blast radius: The potential for widespread downstream effects resulting from a single agent action in interconnected systems.
Agent knowledge base poisoning: The threat of adversaries corrupting or poisoning the agent's information sources.
Agent memory & context manipulation: The risk of tampering with the agent’s memory or state to persistently exploit or compromise it.
Multi-agent exploitation: The threats arising from compromised inter-agent trust, collusion, or manipulation among multiple agents.
Resource & service exhaustion: The threats that exhaust computational, memory, or service resources to degrade or disrupt agent operation.
Supply chain & dependency attacks: The risks from vulnerabilities in third-party dependencies or integrations leading to indirect compromise.
Agent untraceability: The threat of actions being hidden or untraceable due to insufficient logging or auditability.
These threat categories highlight the unique security challenges that agentic AI systems present. They also underscore the need for a new approach to red teaming that is specifically designed for these advanced systems.
Section 2: Red teaming methodology for agentic systems
Based on CSA’s "Agentic AI Red Teaming Guide," a structured methodology is essential for identifying and mitigating the unique security vulnerabilities of agentic AI systems. This approach moves beyond traditional penetration testing by directly addressing the autonomous, non-deterministic, and complex nature of these systems. The methodology is organized into a comprehensive framework that applies a systematic process of preparation, execution, analysis, and reporting across 12 critical threat categories.
Core methodological principles
The effectiveness of this methodology rests on three core principles. First is a dedicated focus on the expanded attack surface. Testing is not confined to the AI model's inputs and outputs; it extends to the entire operational ecosystem. This includes the agent's control system (its decision-making engine), its internal and external knowledge bases, its core goals and instructions, and its interactions with external systems via APIs, databases, and other agents.
Second, the approach is fundamentally actionable and test-driven. Rather than engaging in high-level conceptual discussions, the guide provides red teamers with concrete, step-by-step procedures and example prompts designed to simulate real-world attacks. Finally, the methodology must be continuous and proactive. Because agentic systems can exhibit emergent, unforeseen behaviors over time, red teaming cannot be a one-time event. It must be an ongoing function, performed both pre- and post-deployment, to ensure the system remains secure as it evolves.
A Deeper look at testing the 12 threat categories
The core of the methodology is its application across 12 key vulnerability areas. The guide provides detailed testing requirements for each, transforming abstract risks into tangible test cases. Here are specific examples of how testing is conducted:
Testing agent authorization and control hijacking: This goes beyond simple access control checks. Red teamers will use API testing tools, such as Postman or Burp Suite, to directly inject malicious commands into the agent's control interface and verify if unauthorized actions are executed. This also involves simulating spoofed control signals or manipulating authentication headers to test the agent's ability to reject commands from untrusted sources. A key test for permission escalation involves assigning the agent a task requiring temporarily elevated rights, and then verifying that those rights are fully and immediately relinquished upon task completion, preventing their use for subsequent, unauthorized actions.
Testing goal and instruction manipulation: The objective here is to subvert the agent's intended purpose. Testers will craft ambiguous instructions using homonyms or complex phrasing to probe for vulnerabilities in the agent's natural language understanding. A more advanced technique is "Recursive Goal Subversion," where a sequence of seemingly benign intermediate instructions is given to gradually steer the agent away from its primary mission. The success of the attack is measured by whether the agent deviates from its original objective or seeks clarification.
Testing agent knowledge base poisoning: This involves a multi-pronged attack on the agent's sources of truth. Testers will introduce intentionally biased or malicious data directly into the training datasets to see if it skews the agent's behavior. They will also manipulate external data sources that the agent relies on (e.g., a compromised third-party API) to feed it misleading information in real-time. Finally, they will attempt to directly corrupt the agent's internal knowledge base to test its integrity-monitoring and rollback capabilities.
Testing multi-agent exploitation: In systems with multiple agents, the focus shifts to trust and communication. Red teamers will simulate a man-in-the-middle attack to intercept and alter communications between agents. They will also test trust relationship abuse by compromising one agent and using its valid credentials to issue unauthorized commands to its peers. The resilience of the system is evaluated by its ability to detect anomalous coordination patterns, spoofed identities, or collusive behavior between agents.
This same level of detailed, hands-on testing is applied across all other categories, from exploiting agent hallucinations to assessing the blast radius of a cascading failure.
The four-phase testing process in practice
The execution of these tests is governed by a structured, four-phase process to ensure rigor and deliver actionable results.
Preparation: This is the strategic planning phase. It involves more than just defining scenarios; red teamers will map the agent's expected permissions, identify specific API endpoints and data sources to target, and configure monitoring tools to capture detailed logs of the agent's activity during the tests.
Execution: In this phase, red teamers actively carry out the planned attacks. This is an interactive process of injecting the crafted inputs or simulating the compromised conditions, then meticulously observing the agent's response. Every action, error message, and unexpected behavior is documented in real-time to provide a clear chain of evidence.
Analysis: The raw data from the execution phase is translated into findings. The analysis involves correlating the red team's actions with the agent's logged behavior to identify the root cause of a vulnerability. For example, if an agent repeatedly accepted a malicious command, the analysis would pinpoint a failure in its input validation logic. Findings are then prioritized based on their exploitability and potential business impact.
Reporting: The final phase delivers the value of the engagement. The report provides more than a list of vulnerabilities; it offers a detailed narrative of the attack paths used, the evidence of their success, and, most importantly, actionable mitigation strategies. These recommendations are designed to provide developers and system architects with clear guidance on how to harden the agent's defenses, such as implementing stricter permission controls, enhancing anomaly detection, or improving the integrity of its knowledge sources.
Section 3: Tool illustration: AI red teaming
The theoretical understanding of agentic AI vulnerabilities and red teaming methodologies is crucial; however, practical tools are necessary to implement these concepts effectively. Snyk, a leader in developer security, has stepped up to this challenge with its innovative Snyk AI Red Teaming tool.. This tool, developed by Snyk Labs, tests the security of LLM agents and provides a practical solution for developers and security teams to identify and mitigate prompt-based risks.
It is built on the principle of automating the red teaming process, making it easier for developers to integrate security testing into their existing workflows. It automates the generation of adversarial prompts, which are designed to trick the agent into performing an unintended action. This eliminates the need for developers to manually craft these prompts, saving them time and effort. The prototype also provides a simulation-based testing environment, allowing developers to test their agents in a safe and controlled environment before they are deployed to production.
Now available on Snyk Labs to anyone with a free account, it is highly customizable, allowing developers to tailor the testing process to their specific needs. They can define their own threat models, create their own adversarial prompts, and configure the simulation environment to match their production environment. This flexibility makes the prototype a powerful tool for red teaming a wide range of agentic AI systems.
Snyk's AI Red Teaming is a prime example of how the industry is responding to the security challenges of agentic AI. By providing a practical and automated solution for red teaming, Snyk helps ensure that these advanced systems are developed and deployed in a secure and responsible manner. The prototype is a valuable tool for any organization developing or using agentic AI, and it serves as a clear indication of the direction the industry is heading in terms of AI security.
Section 4: Embedding red teaming into AI development lifecycles
The security of agentic AI systems cannot be an afterthought; it must be an integral part of the development lifecycle. The traditional approach of performing security testing at the end of the development process is not sufficient for these complex and dynamic systems. Instead, red teaming must be embedded into the AI development lifecycle, from the initial design phase to the final deployment and beyond.
One of the key ways to embed red teaming into the development lifecycle is to integrate it with the continuous integration and continuous delivery (CI/CD) pipeline. This involves automating the red teaming process and integrating it into the regular build and deployment process. By doing this, developers can get immediate feedback on the security of their agents, and they can identify and mitigate vulnerabilities before they are deployed to production.
Another important aspect of embedding red teaming into the development lifecycle is to create a feedback loop for prompt-based regression testing. This involves collecting the prompts that are sent to the agent in production and using them to create a regression test suite. This test suite can then be used to test the agent for any new vulnerabilities that may have been introduced during the development process.
The CSA guide recommends that organizations collect all of the prompts that are sent to their agents and that they use these prompts to create a corpus of real-world data. This corpus can then be used to train a machine learning model that identifies new and emerging threats.
Finally, it is important to integrate the outcomes of the red teaming process into the organization's policy enforcement and guardrails. This involves using the results of the red teaming tests to create new security policies and update existing ones.
For example, suppose a red teaming exercise exposes that an AI agent can be tricked into sharing sensitive customer data when prompted with a cleverly disguised request. In response, the organization can implement a policy that requires all AI agents to automatically flag and block requests that attempt to extract sensitive information outside of authorized channels. Additionally, the organization can update the agent's guardrails to include stricter validation checks on user prompts, logging any suspicious attempts, and alerting security teams for further investigation. This ensures that vulnerabilities identified through red teaming are addressed systematically and that the agent operates within safe and compliant boundaries.
Conclusion
Agentic AI represents a new frontier in artificial intelligence, with the potential to revolutionize a wide range of industries. However, this new technology also presents a new set of security challenges that must be addressed. The CSA's "Agentic AI Red Teaming Guide" and CoSAI's "Principles for Secure-by-Design Agentic Systems" provide a good reference framework for securing these advanced systems.
In addition to these frameworks, the industry is also developing innovative tools to automate and enhance the red teaming process. Snyk's LLM Agent Red Teaming Prototype is a prime example of this trend, as it provides a practical and automated solution for identifying and mitigating prompt-based risks.
As the agentic AI ecosystem continues to mature, we can expect to see more innovation in the area of AI security. By working together, we can ensure that agentic AI is developed and deployed in a way that is safe, secure, and beneficial to society.
References
Cloud Security Alliance. (2025, May 28). Agentic AI Red Teaming Guide. Retrieved from https://cloudsecurityalliance.org/artifacts/agentic-ai-red-teaming-guide
Coalition for Secure AI. (2025, July 16). Announcing the CoSAI Principles for Secure-by-Design Agentic Systems. Retrieved from https://www.coalitionforsecureai.org/announcing-the-cosai-principles-for-secure-by-design-agentic-systems/
Snyk. (2025, June 24). Snyk Acquires Invariant Labs to Accelerate Agentic AI Security Innovation. Retrieved from https://snyk.io/news/snyk-acquires-invariant-labs-to-accelerate-agentic-ai-security-innovation/
Snyk Labs. (2025, May 28). Red Team Your LLM Agents Before Attackers Do. Retrieved from https://labs.snyk.io/resources/red-team-your-llm-agents-before-attackers-do/
