How to Penetration Test AI Agents: The Essentials of Understanding AI Security Testing

epth Security Home

Penetration Testing

Penetration Testing
Industries
Case Studies
Company
Resources
Contact Us

In the Eye of the Cyber Storm: Shielding Financial Institutions from Online Threats

https://depthsecurity.com/

In the Eye of the Cyber Storm: Shielding Financial Institutions from Online Threats

H3

H4

H5
H6
Virtual display on table of AI cybersecurity guards

AI agents are becoming capable of much more than generating text. They can retrieve information, interact with applications, access files, call APIs, execute tasks, and make decisions across multi-step workflows.

That added functionality creates a new security question for organizations adopting agentic AI: Can an AI agent be tested like any other application?

The answer is yes, but the testing process has to account for something traditional application testing does not. That the agent can interpret instructions, make decisions, use tools, and change its behavior based on the information it encounters, which creates an attack surface that extends beyond the underlying application.

Testing More Than the Application

A traditional penetration test examines how an application responds to malicious input, unauthorized requests, unexpected behavior, and attempts to bypass security controls. AI agents still need those same fundamentals tested, but the difference is that an agent may sit between the user and multiple systems.

An internal AI assistant, for example, could have access to company documents, databases, email, ticketing systems, or business applications. Each connection gives the agent additional capabilities and potentially gives an attacker another way to influence what the agent does.

Those capabilities create several distinct areas for security testing. OWASP’s 2026 Top 10 for Agentic Applications outlines ten risks that demonstrate how an agent’s instructions and connected systems can become targets for abuse:

  • Agent Goal Hijack – attacks manipulate an agent’s core objectives so it pursues malicious outcomes
  • Tool Misuse & Exploitation – an agent applies authorized tools in a destructive way
  • Identity & Privilege Abuse – attacks exploit cached tokens or inherited human roles to escalate privileges
  • Agentic Supply Chain Vulnerabilities – runtime workflows load compromised third-party plugins, tools or prompt templates with hidden malicious payloads
  • Unexpected Code Execution – flawed agent planning or prompt manipulation causes unsafe generation and execution of external codes
  • Memory & Context Poisoning – attacks corrupt short-term or long-term memory shares and RAG databases
  • Insecure Inter-Agent Communication – multi-agent systems exchange messages without authentication or integrity
  • Cascading Failures – a minor error or single compromised task in a multi-agent network propagates and amplifies into massive disruptions
  • Rogue Agents – compromised autonomous agents diverge from their intended purpose
  • Human-Agent Trust Exploitation – attackers leverage an agent’s authority output to engineer human users into granting unsafe approvals

A security assessment therefore has to look beyond whether the AI produces an appropriate response. It needs to examine what happens when the system is deliberately pushed outside its intended behavior.

What Happens When the Agent Receives Malicious Instructions?

One of the most important areas to test is prompt injection. In a simple chatbot, prompt injection might cause the model to ignore instructions or produce an unexpected response. An agent with access to external systems presents a different level of risk.

Consider an AI agent that reviews incoming documents. An attacker could place malicious instructions inside a document knowing that the agent will process it. The document might appear harmless to an employee. The agent, however, could interpret the embedded instructions as something it should follow.

This is known as indirect prompt injection, and NIST has specifically highlighted it as a security concern for agents that process external content such as emails, websites, documents, and code repositories.

Testing the Agent's Tools

AI agents become significantly more useful when they can call external tools. An agent might have tools for:

  • Searching internal documents
  • Querying a database
  • Accessing a CRM
  • Creating support tickets
  • Sending email
  • Retrieving files
  • Calling APIs
  • Executing code

Those tools may be completely legitimate and properly secured when used directly by an authorized employee. The security concern is whether an attacker can manipulate the agent into using those capabilities in an unintended way. OWASP identifies tool misuse and exploitation as one of the primary risks facing agentic applications.

How Much Access Does the Agent Really Have?

Permissions become especially important when testing AI agents. An organization may carefully restrict what individual employees can access, while an AI agent could be given broad permissions to make it useful across multiple workflows.

That creates a potential problem. If an attacker can manipulate the agent, the attacker may be able to benefit from the access already assigned to it. As stated above, OWASP’s agentic security framework specifically identifies identity and privilege abuse as a major risk category. A penetration test should therefore examine the agent’s effective permissions rather than simply asking whether authentication is enabled.

Testing can evaluate and answer:

  • What information can the agent access?
  • Which systems can it interact with?
  • Which actions can it perform?
  • Does it inherit a user’s permissions?
  • Can it access information belonging to another user?
  • Can it perform privileged operations?
  • Are sensitive actions subject to additional authorization?

The goal is to understand what happens when legitimate access is placed under malicious control.

Can the Agent Be Turned Into an Attack Path?

This is where AI penetration testing becomes particularly interesting. An individual weakness may have limited impact by itself and the risk can increase when multiple capabilities connect.

For example:
Malicious input → AI agent → internal search tool → sensitive document → external action

The agent may not contain a traditional vulnerability in the conventional sense. Instead, the problem could come from the way its instructions, permissions, tools, and data interact.

Recent research is beginning to approach AI agents from this perspective. A July 2026 study introduced a reconnaissance-driven approach to AI-agent penetration testing, arguing that testers can systematically identify an agent’s knowledge, capabilities, and weaknesses before constructing stronger attacks.

That approach closely resembles how penetration testers already evaluate complex application environments: understand the target, identify available paths, determine what can be influenced, and demonstrate the resulting impact.

Testing AI Agents From an Attacker's Perspective

AI security testing needs to examine the difference between what an agent says it will do and what it actually does. An agent might refuse a malicious request in its response while still taking an unauthorized action through a connected tool which could describe a dangerous action without actually carrying it out.

For penetration testers, testing needs to determine whether adversarial input can produce a meaningful change in the system or lead the agent to take an action outside its intended boundaries.

That means testing questions such as:

  • Can sensitive information be accessed or exposed?
  • Can the agent perform an action the user did not authorize?
  • Can it access systems or data outside its intended scope?
  • Can it execute code or trigger another tool?
  • Can it modify or delete data?
  • Can one compromised capability create a path to another resource?

NIST’s recent AI-agent red-team research demonstrates why this approach matters. Its testing found that agents processing external information could be manipulated through indirect prompt injection into taking unintended actions, including attempts to access sensitive information or execute malicious code.

As AI agents gain access to more business systems, security assessments need to establish what an agent is designed to do and then deliberately test what happens when an attacker tries to push beyond those boundaries.

Putting AI to the Test

AI agents are becoming part of the applications, workflows, and infrastructure organizations depend on. Their ability to interpret information and act across connected systems makes them useful, but those same capabilities can create security risks that conventional application testing may not fully address.
Penetration testing provides a way to examine those risks from an attacker’s perspective.

The objective is straightforward: identify how an AI agent can be manipulated, determine what access and capabilities are exposed, and demonstrate what an attacker could accomplish before those weaknesses become an avenue for compromise. As AI moves from answering questions to taking action, testing what it can be convinced to do should become part of the security conversation.

Your AI Agent Has a Job. Ours Is to Break It.

AI agents are becoming more capable, more connected, and more trusted. That makes understanding their attack surface increasingly important. Depth Security can assess AI agents from an attacker’s perspective, testing how malicious input, permissions, tools, and connected systems can be manipulated to produce real-world impact.