AI penetration testing for agents in regulated industries
What a penetration test of an AI agent should cover at a bank, insurer or hospital, how to run it safely, and the evidence that gets the agent approved.
A penetration test of an AI agent has to test what the agent can reach and do, not only what the model says. An agent reads content your team does not control, calls tools with real credentials and acts on its own. At a bank, an insurer or a hospital, security, risk and compliance teams must approve it before it ships. They approve faster when the test gives them evidence, not a list.
This guide covers what to test, how to run the test safely in a regulated environment, and what the report should contain so reviewers can say yes.
Key takeaways
- Test the agent, not only the model: its job, the content it reads, its tools, its identity and its memory.
- Use the OWASP Top 10 for Agentic Applications as the test plan, and record results against MITRE ATLAS and the control framework your reviewers use.
- Attack an isolated copy with written authorization. Nothing should run against production.
- Each finding needs a fix, a retest and a replay: the attack fails, and the agent still does its job.
- Retest when the agent or the attacks change. A point-in-time report goes stale fast.
Why agents need their own penetration test
A web application test checks inputs, sessions and access control. A model evaluation checks what the model says. An agent combines both and adds autonomy: it plans, keeps memory, calls tools and acts with delegated authority.
That changes where attacks come from. The most damaging input is often not typed by a user. It arrives in a document the agent summarizes, an email it triages, a web page it reads or the output of another tool. If the agent treats that text as instructions, an attacker can steer it without touching your network.
Pentest data shows the gap. In Cobalt's 2026 State of Pentesting report, 32% of AI and LLM pentests had high-risk findings, against 12% across all pentests, and AI findings had the lowest resolution rate of any category. Cobalt's AI Pentesting Pulse puts the median time to resolve a high-risk AI finding at 36 days in 2026, up from 19 days in 2025. Finding issues is not the bottleneck. Fixing them safely is.
What to test
The current edition, the OWASP Top 10 for Agentic Applications for 2026, is the clearest public test plan for agents. Use it to scope the test, then write each case around the agent's real job.
| Risk | Example in a regulated workflow | What to test |
|---|---|---|
| ASI01 Agent goal hijack | Hidden text in a loan application tells a review agent to approve it | Indirect prompt injection through every document, email and web source the agent reads |
| ASI02 Tool misuse | A service agent issues refunds or changes account details outside policy | Each tool's parameters, limits and approval rules |
| ASI03 Identity and privilege abuse | The agent's service account can read every customer's records | Which identity the agent uses, and whether it can act beyond the user it serves |
| ASI04 Agentic supply chain | A third-party MCP server or plugin returns malicious instructions | Every server, plugin and model the agent depends on |
| ASI05 Unexpected code execution | An analytics agent runs code that an attacker supplied | Sandboxing, and what code the agent may run |
| ASI06 Memory and context poisoning | A planted note in memory changes later decisions | What the agent stores, and who can write to it |
| ASI07 Insecure inter-agent communication | One agent passes forged instructions to another | Trust and message integrity between agents |
| ASI08 Cascading failures | One bad step triggers a chain of actions | Limits, timeouts and rollback |
| ASI09 Human-agent trust exploitation | The agent persuades a reviewer to approve a risky action | What reviewers see before they approve |
| ASI10 Rogue agents | An agent keeps acting outside its task | Monitoring, and the ability to stop it |
For regulated data, add explicit leakage cases. Can an attacker make the agent reveal account numbers, health records, privileged legal material or credentials, either in a reply or through a tool call that sends data out?
How to run the test safely
Regulated teams cannot test an agent the way a researcher tests a public chatbot. Set the rules before the first attack:
- Written authorization and a fixed scope: which agent, which tools, which data and which attack classes.
- An isolated copy of the agent with the same code, prompts, tools and permissions, connected to test systems and synthetic or masked data.
- No production execution. An attack that needs a real system runs against a stand-in.
- A record of every attack: the input, each step the agent took and the outcome, so a reviewer can reproduce it.
The evidence that gets an agent approved
Reviewers do not need a longer list of findings. They need to know which risks are real, what changed and whether the agent still works. For each finding, the report should include:
- Reproduction: the attack, the steps and the impact, shown on the isolated copy.
- The smallest fix: a permission, a tool change, a code change or a runtime rule, for your team to approve.
- A retest that shows the same attack now fails.
- A replay of the agent's approved tasks that shows the work still runs. A fix that breaks the business gets rolled back.
- A mapping to the frameworks your reviewers use: OWASP, MITRE ATLAS, the NIST AI Risk Management Framework, ISO/IEC 42001 and AIUC-1.
For AIUC-1, OWASP published a crosswalk to the agentic Top 10 in May 2026. It links each Top 10 risk to the matching AIUC-1 requirements. For the evidence that ISO/IEC 42001 and AIUC-1 audits ask for, see AIUC-1 and ISO 42001 for AI agents. For a crosswalk from the agentic Top 10 to MITRE ATLAS and the NIST AI RMF, see Mapping AI agent risks to MITRE ATLAS and the NIST AI RMF.
This package answers the questions reviewers and regulators ask: was the agent tested before deployment, what can it access, who does it act for, what does it record and how is it monitored? The EU AI Act, for example, requires high-risk AI systems to resist attempts to alter their use or outputs by exploiting vulnerabilities (Article 15). Under the Digital Omnibus on AI, these obligations apply from December 2, 2027 to the high-risk uses in Annex III, which include credit scoring and life and health insurance pricing.
Why one test is not enough
Agents change fast. A new tool, a new prompt or a new model version can open a path that last quarter's test never saw, and attack techniques change too. AIUC-1, a certification standard written for AI agents, asks for testing before certification and quarterly retests after it.
Treat the penetration test as the first run, not the last. Retest on every meaningful change. Between tests, enforce the approved rules at runtime and keep a record of every action that was allowed, held for approval or denied.
Questions to ask a provider
- Do you test our agent's real job, tools and permissions, or a generic chatbot?
- Do you reproduce every finding, and drop the ones you cannot reproduce?
- Does each finding come with a fix we can approve, and a retest?
- How do you show that a fix did not break approved workflows?
- Where does the testing run? Can it stay in our cloud or on-premises?
- How do findings map to OWASP, MITRE ATLAS, the NIST AI RMF, ISO/IEC 42001 and AIUC-1?
- What happens when the agent changes next month?
Questions
What is AI penetration testing?
AI penetration testing is an authorized, simulated attack on an AI system to find weaknesses before attackers do. For an AI agent, it covers the inputs the agent reads, the tools it calls, the identity it uses and the actions it can take, not only the model's answers.
How is an AI agent penetration test different from AI red teaming?
The terms overlap. Red teaming often explores how a model can be made to misbehave. A penetration test is authorized and scoped to one system, and it ends with reproducible findings and fixes. A good test of an agent does both: it explores attacks and proves their impact on the agent's real work.
Does an AI penetration test need access to production?
No. Run the attacks on an isolated copy of the agent that has the same code, prompts, tools and permissions, connected to test systems and synthetic or masked data.
Which frameworks should the results map to?
Regulated teams usually map findings to the OWASP Top 10 for Agentic Applications for agent risks, MITRE ATLAS for attacker techniques, and the NIST AI Risk Management Framework, ISO/IEC 42001 or AIUC-1 for governance controls.
How often should an AI agent be retested?
Retest whenever the agent's code, tools, permissions, prompts or model change, and when new attack techniques appear. AIUC-1, a certification standard for AI agents, includes quarterly retests. Between tests, enforce the approved rules at runtime.