How to test an LLM application for prompt injection

What you are actually testing

A large language model receives its instructions and its data in one undifferentiated token stream. The system prompt, the conversation, the retrieved document and the tool output all arrive as text, and the model has no reliable mechanism for deciding which of them has authority. That is the whole vulnerability. Everything else is a question of what the application lets the model reach when the wrong text wins.

This has a practical consequence for testing. A payload that makes a chatbot say something rude is a curiosity. The same payload against an assistant that can read a private repository and open a pull request is a data breach. So the object of the test is never the model in isolation; it is the model plus its data, plus its tools, plus the way its output is rendered.

The methodology below follows the structure PortSwigger uses in its Web Security Academy material on web LLM attacks, which is the clearest public statement of the approach: identify the inputs, work out what data and APIs the model can reach, then probe the resulting attack surface. The detail underneath each step comes from the OWASP AI Testing Guide, whose AITG-APP-01 test is the most complete published payload taxonomy for this class.

Step 1: map every input the model reads

Start with the direct inputs, because they are easy and they are never the interesting ones: the chat box, the API request body, any structured fields the front end passes through. Then find the indirect ones, which are every piece of text that reaches the context window without a human deciding to send it.

• Documents in the retrieval index, and the write path into that index. Who can add a document, and does that person need any privilege at all.
• Email, tickets, chat messages and CRM records the assistant summarizes. Anyone who can file a ticket can write into the context.
• Web pages the model fetches, whether the user supplied the URL or the model chose it.
• Files users upload: PDFs, spreadsheets, images with text, and anything a parser turns into text before the model sees it.
• Repository content read by a coding agent: source files, README files, issue bodies, commit messages, CI configuration.
• Tool and MCP server descriptions, which are text the model reads and trusts as instructions about how to behave.
• Output from other agents in a multi-agent system, which is untrusted content wearing an internal badge.
• Conversation history and long-term memory, which turn a single successful injection into a persistent one.

Write this list down before testing. In practice half the eventual findings are visible on it: an index that any authenticated customer can write to, a summarization feature over a public inbox, an agent that reads pull-request descriptions from forks. Naming the channel is most of the work.

Step 2: map what the model can reach

The second map is of consequences. For every tool, function, plugin and MCP server the model can invoke, record three things: what it does, what identity it runs as, and whether any human confirms it. The identity question is the one that turns a moderate finding into a critical one. In the Supabase case published by General Analysis in July 2025, the assistant reached the database through a connection running with the service role, which bypasses row-level security; an injected support ticket was enough to read an integration-token table and write the secrets back into the ticket thread.

Then record the outbound paths that are not tools at all. Markdown image rendering fetches a URL. A link the user might click carries whatever the model put in the query string. A generated HTML fragment can reference an external stylesheet. In the ForcedLeak case against Salesforce Agentforce, CRM data left the environment as parameters on image requests to a domain that had once been allow-listed, had expired, and had been re-registered by the researchers for a few dollars.

Reach inventory: what to record before probing

Element	Record	Why it changes severity
Tool or function	Name, arguments, side effects	A read tool leaks; a write tool acts
Identity	Service account or end-user token	A shared privileged identity removes every access control behind it
Confirmation	None, implicit, explicit	Approval fatigue makes implicit confirmation equivalent to none
Retrieval scope	Indexes, filters, tenant key	Determines what a successful injection can read
Rendering	Markdown, HTML, images, links	Decides whether disclosure becomes exfiltration
Egress	Allowed destinations from server and browser	The last control that still works after the model is fooled

Step 3: probe, and probe more than once

Only now does payload work begin. Two properties of these systems make naive testing unreliable, and both are called out in the OWASP AI Testing Guide. The first is temperature: the same prompt can succeed and fail on consecutive runs, so a single failed attempt proves nothing. The second is that guardrails are frequently themselves language models, with their own sampling behavior, which means the filter is as non-deterministic as the thing it filters.

The practical rule is to repeat every meaningful attempt across multiple runs and to record the success rate rather than a binary. A payload that works one time in five is a finding, not a near miss. Conversation history matters too: in an application that feeds prior turns back to the model, the same question can produce different answers depending on what came before, which is the mechanism several multi-turn attacks rely on.

The payload families worth running

AITG-APP-01 lists twenty-three payload families. They are building blocks rather than a checklist to run verbatim, and they age quickly as models change, but the families themselves have proven stable. The ones below carry the most weight in an application audit.

• Multi-turn escalation. Shift the topic gradually across turns until the model volunteers what it refused at the start. The crescendo pattern and the echo-chamber pattern both work this way, using the model's own earlier output as evidence that the direction is acceptable.
• Role play and persona. Place the model in a hypothetical where the restricted behavior reads as in character. Old, still effective against thinly prompted applications.
• Context hijacking. Instruct the model to disregard everything before this point and start again. Crude, and still the fastest way to find an application with no separation between instruction and data.
• Obfuscation and token smuggling. Base64, hex, leetspeak, reversed text, homoglyphs and whitespace tricks, aimed at a filter that inspects the literal string rather than the meaning.
• Payload splitting. Deliver the instruction in fragments across fields, turns or documents, and have the model reassemble it.
• Multi-language. Issue the instruction in a language the guardrail was not tuned on, then ask for the answer in the original language.
• Structured output attacks. Inject through a JSON field, a tool argument or a schema-constrained response, where the application assumes structure implies safety.
• Multimodal. Text inside an image, a document, a spreadsheet cell or audio, reaching the model through a parser rather than the chat box.
• Delayed triggers. An instruction that lies dormant until a later condition, so the malicious document is ingested long before it acts. MITRE ATLAS records this as its own technique, AML.T0094.

Two notes on hygiene. Tailor payloads to the target: an application usually layers its own restrictions on top of the model provider's safeguards, and it matters which of the two you are testing. And combine families rather than running them in isolation, because the interesting bypasses are almost always compositions.

Indirect injection: the channels that decide the outcome

Direct injection finds weak prompts. Indirect injection finds architecture problems, and it is where every serious published incident sits. MITRE ATLAS separates the two as AML.T0051.000 and AML.T0051.001, and adds AML.T0093 for prompt infiltration through a public-facing application.

EchoLeak, disclosed in June 2025 and tracked as CVE-2025-32711, is the canonical example: a crafted email the victim never opened was processed by Microsoft 365 Copilot retrieval, and the assistant disclosed information over the network. Microsoft scored the AI command injection at 9.3; the NVD record carries 7.5 from NIST. ShadowLeak, reported to OpenAI by Radware in June 2025 and marked resolved on 3 September 2025, worked the same way against the ChatGPT Deep Research agent: a booby-trapped email, no user interaction beyond asking the agent to summarize the inbox, and exfiltration performed by the agent calling an attacker-controlled URL from the provider's cloud servers rather than from the victim's browser.

Test each ingestion channel on its own terms. Plant an instruction in a document and then ask an innocuous question that will retrieve it. File a ticket and ask the assistant to summarize the queue. Publish a page and point the browsing tool at it. Open a pull request from a fork and let the coding agent read the description. For each one, record whether the instruction was followed, whether the user could have seen anything unusual, and what the application logged.

Proving the exfiltration channel

A finding that ends with the model saying something it should not is incomplete. The question a reader will ask is whether data left the system. Three channels do almost all of the work in practice.

• Rendered images. The model emits a markdown image whose URL embeds the secret. The client fetches it. No click required. This is the ForcedLeak mechanism and it is why image rendering and egress allow-lists belong in the same conversation.
• Links the user is invited to click. Slower, still effective, and much harder to spot in a log because the request comes from a legitimate user action.
• Tool calls. The most direct: the model sends the data itself, through an email tool, an HTTP tool, a database write or a reply on the originating platform. ATLAS files this as AML.T0086, exfiltration via AI agent tool invocation.

Use a controlled collector, never a live third-party endpoint, and record the request as evidence. The screenshot that gets a finding fixed is the one showing the secret arriving at a destination the customer does not control.

Tooling, and where it stops helping

Automation earns its place on breadth. The OWASP AI Testing Guide names three tools for this test: the garak prompt-injection probe, the Prompt Security fuzzer, and promptfoo for adversarial prompt crafting. All three are good at running many variants against one endpoint and at regression-testing a fix.

What they cannot do is understand your application. They do not know that the assistant is allowed to read the HR index, that a particular tool writes to a shared mailbox, or that a specific document type is ingested from a public form. Every incident in the record above required that context. Treat the scanners as coverage insurance and expect the findings that matter to come from a person holding the two maps from steps one and two.

A prompt injection includes instructions for what the tester wants the AI to do, a trigger that causes the model to follow them instead of its own constraints, and an intent that conflicts with those constraints. The way those three interact determines whether an attack succeeds, and why traditional filtering struggles to keep up.

What a pass looks like

There is no configuration that makes an application immune, so a report that claims one is wrong. What a good result looks like is an application where injection still succeeds occasionally and nothing consequential follows: the model has no privileged identity to borrow, the tools it can call are read-only or confirmed, the rendering path cannot originate requests, egress is restricted to destinations the customer controls, and the logs are complete enough to reconstruct what happened.

That is why the closing section of an injection assessment is about controls that sit outside the model. The OWASP remediation list for this test says the same thing in shorter form: isolate user prompts from system instructions, restrict model privileges by design, and require human approval for sensitive operations.

The evidence an auditor keeps

For companies inside the scope of the EU AI Act, this testing produces the record that Article 15 asks for. High-risk systems must achieve an appropriate level of accuracy, robustness and cybersecurity and be resilient against attempts by unauthorized third parties to alter their use, outputs or performance; the article names data poisoning, model poisoning, adversarial examples and confidentiality attacks as the AI-specific vulnerabilities to address. Those requirements apply to Annex III systems from 2 December 2027 and to Annex I product-embedded systems from 2 August 2028.

Keep, at minimum: the input map and the reach inventory, the payload set with run counts and success rates, the transcript for each successful case, the network evidence for each exfiltration, and an explicit list of what was not tested and why. The last one is what distinguishes an audit record from a marketing artifact. For how each of the ten OWASP LLM risks is evidenced, see the OWASP LLM Top 10 audit checklist; for agent-specific cases, see agentic and MCP security.