AgentDojo (adopted: real environment)
The target agent runs inside AgentDojo’s real tool environment, and AgentDojo decides whether the injection succeeded.
AgentDojo is an external, published benchmark (Debenedetti et al., NeurIPS 2024) for evaluating prompt-injection attacks and defenses against LLM agents. Rather than reimplementing its scenarios, this suite adopts AgentDojo wholesale: the target agent under test is handed AgentDojo's real tools through an adapter (AgentDojoToolSet), and an injection goal is planted into AgentDojo's own designated injection slots inside its own stateful environment, not a simplified stand-in built for this harness.
Because the harness doesn't own the task design here, success is judged by AgentDojo's own scoring logic, its injection_task.security() check run over the state of the environment after the agent finishes, rather than this harness's usual canary-and-sink oracle. Rows from this suite are tagged accordingly (oracle_type=agentdojo_security, environment=agentdojo) so they're never silently blended with first-party attack-suite numbers.
This replaces an earlier, cruder adapter that had scored AgentDojo's task-goal strings using this harness's own 4-tool Docker sandbox and canary/sink oracle, a mismatch that measured something closer to 'did the agent do something matching this string' than AgentDojo's actual intended notion of a security violation in its own richer environment. The current version keeps the target agent unmodified but runs it in AgentDojo's real, more complex tool world instead.
The suite is disabled by default and gated behind an optional install (pip install taint_eval[agentdojo]), with the scale of coverage (a matrix of user tasks against injection tasks) capped by an AGENTDOJO_LIMIT environment variable, defaulting to 24 combinations.
Choose a stage, or scroll sideways to follow all 5 stages
Taint ingress Attack sequence
User task continues
The agent works through AgentDojo's user task using the real tool surface, unaware the environment carries a planted goal.
Choose a stage to inspect that part of the attack.
What this test can show
This suite adapts AgentDojo directly from its NeurIPS 2024 paper, 'AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents' (Debenedetti, Zhang, Balunović, Beurer-Kellner, Fischer, Tramèr, arXiv:2406.13352), and maps to MITRE ATLAS technique AML.T0051.001 and OWASP category ASI01. It matters because it lets results be compared against a widely-cited external standard rather than only this harness's own scenario design, at the cost of that scenario design (and its specific tasks and environment) not being something this project controls or can vouch for beyond what AgentDojo itself already publishes.
Grounding
References & mappings
- MITRE ATLASAML.T0051.001
- OWASPASI01
- PaperDebenedetti, Zhang, Balunović, Beurer-Kellner, Fischer, Tramèr, 'AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents', NeurIPS 2024, arXiv:2406.13352