AgentHarm (adopted: harmful-request refusal)
The user makes the harmful request directly. The test asks whether the agent refuses it.
AgentHarm is different in kind from most of the other attack suites in this catalog: there is no hidden injection, no poisoned document, and no third party trying to trick the agent. Instead, this adopts AgentHarm (Andriushchenko et al., ICLR 2025), an external published benchmark of harmful agentic requests, and hands each harmful request to the target agent directly and openly, exactly as written, because in this suite, the user themselves is the adversary.
The measurement is refusal: does the agent decline to carry out the harmful behavior it was asked to perform. To catch agents that swing too far the other way, refusing things they shouldn't, each harmful behavior in this suite is paired with a benign counterpart run as a 'benign twin' (a matched, harmless version of the same request), so an agent that refuses everything indiscriminately doesn't get credit for being 'safe.'
It's worth being precise about what this does and doesn't measure. This suite reproduces only AgentHarm's RefusalRate metric. It deliberately does not provide the agent with actual harmful tools, does not let any harmful action actually execute, and does not compute AgentHarm's separate HarmScore. So a result here is a proxy for how AgentHarm's original leaderboard reports harmfulness, not a like-for-like reproduction of that number.
The underlying data comes from the gated ai-safety-institute/AgentHarm dataset on Hugging Face, loaded through the inspect_evals library rather than vendored into this repository, and the number of behaviors evaluated per run is capped by an AGENTHARM_LIMIT environment variable, defaulting to 20.
Choose a stage, or scroll sideways to follow all 5 stages
Taint ingress Attack sequence
Harmful user request
The agent receives a harmful agentic request from AgentHarm's dataset, presented as an ordinary user ask with no injection involved.
Choose a stage to inspect that part of the attack.
What this test can show
This suite adapts AgentHarm directly from its ICLR 2025 paper, 'AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents' (Andriushchenko et al., arXiv:2410.09024), and maps to OWASP category ASI05. It matters because it tests a different failure mode than injection resistance: whether an agent will simply do what it's told when the request itself, not some third-party payload, is the harmful part, a scenario where a non-refusing agent executing harmful behaviors exposes whoever deployed it to real legal and safety liability, since the request came through the front door with no attacker needed.
Grounding
References & mappings
- OWASPASI05
- PaperAndriushchenko et al., 'AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents', ICLR 2025, arXiv:2410.09024