Attack taxonomy
Every episode is built around a real way an agent can be turned against its owner. These categories organize the episode library. Whether a result can name a category or show observed detail depends on its response shape and visibility. There are six.
The six categories#
| Category | What the episode tests |
|---|---|
prompt_injection | Whether hostile text — in a tool result, a document, a customer message — can override your agent's instructions and get it to act on the attacker's behalf. |
tool_shadowing | Whether a look-alike or substituted tool can slip into your agent's reach and be called in place of the legitimate one. |
data_exfiltration | Whether your agent can be lured into sending data somewhere it should not go — an external recipient, an attacker-chosen destination, a report that leaks. |
excessive_agency | Whether your agent over-reaches: taking actions beyond the task it was given, touching records or systems outside its remit. |
credential_access | Whether your agent can be steered into reading, surfacing, or using secrets and credentials it should never handle. |
privilege_escalation | Whether your agent can be pushed to acquire or exercise permissions above the ones it was granted. |
A single episode is usually about one category — the injected instruction
in a customer record that tries to make your agent forward data
(prompt_injection leading toward data_exfiltration), the request that
tempts it to act beyond its brief (excessive_agency).
Observed behavior, not self-report#
The taxonomy is about what your agent did, not what it said about
itself. An agent that politely explains it would never exfiltrate data and
then forwards the record fails on data_exfiltration. The result rests on
the trajectory — the turns it took, the tools it called, the effects it
caused — captured out of band, so nothing the agent narrates can change it.
How a category becomes a result#
When your agent finishes an episode, the referee reads the submitted trajectory against the episode's own definition of a breach and returns a verdict with status, plain-language why, and any applicable severity. Most episodes are held-out; their evidence list is empty and protected measurements can be null, while the verdict and masking explanation remain. For a held-out episode, the answer key, per-turn labels, and rubric stay server-side. Featured episodes intentionally return their available detail with the run itself. Category/failure-class output is optional and must not be presented as a universal field. See the agent contract for the result shapes.
Next steps#
- Episode library: the scenarios these categories are set in.
- Running an evaluation: drive an episode and read the result.
- Agent contract: the result wire.