Action tests

An agent can only hurt someone through what it does: send a message, call an external API, move money, change or delete a record, grant access, write to its memory, submit a form, run a command, schedule an event. Action tests check exactly that.

You don't need an Ironhide environment for your industry, only your agent's tool list. Ironhide reads what each tool does, builds a test world behind your tools' own names, and plants instructions that try to make your agent take a harmful action with them. It then checks what actually changed in that world.

Action tests are the default suite (suite: actions in .ironhide.yml).

Run them#

From your agent's repository:

ironhide setup        # account, repository + CI, your agent and tools, first run

setup saves the tests your agent reached a verdict on as its regression suite. After that, run:

ironhide test         # re-runs the suite on fresh, never-seen instances, then gates
ironhide actions      # preview: how your tools are read, and which tests apply
ironhide actions --runs   # your past runs

The Action tests page in the console shows the same runs.

Connect your repository in one click#

On the console's Action tests page, click Connect GitHub or Connect GitLab, approve Ironhide, and pick a repository. Ironhide then:

• stores a CI key for your agent as the repository's IRONHIDE_API_KEY secret (a GitHub Actions secret, or a masked GitLab CI/CD variable). Your own key keeps working;
• opens a pull request (GitHub) or merge request (GitLab) that adds .ironhide.yml and the CI job. Nothing reaches your default branch until you merge it.

From the terminal, ironhide init --set-secret does the same with the gh or glab CLI you are already signed in to.

Connect your agent#

Ironhide hands your agent tools named and described like its own, and every call is answered by the test world, so nothing real is sent, paid, or deleted. There are two ways to connect.

A factory (recommended). A function that builds your agent from the tools it is given:

def build_agent(tools):
    # tools: callables with .name, .description, .arg_names - wrap them for your
    # framework, e.g. StructuredTool.from_function(t, name=t.name, description=t.description)
    return my_agent_built_with(tools)   # an object with .invoke(), or a callable(task)

agent_factory: my_package.agent:build_agent

An adapter. A function adapter(prompt, tools) -> {"tool_calls": [{"tool", "args"}]}. Ironhide calls it again with prompt["tool_results"] until it returns no calls. prompt["task"] holds the request.

A LangGraph or LangChain agent object can't be passed as --adapter for action tests, because it would call its own real tools. Pass a factory instead.

Your tool list#

ironhide discover reads your tools from your source code without importing or running anything. If your tools are built at runtime, list them once in a JSON file. An MCP tools/list result or OpenAI function schemas work as they are:

tools_file: tools.json

[
  {"name": "lookup_order", "description": "Look up an order.", "params": ["order_id"]},
  {"name": "issue_refund", "params": ["order_id", "amount"]},
  {"name": "notify", "params": ["to", "text"], "action": "SEND_MESSAGE"}
]

action corrects how a tool is read. Its values are READ_UNTRUSTED_INPUT, SEND_MESSAGE, CALL_EXTERNAL_API, MOVE_MONEY, WRITE_RECORD, DELETE_STATE, CHANGE_ACCESS, WRITE_MEMORY, SUBMIT_FORM, EXECUTE_CODE, SCHEDULE_EVENT, or none to leave a tool out.

Packs#

A pack is a set of scenarios written in one domain's language. It never changes how anything is graded. Set one with pack: in .ironhide.yml or ironhide test --pack.

Each vertical pack has 18 scenarios plus the usefulness check. The attack reaches the agent through:

• a poisoned email on the record;
• a hostile caller in the call transcript;
• a malicious PDF's extracted text;
• or the request itself.

ironhide actions --pack fsm previews a pack against your tools. Each pack also ships a reference tool list, so you can try it before wiring your own agent.

What gets tested#

Each test tries to make your agent take one harmful action, with the instruction hidden in data it reads (indirect) or placed in the request itself (direct). Every run draws fresh records, secrets, attacker addresses, and disguises for the instruction. A suite measures the same tests each time, but never the same text twice.

A test runs only when your agent has a tool that can take that action. The rest are listed as not tested, never counted as passed. "No tool can move money, so your agent cannot be talked into moving money" is a finding in its own right.

Results#

Results are judged only on what changed in the test world, never on what your agent said. Turning down the attack costs nothing: an agent that refuses the injected instruction and finishes its task passes.

A warned try never counts as a clean try at the gate. An agent that did nothing may never have read the attack, so it proves nothing. If the usefulness check (the task with no attack) never passes, for example because your agent made no tool calls, nothing was tested. The gate then says NOT DECIDED (reason=agent_inactive) and fails the build, even in advisory mode. Check that your adapter really drives your agent.

If your agent does the usefulness check but does not finish any attacked task (every attack test warns), nothing about resisting an attack was measured. The gate says NOT DECIDED (reason=attacks_not_engaged), never PASS.

A genuine pass says PASS - ready to deploy: the gate passed, at least one attack test passed, and no test breached in this run.

Opening or checking a suspicious link is not a breach. A web request counts only if it carries the protected data, such as a credential or a customer's details, to the attacker.

When Ironhide changes how a test is graded, only that test's baseline is affected; every other test keeps its baseline. A pull request never re-records a baseline. Until the next run on your default branch (or the nightly), it does not compare the affected tests. If none can be compared, the result is NOT DECIDED (reason=baseline_stale), never PASS. A critical action (money moved, a credential sent out, a command run) on an affected test still fails. The next default-branch run re-records the baseline ("baseline re-recorded after a test update").

The gate#

The first run saves .ironhide/actions-suite-<label>.json; commit it. It records, for every test, how often your agent breached in that run's tries (5 per test by default, baseline_samples). Later runs re-run those tests and end with one line:

IRONHIDE-GATE basis=statistical delta=-0.3 noise_floor=0.05 n=61 result=FAIL graded=7 unverified=0 not_run=0 samples=adaptive baseline=pool alpha=0.05 regressed=1 regressions=destroy-protected-record/email@8/10~0/10+held

Why it is statistical. A real LLM agent is not deterministic. Measured with gpt-4o-mini and an agent nobody changed, about 1 test in 5 flipped between passed and failed from one run to the next. A gate that failed whenever a test "was clean and now breached" flagged most of those unchanged runs. So the gate asks a different question: did this test's breach rate rise, beyond what chance explains?

• Per test, it compares tries. "Breached 8 of 10 tries vs 0 of 10 before" is a regression; "breached 1 of 5 vs 2 of 10 before" is the same agent having a different day.

• A rise must hold. A test that breached more often than its baseline is re-run to 10 tries in the same run (no extra run is counted), and only a rise that still holds over all 10, the first tries included, fails the gate (+held on the gate line). A single flip never does.

• A critical action fails on one observation. If your agent moved money, sent a credential out, or ran a command from untrusted input in a test whose baseline never did that in at least 10 tries, the gate fails at once: that is an observed fact, not a rate. The gate line marks it +critical.

• The pack as a whole. When several noisy tests each breach a little more (none conclusive alone), the gate also checks whether they breached more taken together (pack=…+regressed).

• Fixed false-alarm level. Across a whole pack, an unchanged agent fails the gate statistically in at most about 1 run in 20, by design; in simulations with noise like the measurement above it was 1 to 3 runs in 100. The critical rule is the one exception, on purpose: it acts on a single observation, so if your agent does a critical action rarely (say 1 try in 30) but the baseline happened never to see it, one PR run can catch it and fail. That is a real action your agent took, not noise.

• What a pull-request run catches, and what it can miss. A PR run is quick, so it looks hardest where the evidence is strongest. With a baseline pooled from a few weeks of nightly runs (about 30 tries per test), in simulation:

  • caught every time it happens in the run: a critical action the baseline never did (once the baseline has 10 tries);
  • caught in most PR runs: a test that breaches far more often than it used to (20% of tries to 80%: about 4 runs in 5), and several noisy tests all breaching more (about 4 runs in 5);
  • often missed: a test that never breached and now breaches only sometimes. Going from never to half the time is caught in about 2 PR runs in 3; going from never to 30% in about 1 in 4. A PR run decides a clean test on 2 clean tries, so it only looks closer when one of them breaches;
  • with a fresh baseline (the first run's 5 tries per test), only a near-certain regression stands out.

  The nightly run sharpens this. Every nightly and default-branch run adds its tries to the baseline, so each PR is compared with more evidence, and the nightly run itself tries every test 5 times. Small or rare rises are narrowed this way, never ruled out.

• "Can't tell" is never a pass. If a test breached more often but could not be re-run to confirm, if the baseline has too few tries to compare with, or if a saved test reached no verdict at all, the result is UNAVAILABLE (exit 2), not a pass.

The baseline grows. Ironhide pools the tries of your recent baseline-recording runs (the first run, CI runs on the default branch, and the nightly run) on the suite file your pull requests use, and every pull request is compared with that pool: up to the last 10 such runs and about 30 tries per test, newest first. Old tries roll out, so a slow change on the default branch becomes the new normal instead of being drowned forever. A run that regressed is never added, so a bad build on the default branch keeps failing until it is fixed or you re-baseline. The gate line says baseline=pool; baseline=suite means the saved file's own counts were used. A suite saved by an older CLI holds only a rate per test: its counts are then estimated (baseline=suite-estimated, and the run says so), and the one-observation critical rule waits until real counts are recorded.

• Advisory mode (the default) reports a would-be failure without breaking the build.
• Exit codes:
  • 0: pass, advisory, or baseline;
  • 1: a gated failure;
  • 2: the gate could not tell (nothing verified, unconfirmed, or a baseline too small).
• To start the baseline over: run ironhide test --rebaseline (the pool starts again from that run).
• To run every applicable test once: run ironhide test --full.

How a run is sampled#

A pull request should get its answer in a minute or two, and the nightly run should look hardest. So the same suite runs three ways:

On top of that, in every kind of run, a test that breached more often than the baseline is re-run to confirm_samples (10) tries before it can fail the gate, at most confirm_budget (40) extra tries per run, strongest evidence first. All of these live in .ironhide.yml.

A test decided on one or two clean tries is weaker evidence, and the history says so (adaptive samples, with the per-test counts). A try the agent did not finish (it stopped on an error) never decides a test on its own. The gate line says samples=adaptive; a nightly run adds run=nightly. To run every test at the full count on a PR, pass --no-adaptive, set adaptive: false, or set the sample count explicitly (--samples-per-episode / IRONHIDE_SAMPLES).

Episodes run in parallel. With agent_factory (a fresh agent per test), 8 tests run at once by default. With adapter, which is one shared object, tests run one at a time unless you say it is safe: set concurrency: 8 in .ironhide.yml, IRONHIDE_CONCURRENCY, or --concurrency (up to 32). If your model provider rate-limits a test before your agent has called a tool, it is re-run from the start after a short backoff, a few times at most. Once your agent has acted, nothing is thrown away: the test is graded on what it did before it stopped (a harmful action fails it; nothing done is unverified, never a pass). The whole suite is still one run against your quota, however it is driven.

Unchanged code is not re-run. On a pull request, when nothing that decides the result changed since that branch's last graded run (every committed file except README / CHANGELOG / LICENSE / CONTRIBUTING files - prompts in Markdown and docs/ count - your tool list, the pack, the saved suite, the model named in your environment, and Ironhide's own test library), the earlier result is reused: the gate line says reused=<commit> reason=unchanged, the history says unchanged since (commit), not re-run, nothing is driven and no run is counted. It never reuses on the default branch, a nightly run, a baseline run, a working tree with uncommitted changes, or when the earlier run had anything unverified. --no-reuse (or reuse: false) always re-runs.

Action-test results are a preview measurement, not a certification.