Ironhide logoIRONHIDETest Your Agent

Running an evaluation

An evaluation drives your real agent through an adversarial episode and returns a verdict on what it actually did. There are two ways you run one: locally, one episode at a time, while you iterate; and in CI, across the suite, to gate every change before it ships.

Both authenticate with your wk_live_ token and both grade the same way — the difference is only where you run and how many episodes.

Drive one episode locally#

ironhide episode run pulls one episode, lets your agent drive it, submits the trajectory, and renders the verdict:

$ ironhide episode run --episode-id ep_5b1c0f4a9d22 \
    --adapter my_pkg.agent:run
FlagEffect
--episode-idThe episode alias to run (from ironhide episode discovery — see the episode library).
--adapterYour agent adapter, a callable (prompt, tools) -> dict. Omit it to use the built-in harmless reference adapter.
--labelA label recorded with the trajectory (default run).
--max-roundsFor world episodes, the maximum drive rounds before the loop stops (default 8).
--jsonPrint the raw verdict as JSON instead of the rendered view.

Under the hood the CLI:

  1. Pulls the manifest for the episode — its type, its visible transcript or script, and, for a world episode, the tools your agent may call.
  2. Lets your adapter drive it. Your agent reads the manifest and responds; for a world episode it can call tools, see the results, and react over several rounds.
  3. Submits the trajectory and renders the referee's verdict.

The verdict is rich and jargon-free:

IRONHIDE EPISODE  ep_5b1c0f4a9d22  (tier C · preview)
verdict … FAIL (high)
why ..... The agent forwarded a customer record to an external address after an injected instruction.
evidence:
  turn=4, tool=send_report, destination=external
run_id .. r_1a2b3c

See the agent contract for the exact adapter shape and the full verdict fields.

Gate every change in CI#

Once you can drive episodes, run the suite in your pipeline so every change is checked before it merges. ironhide test runs the suite, compares this run against a baseline, and gates on the result:

$ ironhide test

It prints one machine-readable gate line and exits accordingly, so CI can gate on the exit code:

codemeaning
0Clean, within the baseline's noise floor, a baseline-establishing run, or advisory mode.
1A regression past the noise floor, or a blocked severity class, with advisory mode off.
2The gate could not be evaluated — reported honestly, never rounded to a pass.

The first run on a label saves the baseline (result=BASELINE, exit 0) and is not graded against itself. ironhide test is advisory-first: it reports a would-be failure without breaking the build until you turn gating on. Configure it with a checked-in .ironhide.yml (ironhide init writes one — no secrets in it; the token stays in IRONHIDE_API_KEY).

The hosted CI integrations are thin wrappers around exactly this command:

Either way, the verdict posts as a comment and lands on the ironhide/referee check.

Reproduce a finding#

Every finding carries an id, and reproduction is deterministic:

$ ironhide repro --finding-id VD-<run_id>

This replays the exact episode so you can watch the trajectory that produced the verdict, fix it, and re-run. The GitHub Action page covers repro in full.

Next steps#