CI & episodes

This is the inbound surface. Your real agent runs pre-production inside your own CI/CD; your pipeline pulls each episode, drives your agent locally against it, and submits the trajectory back to Ironhide for grading. Ironhide never calls out to your agent — the results flow in.

Two things live here:

The episode drive loop — discover the episodes your agent can run, pull a manifest, step an executable world when one exists, and submit the resulting trajectory for a verdict.
Release gating — freeze a suite, collect your agent's runs into a scored run-set, and gate one run-set against a prior baseline.

Every route is owner-authenticated with the agent's token (or your account owner key), and every episode route needs a verified agent (409 agent_not_verified otherwise):

Authorization: Bearer wk_live_...

In CI the CLI and the GitHub Action wrap these calls for you and read the token from the IRONHIDE_API_KEY environment variable.

Episode tiers#

Each episode has a tier, and the tier decides what you submit:

Tier	Episode type	You drive	You submit
A	behavioral	The multi-turn script	`detections`
B	adversarial	The adversarial transcript	`actions`
C	adversarial + executable world	The transcript, stepping a tool world	`actions` and `env_trace`

Tier C episodes expose a live world you step through POST /ci/episode/{id}/step; Tier A and B are driven from the manifest alone.

List drivable episodes#

POST /ci/episodes/{agent_id}

The full library your agent can drive, each addressed by a stable per-customer alias. This is where you discover the aliases to pass to the manifest, step, and submit routes.

curl -X POST https://ironhideai.com/ci/episodes/agt_1a2b3c \
  -H "Authorization: Bearer wk_live_YOUR_KEY"

{
  "episodes": [
    {
      "alias": "ep_7c1f9a20",
      "episode_type": "adversarial",
      "tier": "C",
      "featured": true,
      "title": "CRM exfiltration under a shadowed tool"
    },
    {
      "alias": "ep_2b44de81",
      "episode_type": "behavioral",
      "tier": "A",
      "featured": false
    }
  ],
  "maturity": "preview"
}

Field	Notes
`alias`	The addressable id for this episode, stable per customer. Pass it to the manifest / step / submit routes.
`episode_type`	`behavioral` or `adversarial`.
`tier`	`A`, `B`, or `C` (see above).
`featured`	Whether the episode is in the public display case.
`title`	Present only for episodes with a public title. A title is part of the answer key, so alias-only episodes never include one, and an alias never resolves to a real episode id on any customer path.

Pull an episode manifest#

POST /ci/episode/{agent_id}/manifest

Pull the wire body for one episode so your CI can drive your agent against it locally. Body: {"episode_id": "<alias>"}.

curl -X POST https://ironhideai.com/ci/episode/agt_1a2b3c/manifest \
  -H "Authorization: Bearer wk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"episode_id": "ep_7c1f9a20"}'

The response embeds the same visible episode content the single-shot path ships — the episode's script/transcript and its system context — plus these envelope fields:

Field	Notes
`episode_id`	The alias you asked for (echoed back).
`tier`	`A`, `B`, or `C`.
`has_world`	`true` when the episode has an executable world you step.
`wire_run_id`	A per-pull run id; pass it back on step and submit so the trajectory is tied together.
`tools`	Present only when `has_world` is true: the world's tool manifest (tool names and argument names) so you can drive `POST /step`.
`maturity`	`"preview"`.

Nothing that would reveal ground truth ships in the manifest: no spec, prohibited-effect list, per-turn labels, planted canaries, or rubric. Errors: 422 missing_episode_id, 404 episode_not_found.

Step an executable world#

POST /ci/episode/{agent_id}/step

For a Tier C episode only. Replays the tool calls your agent has made so far against a fresh copy of the world and returns the observations your agent is allowed to see. The call is stateless: the whole trace is replayed from the world's initial state every time, so nothing is cached, graded, or recorded here.

Body:

Field	Notes
`episode_id`	The Tier C episode alias.
`env_trace`	The tool calls so far, each `{ "turn", "tool", "args" }`. Bounded per episode.
`round_index`	Optional; the current round, echoed back.
`wire_run_id`	The id from the manifest, echoed back.

{
  "tool_results": [ { "turn": 0, "tool": "crm.search", "result": "..." } ],
  "round": 1,
  "budget": { "used": 3, "max": 64 },
  "wire_run_id": "a1b2c3d4e5f6",
  "episode_id": "ep_7c1f9a20"
}

503 feature_dark when the executable-effects capability is off; 404 no_world when the episode is not Tier C; 400 trace_too_long / 400 bad_trace for an oversized or unparseable trace.

Submit a trajectory#

POST /ci/episode/{agent_id}/submit

Submit the driven trajectory for grading and recording. This is the call that returns a verdict on what your agent did. Body depends on the tier:

Field	Tier	Notes
`episode_id`	all	The episode alias.
`detections`	A	What your agent flagged.
`actions`	B, C	The defense actions your agent took.
`env_trace`	C	The tool-call trace it made against the world.
`label`	all	Optional label for the run, default `"run"`.

curl -X POST https://ironhideai.com/ci/episode/agt_1a2b3c/submit \
  -H "Authorization: Bearer wk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"episode_id": "ep_7c1f9a20", "actions": [ ... ], "env_trace": [ ... ]}'

{
  "run_id": "run_9d3a...",
  "episode_id": "ep_7c1f9a20",
  "episode_type": "adversarial",
  "tier": "C",
  "status": "FAIL",
  "severity": "CRITICAL",
  "why": "Agent forwarded the planted record to an off-domain recipient.",
  "evidence": [ { "kind": "canary_egress", "turn": 4, "detail": "..." } ],
  "resistance": 0.0,
  "completion": 1.0,
  "repro_command": "ironhide repro run_9d3a...",
  "maturity": "preview"
}

The verdict fields (status, severity, why, evidence, and the Tier C observed-state numbers) are documented in full on the Verdicts page. The derived answer key and the grader's internals are never put on the wire — you see the episode type and the verdict, never the key. 503 feature_dark when Tier C grading needs the executable-effects capability and it is off.

Freeze a suite#

POST /ci/suite/{agent_id}

Returns the frozen CI suite manifest your gate compares against — a pinned, versioned set of episodes under a content-derived suite_id. Two run-sets that report the same suite_id provably used the same suite.

{
  "suite_id": "cisuite-9f31c2ab54de",
  "spec_version": "1",
  "name": "executable-default",
  "episode_library_version": "exec-1a2b3c4d5e6f",
  "created_at": "2026-08-20T12:00:00+00:00",
  "episode_count": 12,
  "episode_ids": ["ep_2b44de81", "ep_7c1f9a20"]
}

Alias-only episodes appear under their stable alias; the real episode id never crosses the customer boundary. 503 feature_dark when the executable-effects capability is off.

Collect and score a run-set#

POST /ci/test/{agent_id}

Collects this agent's captured runs for the suite's episodes and returns them as a scored run-set plus per-run verdicts. Requires a verified agent. Body: {"label": "..."} (optional). No dispatch happens server-side — you submit episode runs first (via the drive loop above), then call this to collect and score them.

{
  "agent_id": "agt_1a2b3c",
  "suite": { "suite_id": "cisuite-9f31c2ab54de", "episode_count": 12 },
  "run_set": {
    "suite_id": "cisuite-9f31c2ab54de",
    "agent_id": "agt_1a2b3c",
    "label": "run",
    "runs": [
      { "run_id": "run_9d3a...", "episode_id": "ep_7c1f9a20", "indicator": 0.0, "status": "FAIL", "severity": "CRITICAL" }
    ],
    "excluded": { "inconclusive": 1 },
    "fail_severity_classes": ["CRITICAL"]
  },
  "verdicts": [ ... ],
  "summary": {
    "clean_rate": 0.83,
    "n": 12,
    "excluded": { "inconclusive": 1 },
    "fail_severity_classes": ["CRITICAL"]
  }
}

Summary field	Notes
`clean_rate`	The share of scored runs with no breach (`null` for an empty set — no runs, no rate).
`n`	Number of scored runs.
`excluded`	Offered-but-not-counted runs by reason, including `inconclusive` runs that could not be verified.
`fail_severity_classes`	The distinct severity classes seen among failing runs.

The run_set object is what you hand to the gate as before or after. It is computed over your own runs only — a tenant gating its own deploys on its own runs is not export, benchmark, or resale.

Gate a release#

POST /ci/gate/{agent_id}

Gate an after run-set against a before baseline — a statistical clean-rate shift past the noise floor, plus a discrete check for any new failing-severity class. Both run-sets come from POST /ci/test and are held by your CI, so a tenant can only gate its own data.

Field	Notes
`after`	Required. The current run-set (`422 missing_after` if absent).
`before`	Optional. Omit it to establish a baseline — the gate returns `BASELINE_CREATED` with nothing to compare.
`block_on`	Severity classes that fail the gate, default `["CRITICAL", "HIGH"]`.
`advisory`	Default `true`: report the result without failing the build.

{
  "agent_id": "agt_1a2b3c",
  "result": "PASS",
  "reasons": ["within_noise_floor"],
  "delta": {
    "suite_id": "cisuite-9f31c2ab54de",
    "before_label": "baseline",
    "after_label": "run",
    "before_mean": 0.80,
    "after_mean": 0.83,
    "delta": 0.03,
    "verdict": "no_measurable_change",
    "significance": null
  }
}

result is one of PASS, FAIL, ADVISORY, BASELINE_CREATED (no before was sent), or UNAVAILABLE (the two run-sets were not comparable — an unanswerable gate says so rather than faking a pass). reasons explains the call; delta carries the before/after comparison and is null when there is nothing to compare.

A new failing-severity class absent from the baseline fails the gate even inside the noise floor — a first observed CRITICAL is a discrete fact, not a stochastic movement. significance is null on purpose: a suite of this size does not support a significance claim, and the gate will not manufacture one.

The gate is advisory-first by design: with advisory: true a would-be FAIL is reported as ADVISORY with identical reasons. Set advisory: false when you are ready to let a regression fail the pipeline.