GitLab CI

Ironhide runs your agent's tests on every GitLab merge request and compares them with a baseline recorded on your default branch, the same way the GitHub Action does on GitHub.

Connect with ironhide link#

From your agent's repository:

ironhide link

After you approve once in the browser (and authorize GitLab, the first time for this project), Ironhide:

  • stores your Ironhide CI key as a masked IRONHIDE_API_KEY CI/CD variable on the project;
  • opens a merge request that adds .ironhide.yml (your agent already filled in) and the Ironhide job to .gitlab-ci.yml.

The terminal prints the merge request link. The console's GitLab setup does the same without the CLI. Full walkthrough: Link your repository.

Ironhide never keeps account-wide access. GitLab can't limit a sign-in to one project, so your authorization (scope api) is used once, during setup, to create a token scoped to this one project, and is revoked right after. Ironhide keeps only that project token (it expires within a year) to switch on the nightly schedule; Disconnect in the console revokes it.

What the job does#

  • On every merge request it drives your agent through its saved regression suite and reports passed / failed / unverified against the baseline.
  • On every push to your default branch it records the baseline that merge requests are compared with.
  • It caches the baseline between pipelines, so each merge request is compared with your default branch's latest result, not with itself.

Two framing rules apply, exactly as on GitHub:

  • Advisory-first. The job starts in advisory mode: a would-be failure is reported without failing the pipeline. Set advisory: false in .ironhide.yml when you want a regression to fail it.
  • Preview. Results carry a preview stamp: an honest measurement, not a certification.

The nightly run#

GitLab cannot declare a schedule in .gitlab-ci.yml, so ironhide link creates a daily pipeline schedule on your default branch (around 03:00 UTC). Scheduled pipelines run every action test 5 times against the baseline and alert you on a regression; merge requests run one sample per test and re-run only what looks suspicious (how a run is sampled). The schedule is created paused: nightly starts after your first run on the default branch. When the Ironhide job's first pipeline on that branch succeeds (you merged the merge request and it works), Ironhide turns the schedule on with its project token, so it never runs your pipeline on your CI minutes before then. (If your GitLab plan has no project tokens, Ironhide keeps no access and the merge request asks you to switch the schedule on yourself.) Like any pipeline schedule it runs your pipeline on that branch; delete it under Build → Pipeline schedules to opt out. If it could not be created (or turned on), the merge request and the console's repository connection say so; add a daily schedule on the default branch by hand.

CI/CD variables#

ironhide link sets IRONHIDE_API_KEY for you. If your agent calls a model provider (for example it reads OPENAI_API_KEY), add that key as a masked variable under Settings → CI/CD → Variables; the merge request lists the ones it found, and ironhide link offers to set them with glab from your local environment, without the value ever going to Ironhide.

The CLI's own settings can also come from CI/CD variables, which override .ironhide.yml:

variablewhat it does
IRONHIDE_API_KEYYour agent's key. Masked; never commit it.
IRONHIDE_ADVISORY"true" reports without failing; "false" gates.
IRONHIDE_BLOCK_ONComma-separated severity classes (critical, high, medium, low) whose first appearance fails the gate outright when gating.
IRONHIDE_SAMPLESRuns per test. More samples, tighter noise floor. Setting it turns off adaptive sampling on merge requests.
IRONHIDE_CONCURRENCYAction tests run at once (default 8 with agent_factory, 1 with adapter).
IRONHIDE_SUITEactions (action tests) or library.
IRONHIDE_PACKAction-test scenario pack: general, fsm, crm or erp.
IRONHIDE_TOOLSA JSON tool list, for tools built at runtime.
IRONHIDE_URLIronhide server base URL.

Results, the gate line, and exit codes#

Results, the IRONHIDE-GATE line and exit codes are the same as on GitHub; see The results and The gate line and exit codes. In short, the job exits 0 for a pass, an advisory result or a baseline run, 1 for a gated failure, and 2 when nothing could be verified, which is never rounded to a pass.

Reproduce a finding locally#

Run the same tests on your machine from the same checkout:

ironhide test

For a run that stored its executable tool-call trace, re-verify the stored verdict with its finding id:

ironhide repro --finding-id VD-<run_id>

The CLI prints deterministic: true for a matching verdict, deterministic: false for a mismatch, or deterministic: spec_changed when the server declines comparison against a changed spec. Tier A/B and Tier W runs are not replayable. See reading your report.

The older @v1 component#

Pipelines set up before ironhide link may still include the published component ironhide-ai/ironhide-scan/ironhide@v1 (or its remote templates/gitlab-ci.yml). Replace it: run ironhide link, or remove the include and add the job from the merge request link opens.

That component is not a working referee. It does not run your agent itself (it runs ironhide test without a driver), and it does not keep the baseline between pipelines: it declares no cache: and runs only on merge-request pipelines, so every pipeline starts a fresh baseline and nothing is compared. A green job from it is not evidence about your agent.

The merge-request note and gate line#

When a GITLAB_TOKEN variable is set, the @v1 component posts one note on the merge request. The note has exactly three parts: a heading with the result, a one-line observed-state disclaimer, and the machine-readable gate line. It carries no per-run rows, severities, episode aliases or repro commands. The merge-request note carries no repro command; use ironhide repro as shown above.

Where to go next#