GitHub Action

Using the GitHub Action

A composite action usable from any repository. It installs the harness, runs the gates, writes both forms of the evidence pack, posts the document to the job summary, and fails the job when a gate fails.

Usage from another repository

name: ai-gates

on: [pull_request]

permissions:
  contents: read

jobs:
  gauntlet:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - id: gauntlet
        uses: ChelseaKR/gauntlet@<commit-sha>
        with:
          cases: eval/cases
          target-callable: myapp.evalapi:make_target
          baseline: eval/baseline-results.json
      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
        if: always()
        with:
          name: gauntlet-evidence
          path: |
            gauntlet-results.json
            gauntlet-evidence.md
            gauntlet-evidence.json
      - run: echo "cases ${{ steps.gauntlet.outputs.cases-passed }}/${{ steps.gauntlet.outputs.cases-total }}"

Pin the action to a commit SHA, the way the repository pins the actions it uses itself. Nothing is published to a package registry and no release tag is implied. From inside this repository the same steps run against a local checkout with uses: ./.

A failing gate is the expected outcome of a working gate, so the action does not abort before the evidence pack exists: the gates step captures a non-zero exit, the pack is built and posted, and a separate step blocks the merge. Setting fail-on-gate-failure to false reports without blocking.

Inputs

Every input the action declares, read from action.yml at build time.
Input	Default	Meaning
`baseline`	none	Path to an earlier results JSON. When present, the evidence pack reports whole-run drift against it: gates added or removed, pass-rate deltas per gate and per language, and the cases that newly fail or newly pass. A missing file is reported, not fatal.
`cases`	none	Directory of *.yaml case files. Leave empty to use the built-in bilingual suites, which evaluate the in-repo toy target and are useful mainly for trying the action out.
`fail-on-gate-failure`	`true`	Fail the step when any gate fails. Set to "false" to report without blocking.
`job-summary`	`true`	Write the evidence document to the GitHub job summary.
`json-path`	`gauntlet-evidence.json`	Path the machine-readable evidence pack is written to.
`python-version`	`3.12`	Python version used to run the harness.
`report-path`	`gauntlet-evidence.md`	Path the human-readable evidence document is written to.
`results-path`	`gauntlet-results.json`	Path the results JSON is written to.
`target-callable`	none	Python target as "module.path:factory", where the factory returns an object with a name attribute and an ask(prompt, language) method. The module must be importable from working-directory. Mutually exclusive with target-url.
`target-url`	none	HTTP endpoint to evaluate. It receives POST {"prompt","language"} and must answer with {"text","citations","context_ids","refused","escalated"}. Mutually exclusive with target-callable.
`uv-version`	`0.11.29`	uv version installed to run the harness.
`working-directory`	`.`	Directory the gates run from. Relative paths above are resolved against it.

Outputs

Every output the action declares, read from action.yml at build time.
Output	Meaning
`cases-failed`	Number of cases that failed.
`cases-passed`	Number of cases that passed.
`cases-total`	Number of cases that ran.
`drift-computed`	"true" when a baseline was supplied and whole-run drift was computed.
`drift-newly-failing`	Cases that passed in the baseline and fail now.
`drift-newly-passing`	Cases that failed in the baseline and pass now.
`gates-failed`	Number of gates that did not meet their threshold.
`gates-passed`	Number of gates that met their threshold.
`gates-total`	Number of gates that ran.
`json-path`	Path of the machine-readable evidence pack that was written.
`passed`	"true" when every gate met its threshold.
`report-path`	Path of the human-readable evidence document that was written.
`results-digest`	sha256 of what the run observed, excluding the clock. Equal digests mean equal runs.
`results-path`	Path of the results JSON that was written.

Counts come from the harness. Nothing in the action asserts a number the run did not produce.