Documentation

Gauntlet

Merge-blocking evaluation gates for generative AI features, plus an evidence pack that cross-references what the gates found to California's published GenAI risk and procurement framework.

What it is

Gauntlet runs YAML-driven gate suites against any HTTP endpoint or Python callable, fails the build when a gate fails, and emits the run in two forms: a versioned JSON pack a machine can diff, and a document a reviewer can attach to a risk assessment.

The built-in suites carry 5 gates and 66 cases, counted from the suites the harness loads. It depends on no model vendor, reaches the network only where the operator points it at an HTTP target, and ships a deliberately breakable toy target so a reviewer can watch each gate fail on purpose.

What it is not

• Not a compliance certification. The language is "aligned to", never "approved by" or "compliant with". The State of California, the California Department of Technology, and the Department of General Services have not reviewed, approved, endorsed, or certified this project or anything it emits.
• Not a model benchmark. A gate result describes one deployed feature in its context: prompts, retrieval, guardrails, routing. It says nothing about a foundation model in the abstract.
• Not a red-team service. It is the fixture that keeps red-team findings regression-tested after the exercise ends.
• Not a way to verify an honest target. Grounding identifiers are checked against the context the target claims to have retrieved. A dishonest target is out of scope, and every evidence pack says so on its face.

Quickstart

Install

Nothing here is published to PyPI or any other package registry. Install from a checkout, and pin the GitHub Action to a commit SHA.

git clone https://github.com/ChelseaKR/gauntlet
cd gauntlet
uv sync

# The built-in bilingual suites against the in-repo toy target.
uv run gauntlet run --out results.json

Write a case file

Case files are YAML, one gate per file. The loader is strict: unknown keys, unknown enum values, duplicate ids, and malformed YAML are rejected with a located error rather than silently skewing a result. English and Spanish cases are peers, added and changed together.

suite: my-grounding
gate: grounding          # grounding | adversarial | refusal | false_positive | golden
version: 1               # bump when the suite changes
threshold: 1.0           # fraction of cases that must pass
cases:
  - id: gnd-en-hours
    language: en
    prompt: What are the library hours?
    expect_grounded: true
    must_contain: ["library"]
  - id: gnd-es-horario
    language: es
    prompt: ¿Cuál es el horario de la biblioteca?
    expect_grounded: true
    must_contain: ["biblioteca"]

Run the gates

uv run gauntlet run --cases path/to/cases \
  --http-url https://your-service.example/evaluate --out results.json

# or a Python target: a factory returning an object with
# a name attribute and an ask(prompt, language) method
uv run gauntlet run --cases path/to/cases \
  --callable your_package.module:make_target --out results.json

Read the result

The command prints one line per gate and one verdict, then writes the results JSON. This is the real output of the built-in suites against the healthy toy target, produced while these pages were built:

target: toy
  [PASS] adversarial: 24/24 (threshold 1; en 12/12, es 12/12)
  [PASS] false_positive: 12/12 (threshold 1; en 6/6, es 6/6)
  [PASS] golden: 8/8 (threshold 1; en 4/4, es 4/4)
  [PASS] grounding: 12/12 (threshold 1; en 6/6, es 6/6)
  [PASS] refusal: 10/10 (threshold 1; en 5/5, es 5/5)
overall: PASS

gauntlet run exits 1 when any gate misses its threshold, so it blocks a merge on its own. It exits 2 when the harness itself could not run, which is a different problem and is reported differently. It exits 4 when the run cannot be scored: the target returned responses with nothing readable in them and no loaded suite would have failed it for that, so a pass rate would be made entirely of checks that silence satisfies. Turn the results file into the evidence pack with gauntlet report.

Read next

• The gate inventory

  What each gate enforces, the case counts per language, and the self-test doctrine that proves every gate can fail.

• The evidence pack

  What a run emits, with real excerpts, and the limits every pack carries on its own face.

• The California mapping

  Its purpose, its limits, the identifiers that were read, and the identifiers that were deliberately omitted.

• The GitHub Action

  Running the gates from another repository, with every input and output the action declares.

Where this comes from

The discipline is drawn from team-scale platform work on a statewide platform: a merge-blocking adversarial suite in English and Spanish, grounding assertions that fail a release when an answer cannot cite its source, golden-answer regression, and refusal and crisis-routing drills. The shared safety infrastructure shipped. The assistant it protected did not launch to residents, because the gates said it was not ready. That judgment is the product this repository makes reusable. Every line here is written fresh; no employer code is included.