Versioned behavioral state
.aistate.yml declares prompts, models, evals and probes; snapshots record them alongside observed behavior.
Open source · LLM evaluation
AIDrift: behavioral drift detection and release gates for AI.
AIDrift treats prompts, model settings, eval expectations, provider probes and observed outputs as versioned behavioral state. It captures a baseline, re-samples behavior on every change and fails the build only when a regression is statistically supported, so an LLM feature cannot quietly change before it reaches users. Think of it as a Terraform plan for AI behavior.
v0.9.0-beta.1Apache-2.0TypeScript, Node 22+GitHub Action
What AIDrift is
AIDrift is a local-first, Git-native command-line tool for versioning, diffing, testing and gating the behavioral state of AI systems. It snapshots prompts, model configuration, evals and provider probes, compares sampled outputs against a committed baseline with Fisher exact and Welch t-tests, and exits with code 1 when behavior has regressed.
What it is not
- It is not a hosted service or dashboard. There is no telemetry, and state stays in your repository.
- It is not an LLM-as-judge framework: llm_judge, tool_call, latency, cost and custom assertions are rejected in this release.
- It does not execute RAG pipelines, tools or adapters yet. Manifests that declare them fail closed.
- The offline mock provider proves the workflow, not the quality of a real model.
At a glance
AIDrift in eight facts.
- Type
- Open-source llm drift testing tool
- Licence
- Apache-2.0
- Language
- TypeScript
- Runs on
- Node.js 22.14 or newer, GitHub Actions
- Latest release
- v0.9.0-beta.1, 9 Sep 2026
- Status
- Public beta
The problem
Prompt edits, model version bumps and silent provider updates change what an AI feature does, and most teams find out from their users. Ad hoc prompt checks pass or fail on a single sample, which is noise: a model that answers correctly four times in five can look fixed or broken depending on the run.
How AIDrift approaches it
AIDrift samples each assertion several times, keeps the full baseline and current distributions, and reports drift only when a statistical test supports it. Artifact changes are shown as context rather than failures, because an intentional prompt change is not itself a regression; the decision rests on sampled behavior.
How it works
From input to decision.
01 InitScaffold .aistate.yml
02 ValidateParse and check the manifest
03 SnapshotCapture a behavioral baseline
04 PlanSample and compare behavior
05 Probe20 provider-drift probes
06 CheckCI gate: exit 0, 1 or 2
Capabilities
What it does.
Statistically supported verdicts
PASS, WARN, DRIFT, INSUFFICIENT, ERROR or NEW, backed by Fisher exact or Welch t-test evidence.
20 built-in provider probes
Deterministic, structural, semantic, behavioral and performance probes catch a provider changing underneath you.
OpenAI, Anthropic and offline mock
Live adapters for OpenAI Chat Completions and Anthropic Messages; a deterministic mock runs with no key, network or cost.
Enforced cost ceilings
Live-provider checks need an explicit cost budget, and a model with an unknown price can never pass a budget implicitly.
GitHub Action with evidence
Workflow annotations, JSON and JUnit evidence uploads, and one upserted pull-request comment per run.
Quickstart
Running in five minutes.
An offline five-minute quickstart: init creates a mock model, an eval target and a starter assertion, so no API key or YAML editing is needed. The last command exits 0. Replace the starter assertion with domain-specific behavior before using AIDrift as a production gate.
Install
npm install --global @zettacore/aidrift@next
mkdir aidrift-demo && cd aidrift-demo
aidrift init --yes
aidrift validate
aidrift snapshot --with-evals --with-probes --samples 5 --label baseline
aidrift plan --samples 5
aidrift check --samples 5Pull-request gate
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: Parth2412/aidrift/packages/action@v0.9.0-beta.1
with:
samples: "5"
timeout: "120"
comment-mode: upsert
github-token: ${{ github.token }}Reference
Eight commands, one workflow.
Every command available in 0.9.0-beta.1. check is the one CI runs.
| Command | What it does |
|---|---|
aidrift init | Scaffold .aistate.yml for an existing AI project |
aidrift validate | Parse and validate .aistate.yml |
aidrift snapshot | Capture declared artifacts and optional behavioral baselines |
aidrift history | List all local snapshots |
aidrift diff | Compare current state, or two snapshots |
aidrift plan | Execute the declared AI target and compare sampled behavior |
aidrift probe | Run provider-drift probes |
aidrift check | Run the non-interactive CI regression gate |
Who it is for
Built for.
Teams shipping LLM features
Catch a prompt or model change that alters behavior before it merges, not after users report it.
Platform and MLOps engineers
A non-interactive gate with schema-versioned JSON output, run-level deadlines and enforced cost ceilings.
Products built on third-party models
Probe for provider-side drift when a hosted model changes without a version bump.
Teams that must explain AI behavior
A versioned, reproducible baseline of what the system did at each release.
Limitations
What it does not do yet.
Stated in the project’s own documentation, so you can decide where it fits.
- Prerelease software: CLI, TypeScript, evidence-schema and manifest details can change before 1.0.0.
- Live-provider checks make billable calls, need an explicit cost ceiling, and are not retried automatically.
- Supported assertion types are contains, regex and json_schema; JSON Schema patterns use RE2 syntax.
- Snapshots are stored locally only. There is no PyPI package, cloud service or dashboard.
Maintainer
Built and maintained by Parth Kaloliya
Founder & Director, ZettaCore Labs. AIDrift is developed in the open: issues, pull requests and the full release history are on GitHub.
More open source
The other tools.
AI & Intelligent Systems
We build applied AI systems - retrieval, agents, forecasting and vision - that sit inside real workflows and are evaluated against your data, not a public leaderboard.
Software Engineering
Web, mobile, SaaS and enterprise systems built on durable foundations - clear domain models, typed boundaries and tested paths.
FAQ
Questions we get asked.
AIDrift is an open-source, Git-native CLI and GitHub Action from ZettaCore Labs that versions the behavioral state of AI systems, including prompts, model settings, evals and provider probes, and fails a CI check when sampled behavior regresses against a committed baseline with statistical support.
An eval suite typically scores outputs. AIDrift compares sampled behavior against a committed baseline, treats prompts and model settings as versioned state, and turns the comparison into a pass or fail CI decision backed by a statistical test.
Not to try it. The deterministic mock provider runs offline with no key, network request or cost. Live checks against OpenAI or Anthropic read credentials only from environment variables and require an explicit cost budget.
aidrift check returns 0 for a passing gate, 1 for a statistically supported regression, and 2 for a configuration or runtime failure. Artifact changes alone never fail the check.
contains, regex and json_schema. Regex assertions are screened for unsafe backtracking and have a 250 ms execution deadline, and JSON Schema patterns use RE2 syntax.
It is a public beta, 0.9.0-beta.1, published on npm’s next channel. Pin an exact version and replace the starter assertion with domain-specific behavior before using it as a production gate.
Want these controls in your delivery pipeline?
We implement provenance gates, LLM regression testing and secrets hygiene for teams shipping AI and fintech systems. Tell us what you run today.