Fail the build when your prompt gets dumber: evalgate for prompt regression CI

Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality. So I built evalgate, a small TypeScript tool that treats prompt and agent quality like a build artifact. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score regresses. Then it posts the delta table as a PR comment. The important design decision is what question CI is allowed to ask. "Is this prompt good?" is subjective and unwinnable in an automated gate. "Is this worse than it was on main?" is objective and answerable. evalgate is built around that second question. You capture a baseline once, and from then on every change is judged as a delta against it, not against some absolute notion of goodness. The second decision was that the whole thing has to run with zero API keys. evalgate ships a deterministic mock provider, so you can run a suite, save a baseline, compare runs, and execute the full test suite completely offline. The project itself has 67 tests and none of them touch the network. Every feature has to work in mock mode before it counts as done. A suite is a YAML (or JSON) file that lives in version control next to the code it checks. Each case has an input, an expected reference value, and one or more scorers. Here is a minimal one: name: my-agent provider: mock # works with no API key threshold: 0.9 # mean score required to pass cases: - id: greeting input: prompt: | Reply with the standard greeting. exactly: Hi there! How can I help you today? expected: "Hi there! How can I help you today?" scorers: - type: exact-match - type: latency budgetMs: 500 A case passes when every one of its scorers passes, and its numeric score is the weighted mean of the individual scorer scores. There are 10 scorers in the catalog, covering the range of things you actually want to assert about model output: exact-match, regex, contains, and not-contains for string-level checks (contains gives partial credit across multiple substrings). json-schema for structured output, so you can assert the model returns valid JSON matching a schema. embedding-similarity for "close enough in meaning" via cosine similarity. llm-judge and rubric for the softer, criteria-based judgments. latency and cost for budget gates, so a change that makes the agent slow or expensive can also fail the gate. Two of those scorers are pluggable and ship with deterministic offline fallbacks, which is what keeps the mock-first rule honest. embedding-similarity uses the provider's embed() if it has one, and otherwise falls back to a stable local bag-of-hashed-words embedding. llm-judge calls a real provider and parses a {score, reason} JSON reply, but on the mock provider it computes a reproducible word-overlap score instead. So a suite that uses judges and embeddings still runs identically on every machine with no keys. The workflow is three commands. You run a suite, save a baseline, then compare later runs against it: npx @royalpinto007/evalgate run suite.eval.yaml npx @royalpinto007/evalgate baseline suite.eval.yaml --out baseline.json npx @royalpinto007/evalgate compare suite.eval.yaml --base baseline.json --tolerance 0.01 compare exits non-zero when any case regresses beyond the tolerance, and that non-zero exit is what fails the CI job. The tolerance matters because model output is not perfectly stable; you usually want a small allowed drift before something counts as a real regression. The part that makes it feel like CI rather than a script is the GitHub Action. On each pull request it re-runs the suite and upserts a single comment, editing its own comment in place instead of stacking a new one on every push. A regression renders like this: ### evalgate: support-agent FAIL - Quality regressed. 5 case(s) got worse. Overall score: 94.2% (base) -> 60.1% (head) = -34.2pp | Case | Base | Head | Delta | Change | | ------------------- | ------ | ----- | -------- | ------ | | refund-intent-json | 100.0% | 0.0% | -100.0pp | down | | order-id-format | 100.0% | 0.0% | -100.0pp | down | | greeting-exact | 100.0% | 66.7% | -33.3pp | down | | judge-helpfulness | 73.6% | 69.1% | -4.5pp | down | | paraphrase-quality | 86.0% | 84.6% | -1.3pp | down | You get the overall movement in percentage points and a per-case breakdown of exactly what got worse, right in the review, before the merge. Under the hood the same logic is exposed as a library, so loadSuite, runSuite, compareRuns, and renderCompareMarkdown are all importable if you want to wire evalgate into something other than the Action, and both scorers and providers are registrable so you can add your own. evalgate tells you the score moved; it does not tell you whether the new score is the correct one. If your prompt genuinely got better and the reference expectations are now stale, evalgate will still flag a delta, and it is on you to update the baseline. The gate is a change detector, not an oracle. That is also true for the softer scorers: llm-judge and embedding-similarity are only as trustworthy as the judge model and the criteria you write, so a green suite built on weak criteria is a false sense of safety. I lean on the exact, regex, and schema scorers for anything I want to be strict about, and treat the model-based scores as directional signal rather than ground truth. The thing I wanted was simple: make prompt quality something a pull request can fail on, the same way a broken type or a failing test does. evalgate does that with a declarative suite, a baseline plus delta engine, 10 scorers, and a GitHub Action that comments the regression inline, all runnable offline through a deterministic mock provider. Repo: https://github.com/royalpinto007/evalgate Package: https://www.npmjs.com/package/@royalpinto007/evalgate