Your CLAUDE.md has zero tests. Here's what happened when I broke mine on purpose.

Claude Code shipped roughly 25 releases last month. My setup, 400 lines of CLAUDE.md, four skills and a guard hook, had exactly zero tests against any of them. That bothered me, because this configuration has behavior, and behavior breaks. A new model version ships and suddenly a skill stops triggering. Nothing errors. Nothing goes red. The agent just quietly stops doing the thing you taught it, and you notice three weeks later when the code review comes back weird. So I built CI for it: config-drift-checker turns your CLAUDE.md, skills and hooks into eval cases and re-runs them on every Claude Code release and every PR that touches the setup. But a tester you've never seen fail is worthless. So I sabotaged my own setup to test the tester. I deleted the skills entry from my plugin manifest, expecting everything to break. Nothing broke. Claude Code auto-discovers the skills directory, so the manifest key does nothing. My first sabotage was a no-op, which is exactly the kind of thing you only learn by trying to break your own system. This time I rewrote one skill's trigger description the way a careless PR would. The skill still existed, still had all its content, but its description now talked about Terraform instead of Spring services. Before After sabotage Suite score 1.00 0.36 Tripwire case (did the skill fire?) 1.00 0.00 Content cases 1.00 0.33 to 0.60 The content cases dropped because the agent no longer followed conventions it used to follow. And one case went to exactly zero: the tripwire. A tripwire case has a single grader: Was the skill actually invoked? Not "did the output look right". Just "did the trigger fire". When a trigger breaks, that case reads 0 out of 3 runs, every run, every day, and no amount of model randomness produces that pattern. One failed run doesn't mean your rule broke. Models are stochastic; sometimes they just ignore an instruction once. Telling "rule stopped firing" apart from "model ignored it run" turned out to be the real engineering problem. Three runs per case. A genuine flake usually recovers within the same three runs. A broken trigger never does. Noise bands learned from history. Each case's expected spread comes from its own past runs. One of my cases naturally swings by 0.75, so a fixed threshold would either alarm daily or catch nothing. A drop inside the band gets an amber "noisy" flag, never a red check. Guards so real breaks can't hide in the band. If no current run reaches the baseline, or the case has been below the bar for consecutive runs, it escalates to red anyway. Five failures in a row is not noise. Likely refusals (zero tool calls, one turn, short reply) get labeled separately, so they don't masquerade as broken rules. The whole broken run is published, unedited: The red report from the sabotage run It runs as a GitHub Action on your runner with your key: - uses: jameskomo/config-drift-checker/action@v0 with: plugin-dir: . Costs $0 on a Claude Pro or Max plan via a subscription token. Repo: https://github.com/jameskomo/config-drift-checker If you've built your own version of this in-house (I keep meeting people who have), I'd genuinely like to compare notes, especially on how you separate "rule stopped firing" from "model ignored it this run".