Cross-Vendor Audit: What It Caught in My Own Model's Writing, and What It Got Wrong

Originally published on hexisteme notes. I write these engineering notes with one main model, and until recently I also reviewed them with that same model. Same family writes, same family checks its own work. That sounded fine right up until I had ten queued posts sitting in a publish backlog and a nagging thought: if the writer and the reviewer come from the same training distribution, what exactly is the review checking for? So I ran an experiment. I took the queue and had a different vendor's model audit it before anything went out — not to replace my own review, but to see what a genuinely different set of weights would flag that mine hadn't. The mechanics were deliberately boring. I copied the ten queued articles into a scratchpad directory and exposed only that copy to the auditor via --add-dir — the auditor never got write access to the originals, so nothing it did could touch the source of truth by accident. The audit itself ran as agy --model gemini-3.1-pro-high, pointed at the copy directory, with one instruction: find technical factual errors, broken sentences, cross-article inconsistencies, unsupported claims, and tone violations, and verify each one yourself on the web before reporting it. I wanted a model that would check its own homework, not just pattern-match on "this looks wrong." It came back with seven findings. Seven findings from a different vendor is not the same thing as seven confirmed bugs. I re-verified every single one independently — grepping the original text, checking official documentation, and where possible checking against a real machine — before touching anything. Of the seven, six held up and got fixed. One didn't: the auditor flagged a sentence as an error, and when I went back to the primary source, it turned out to be the auditor misreading a perfectly correct sentence, not a defect in the writing. Without the re-verification step, I would have "fixed" a sentence that wasn't broken. That's the whole discipline in one sentence: a cross-family audit is a finding generator, not a verdict. If you skip the independent check, you're not eliminating your own family's blind spot, you're just trading it for the auditor's. The three most serious findings — the ones I'd call blockers — had an identical signature: Linux conventions written into code examples that simply don't hold on macOS. All three were sitting in copy-pasteable code blocks, which is what made them dangerous rather than cosmetic. # What the post said What's actually true on macOS 1 Verify a hash with sha256sum macOS doesn't ship sha256sum — the equivalent is shasum -a 256 2 Inspect a plist with plutil -convert json <file> That invocation destroys the original file in place — writing to stdout requires -o - 3 Implement a "nag once" guard as an in-memory set Hooks spawn a fresh process on every single invocation, so in-memory state never survives between calls — this silently becomes "nag every time," and the guard needs to live on disk instead Line up what each one does to a reader who trusts the code block and pastes it in: the first one just breaks immediately, a command-not-found. The second one is worse — it quietly destroys the very file you were trying to inspect. The third is the sneakiest of the three, because nothing crashes and nothing errors; the guard just silently fails to do the one thing it was written to do, and you'd only notice from the symptom, not from any error message. The other three confirmed findings were smaller but still real. One was a broken sentence — leftover debris from an earlier editing pass that never got cleaned up. One was a cross-article inconsistency: two different posts in the queue stated the same underlying fact in ways that didn't quite agree with each other, fixed by adding a hedge to the looser of the two so they lined up. The last one was the most interesting: a number attributed in the text to "our internal evaluation" that, on checking, actually traced back to a public paper (arXiv 2310.13548) rather than anything I'd measured myself. The fix wasn't deleting the number — the number was fine — it was giving it the right citation instead of an implied one that overstated how much of that result was mine. None of the blocker findings are subtle once you see them. sha256sum not existing on macOS is not an obscure fact. So why did a model write it, and why did the same-family review pass it? My working explanation is that shared training distribution produces shared error distribution. Conventions like sha256sum and destructive plutil usage are the kind of thing that shows up constantly in Linux-centric documentation and Stack Overflow answers, which means a model trained heavily on that material reaches for them by default — and if the reviewer is trained on largely the same material, it reaches for the same defaults when checking, so nothing looks wrong. Author and reviewer aren't independent checks on each other if they're wrong in the same direction for the same reason. This is really just the content-QA version of a limitation that shows up under other names too: a judge that shares a policy model's blind spots isn't a real check on that policy, because "checking" and "generating" are drawing from the same well. The value of routing the audit to a different vendor isn't that gemini-3.1-pro-high is smarter about macOS specifically — it's that its errors, if any, are uncorrelated with mine, so the things it does get right are the things most likely to be exactly what my own review structurally can't see. One detail that's easy to overlook: these posts exist in two places at once — a source markdown file in the publish queue and an already-live canonical HTML page. Fixing a confirmed finding in only one of those two locations doesn't close the bug, it just adds a new inconsistency between the draft and the thing readers are actually looking at. Every confirmed fix went into both the queue markdown and the live canonical HTML in the same commit before pushing, specifically because a crosspost setup punishes half-fixes. The whole procedure is cheap enough that there's no real argument against running it every time: 1. cp queue/*.md scratchpad/audit-copy/ # protect the originals 2. agy --model gemini-3.1-pro-high --add-dir scratchpad/audit-copy \ "find technical factual errors / cross-article inconsistencies / broken sentences in these posts, and verify each one via web search" 3. For each of the N findings: re-verify independently — grep the primary source, check official docs or a real device — never take the report as-is 4. Only confirmed findings get fixed, in both the queue md and the live canonical html, together — then push A copy directory, read-only exposure, and one prompt — no pipeline changes required. Out of ten posts, seven flagged findings, six confirmed and fixed, one correctly thrown out, and zero tone or style complaints from a model with no particular reason to share my writing voice. The takeaway I'm carrying forward isn't "cross-vendor audits are always right" — one in seven wasn't. It's that a same-family review loop has a structural hole exactly where the writer and the checker learned the same defaults, and the only way I've found to see that hole is to put a genuinely different model on the other side of it, then refuse to believe anything it says until it's been checked against the primary source. Next queue of ten-plus posts gets the same pass before anything goes out. More notes at hexisteme.github.io/notes.