We Made Two AIs Debate and Hired a Third as the Arbiter — 6 Test Rounds, 7 Bugs Fixed

gging log, costs included. Pattern Mechanism Measured cost Use when Cross-vendor review A drafts → B structured-review gate → C revises ~$0.001/run Default quality tier Parallel + judge N models answer → judge picks by hard constraints $0.0013/run Tasks with length/format caps MoA aggregation N models answer → synthesizer merges the best of each $0.006/run Open-ended generation Revision loop Review bounces → revise → re-review → escalate to a stronger model when exhausted $0.002/escalation Unstable first drafts Debate Two debaters open independently → K rebuttal rounds → arbiter verdict $0.02–0.04/run Analysis / decisions / risk assessment The single most useful finding: pattern-task fit beats model quality. With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs (rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps), while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you "the better of two drafts"; only synthesis takes the best of both.

Five samples across six rounds: Qwen-plus self-reported "exactly 10 words" — actual: 14 Qwen-plus self-reported "about 580 characters" — actual: 799 Claude haiku under-reported by 1–38 chars in earlier runs No model's self-count is trustworthy. So bayze treats length like memory safety: the engine counts, deterministically — every output passes max_chars / max_words checks, and the model's opinion is never consulted. We stress-tested strategy: debate with a real question: "Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?" Rounds 1–2, two invisible bugs. The test agent complained "my constraints are being ignored." They were — but not by the models: the context field was accepted by the MCP layer and silently dropped by the orchestrator.

Separately, maxOutputTokens: 2000 allowed ~3000 characters of output while a max_chars: 2000 gate rejected at 2000 — the engine was permitting output it was guaranteed to reject. Round 3, the debate actually worked. Rebuttals quoted each other point by point. One side caught the other promising "RPO < 5s" in its opening while admitting "primary failover takes 3.2 hours" in the same document.

Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run. Rounds 4–5, engineering answers to "vibes" problems. Models treated "opening / rebuttal / verdict" discipline as their assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position.

A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by self-calibration: measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen fabricated "we audited this team's real invoices (redacted)" plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict. Round 6, the last design flaw.

Both models ended up on the same side ("worth migrating"). Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional stances: [pro, con], with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it.

Condensed from two debates plus verdicts: Don't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap (containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first) — then using "we can leave" as negotiating leverage.

Caveat that matters more than the conclusion: every dollar figure both models produced was fabricated (the arbiter flagged and excluded them). What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible. npx -y bayze-mcp@0.14.0 mcp --config your-config.yaml GitHub (bilingual README): https://github.com/caijianxiong2007-prog/bayze-mcp Gitee mirror: https://gitee.com/collection-master/bayze-mcp npm: https://www.npmjs.com/package/bayze-mcp Every pattern's design trade-offs are recorded as ADR D1–D8 in docs/adr/0006-multi-ai-collaboration.md MIT, local-first, keys never leave your environment, 193 automated tests.

If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.