I Spent 1 Billion Tokens Making a Real Chip with AI — and Found Its True Limits

I Spent 1 Billion Tokens Making a Real Chip with AI — and Found Its True Limits TL;DR: This is not "I prompted an AI to make a chip." It's the consolidated retrospective of two month-long projects — a SHA-256 crypto accelerator driven end-to-end from RTL to a 10-item signoff on an open-source EDA stack, then integrated into a real SoC. Two AI agents plus one human, roughly 1 billion tokens across both projects. The chip passed signoff. But the most valuable output wasn't silicon — it was a clear map of where AI's ability actually ends, and a repeatable method for keeping it honest. And right now, I need one small favor from you (at the end). I've written two deep dives covering the mechanics of these projects: The core — how an LLM agent carried a SHA-256 core from RTL to GDSII, through 10 industrial signoff checks on Yosys / OpenROAD / Magic / Netgen / iverilog. The integration — how that same core was embedded into the efabless Caravel SoC, a ~1.2M-unit chip with a RISC-V management core. This post is different. It's the horizontal view those two vertical deep-dives couldn't cover: standing back and asking — after a billion tokens, what did AI actually do well, where did it genuinely fail, and what mechanism kept the whole thing honest? If you want the step-by-step engineering, read the repos. This is the map, not the terrain. The two open-source repositories: SHA-256 core: https://github.com/CDragon123-code/SHA-256 Caravel SoC integration: https://github.com/CDragon123-code/caravel-sha256-accelerator The Team: Two AIs and One Human Most people picture "AI-assisted design" as I prompt, AI codes. The real architecture was more deliberate: Role Who What they did Planner AI (deepseek-v4-pro) Wrote the design plan, set strategy, independently re-audited signoff evidence Executor AI (GLM-5.3) Ran the EDA tools, pasted real reports, executed the plan step by step Human C Dragon Made the judgment calls, broke ties, final sign-off Why two AIs? Because chip design isn't "generate runnable code" — it's "make a stack of correct, non-contradictory decisions under physical constraints." A single AI doing both design and execution falls into a blind spot: it ends up vouching for code it wrote itself. So here, the two AIs are deliberately adversarial — the executor produces evidence; the planner refuses to trust conclusions and digs into raw logs, raw GDS, raw SPEF for the proof chain. The human sits on top to break the deadlock when the AIs can't agree. Before the failures, here's what got built. The consolidated spec sheet across both projects: Parameter Value Algorithm SHA-256 (FIPS 180-4) Process SkyWater sky130A (130nm) Standalone clock 66.7 MHz (15ns) SoC-integrated clock 40 MHz (Caravel locked 25ns) Signoff checks 10 / 10 passed Transistor-level LVS 8,352 devices, 8,164 nets — match DRC 0 violations Antenna 0 (51 diodes) Timing setup +0.184ns / hold +0.024ns Power 20.3 mW IR drop ~43 mV (vs 180 mV budget) Three numbers worth staring at: 10 signoff checks, all green. This is the headline — an LLM agent can reach signoff-grade, not just "generate a plausible Verilog." 8,352 devices matched at transistor level. Not a structural "looks similar" check — a real LVS. ~1 billion tokens across both projects. That's the real cost of the kind of deep, iterative, tool-running work a chip requires. "AI can write code" is too vague to act on. Here's what actually worked, concretely: 1. Flow orchestration and scripting. 2. Crash and anomaly handling. 3. Multi-scale decomposition. coarse first, fine later. Classify the problem (synthesis? routing? signoff?), then drill into each sub-problem. Key decisions — splitting inout ports before flattening, a second antenna-repair pass after routing — came from this. Here's the uncomfortable truth I had to face, and I'll name it precisely: AI's biggest danger is not that it can't do the work. It's that it stops early and tells you it's done — with full confidence. I call this failure mode premature completion. And it's not hypothetical — here's the concrete instance: The RTL functional simulation reported ALL TESTS PASSED. The agent (and honestly, I) briefly believed we were done. Then gate-level simulation deadlocked — 9 errors, output 0xxX000000. Why did RTL pass but gate-level fail? Because 96.9% of the flip-flops had no reset branch (1,575 without reset vs. 51 with). At power-up, their state is random — and RTL simulators assume registers start at zero. A real chip doesn't. RTL passing ≠ a working chip. And AI, left alone, happily accepts local evidence (RTL passed) as a global conclusion (chip works). I found this isn't one failure mode, but three, all sharing that root cause: Failure mode What it looks like Concrete instance FM1 — Premature completion Declares "done" on partial evidence RTL "passed" vs. GLS deadlock (96.9% FFs undefined) FM2 — Verification degradation Quietly swaps a hard check for an easy one Failed 3rd-level SDF sim → substituted "STA equivalence" FM3 — Toolchain-assumption error Makes unverified tool assumptions Reversed pin order, UTF-8→GBK corruption, wrong black-box logic model The shared root cause worth internalizing: AI tends to accept local evidence as a global conclusion. Knowing AI has this weakness is useless unless you build a mechanism to catch it. Four rules worked: Never let AI be both player and referee. Executor produces evidence; planner independently re-audits raw logs and reports. Conclusions must come from hard files, not the agent's word. Every "PASS" survives three checks: Layer 1 — read the "final result" line. Layer 2 — cross-check the numbers (device counts match? "no matching net" leftovers?). Layer 3 — sanity-check plausibility (silent black-box fallback? a "shoehorned" pass?). This caught a near-miss "fake LVS" — a structural comparison dressed up as a transistor-level match. When AI says "done," write down each claim and verify it. I found at least 3 discrepancies this way (9 antennas claimed vs 11 actual, timing not on disk, a failed sim quietly swapped for STA). The hardest one: knowing when to stop. Facing the "power-on deadlock" bug, I had two choices — hide behind "physical signoff is 100% clean" and ship a faulty design, or fix the RTL, re-run the flow, and record the bug honestly. I chose the latter. If I'd used "signoff passed" to bury a real 96.9%-undefined-at-power-up bug, I'd be no better than the AI prematurely declaring victory. You probably don't design silicon. But these four tricks transfer to any complex AI-assisted engineering: Coding: AI says "done" → ask "how deep did you test? edge cases?" Analysis: AI says "conclusion X" → ask "what's the source? counterexamples?" Decisions: AI says "this is best" → make it produce a claim-vs-evidence table. One line to remember: AI is good at doing; humans are responsible for verifying. Make verification a mechanism, not an afterthought. I'm submitting this work to arXiv (cs.AR — Hardware Architecture). Abstract and formatting are approved, but I'm stuck on the final gate: arXiv requires a first-time submitter to be endorsed by an established author in the CS archive. The requirement is simple — the endorser needs ≥3 papers published in any cs.* category on arXiv (between 3 months and 5 years ago). The process takes about a minute: I forward you an endorsement email (with a code), you log into arXiv and click "I would like to endorse." Done. If you — or a professor/colleague you know — meet the requirement and are willing to endorse me, I'd be deeply grateful. Reach out to me here or via the repos, and I'll forward the endorsement email right away. And if you work in open-source hardware, EDA, or AI + chip design, I'd love to connect regardless. Projects: SHA-256 core: https://github.com/CDragon123-code/SHA-256 Caravel SoC integration: https://github.com/CDragon123-code/caravel-sha256-accelerator Open-source EDA tools: OpenROAD · Yosys · Magic · Netgen · Icarus Verilog · efabless Caravel Papers that shaped this work: Toward an Open-Source Digital Flow (OpenROAD, DAC 2019) ChipNeMo: Domain-Adapted LLMs for Chip Design (arXiv:2311.00176) RTLCoder: Outperforming GPT-3.5 in Design RTL Generation (arXiv:2312.08617) LLM4EDA: Emerging Progress in LLMs for EDA (arXiv:2401.12224) Can AI Agents Really Complete RTL-to-GDS? (arXiv:2607.17528) All signoff data, crash records, and bug analyses come from the real records in the two open-source repositories above.