Platform Strategy That Ships: The 90-Day System I Trust More Than AI Roadmaps

I want to start with a failure, because it taught me more than the wins. We built an AI code review bot. The pitch was simple. It sounded like free velocity. The first thing it did was add about half a day to PR cycle time. Not because it broke CI. By around day 60, engagement in the targeted group dropped into the single digits. That was the lesson. If automation isn't trusted, it doesn't save time. I used to think that was a soft, cultural problem. "We need better change management." Now I think it's much more practical, and much more predictable. If a tool is noisy, it doesn't just fail to help. People become cautious. That is not an AI problem. The success criteria looked reasonable on paper: Engagement: >80% meaningful interaction (open the review + leave a signal) Competitive: >25% more helpful than a Copilot-style review experience (Copilot stayed on, we inserted our tool in the same path, and compared thumbs up/down + which suggestions were accepted) Satisfaction: >70% across our language mix (Swift, Kotlin, BrightScript, TypeScript, C++, Java, Go, Python) Outcomes: reduce iteration from ~4-5 commits/PR to ~2 and PR cycle time from ~2.5 days to ~1 day And we had a cultural motivation that never shows up in a slide deck: If you can get to "AI-first pass, then one human pass," you don't just save time. We also had a cost model. We ran ~8,000 PRs/month. PR authors iterated ~4-5 commits. $6. If AI review could save ~3 iteration commits per PR, that's ~24,000 fewer commit-runs/month, or roughly ~$144K/month. (Commits don't always map 1:1 to CI runs. Treat that as an order-of-magnitude model.) We ran the experiment on a fraction of traffic. ~1,500 PRs. At first, the bot looked like it was "working". Then we did the one thing that matters. When we asked developers if it helped, 65%+ said no. The quote that ended the debate was brutally specific: "Maybe one useful comment out of 10, and some are wildly wrong." And it wasn't a clean "works" vs "doesn't work" story. Best: Java and Python Worst: Swift and BrightScript In 2025, that cross-language gap was big enough to break trust. What I didn't anticipate: noise is not neutral. It also creates a weird new kind of social load. If a human reviewer leaves a bad comment, you can talk to them. Once a bot is in the workflow, the default expectation becomes: "someone is accountable for what the bot says." That someone is usually the platform team. The most humbling part: it didn't just fail to help. Early on, it made things worse. ~0.5 days because authors read suggestions just to discard most of them. This is the part I wish more leaders would internalize. Bad automation doesn't look like failure. It just doesn't look like impact. If you are reading this and thinking, "but that's just tuning," I agree. The problem is what happens to trust while you tune. So we killed it. Not because killing is fun. It was the moment we stopped pretending that "shipping" is the same thing as "posting a bot comment". If you're wondering what "kill" looked like in practice: we wrote the closure report we summarized the evidence we made it explicit what we'd keep (telemetry, infra patterns, some of the prompt work) and we made it explicit what we'd stop paying for (comments that developers didn't trust) That doesn't make the miss feel good. After that, I stopped trusting slide-deck strategy. pick a small number of bets name owners define ship criteria before build define kill criteria before build keep a decision log The reason I like the 90-day framing is that it forces honesty. You can hide behind a roadmap for a year. By day 30, you should know whether you have a signal. The mistake I see teams make is dragging the decision out because the pilot is "almost good enough". If you're in that place, you're not running a bet. In our case, engagement fell off a cliff around day 60. This isn't just for AI. It's platform strategy. These were separate 90-day cycles across different years, not one big program. 2018: telemetry + anomaly detection + correlation It reached ~20 DAU / ~150 MAU. ~$5M avoided (directional). Here's what people miss. Also, this bet didn't succeed because we built a beautiful dashboard. Alerts showed up in Teams and email. 2023: CI/CD runners on AWS ~$120K/month less in direct runner infrastructure spend and ~30% faster on the workflows that mattered. That bet wasn't glamorous. If you're trying to make any kind of AI program work, this is the boring prerequisite. 2025: Backstage as an internal developer portal ~30 DAU / ~500 MAU. This bet taught me something uncomfortable. If developers don't know where to go to onboard, request access, provision a service, or find ownership, AI doesn't fix it. Backstage worked for us because it made one path the default. And the AI model access workflow mattered more than I expected. That shift is quiet. The model landscape is better now than it was in 2025. But I would change the shape of the bet. I would segment by language from day 1 (different thresholds, different model choices) I would start with "summary only" and "ask-to-expand" rather than defaulting to full comment spam I would treat false positives as the primary enemy, not missing edge cases I would measure acceptance and downstream rework, not just thumbs up/down Most importantly: I would assume trust is fragile. If you're leading platform or AI work right now: What are your kill criteria? Have you killed something popular? Have you had a pilot that made cycle time worse before it got better? And the one I'm genuinely curious about: If you could only run one 90-day bet next quarter, what would you pick and what would make you stop?