How I Cut Our AI API Bill by 95% — The Engineering Playbook

How I Cut Our AI API Bill by 95% — The Engineering Playbook When our finance lead forwarded me the AWS bill for March, I almost choked on my coffee. We were a team of nine engineers shipping AI features, and somehow we'd burned through enough on inference to cover two salaries. The worst part? I hadn't even noticed because the charges were scattered across OpenAI, Anthropic, and a couple of side experiments. That's the moment I decided to actually treat LLM spending like a real infrastructure problem instead of a credit card swipe. What follows is the playbook I wish I'd had on day one. These aren't theoretical tips — they're the exact moves I made across three products to get our run-rate down to roughly 5% of where it started, without shipping worse software. Here's the dirty secret nobody tells you in the LLM hype cycle: most teams default to the most famous model for every single call. GPT-4o for everything. Claude Sonnet for everything. Then they wonder why their "simple AI feature" costs them a kidney. The model selection decision is where I recovered the majority of my budget. When you look at it rationally, the gap between the flagship tier and the cheap-tier models is absurd for tasks that don't require frontier reasoning. This is the matrix I landed on, and it still governs our routing today: Task What I Used To Use What I Use Now Cost Cut Simple chat GPT-4o ($10/M out) DeepSeek V4 Flash ($0.25/M) 97.5% Classification GPT-4o-mini ($0.60/M) Qwen3-8B ($0.01/M) 98.3% Code generation GPT-4o ($10/M) DeepSeek Coder ($0.25/M) 97.5% Summarization GPT-4o ($10/M) Qwen3-32B ($0.28/M) 97.2% Translation GPT-4o ($10/M) Qwen-MT-Turbo ($0.30/M) 97% I want you to really sit with the classification row. Qwen3-8B at $0.01 per million output tokens. That's sixty times cheaper than GPT-4o-mini. For a binary sentiment classifier, the accuracy difference in my benchmarks was under 1.5 percentage points. The ROI math isn't even close. The code for the basic version looks like this. I run this through Global API so I have a single billing surface and zero vendor lock-in: from openai import OpenAI client = OpenAI( api_key="sk-global-xxxxx", base_url="https://global-apis.com/v1" ) MODEL_MAP = { "chat": "deepseek-v4-flash", "code": "deepseek-coder", "simple": "Qwen/Qwen3-8B", "reasoning": "deepseek-reasoner", } def pick_model(user_input: str) -> str: complexity = classify_complexity(user_input) return MODEL_MAP[complexity] def chat(user_input: str) -> str: model = pick_model(user_input) resp = client.chat.completions.create( model=model, messages=[{"role": "user", "content": user_input}] ) return resp.choices[0].message.content That classify_complexity function is itself a cheap-model call. Qwen3-8B looks at the input, decides if it's a "simple" or "reasoning" task, and we route accordingly. The classifier costs us fractions of a cent per request. Model selection is the foundation. Tiered routing is what made the system actually production-ready at scale. The idea: don't ask the expensive model unless you've earned it. I built a three-tier ladder where most requests die on tier one, the harder ones climb to tier two, and only the genuinely hard ones hit the premium tier. def smart_generate(prompt: str, max_budget: float = 0.50) -> str: cheap_resp = call_model("Qwen/Qwen3-8B", prompt) if quality_check(cheap_resp) >= 0.8: return cheap_resp # Tier 2 — Standard at $0.25/M mid_resp = call_model("deepseek-v4-flash", prompt) if quality_check(mid_resp) >= 0.9: return mid_resp # Tier 3 — Premium at $0.78 to $2.50/M return call_model("deepseek-reasoner", prompt) In my actual deployment, roughly 80% of requests resolved on tier one, about 15% escalated to tier two, and only 5% ever saw the reasoning model. The quality_check function runs a small evaluator — sometimes another cheap-model-as-judge, sometimes a heuristic for confidence — and gates the escalation. I shipped this into our customer support chatbot in February. We went from $420/month to $28/month. Same product, same users, same answer quality (we ran blind A/B tests against human raters and the satisfaction scores were statistically indistinguishable). The CFO literally emailed me a thank-you. The architecture decision here matters more than the numbers. Once you commit to tiered routing, you've built a system that can absorb new model releases without rewriting any application code. When DeepSeek dropped a new flash model last quarter, I swapped it into tier two and saved another 30% with a one-line config change. That's the lock-in escape hatch I want from any AI infrastructure. I'll be honest: I was skeptical about response caching at first. "How often do users actually ask the same question twice?" I thought. Then I instrumented it and shut up. FAQ bots, documentation search, "what are your hours," "explain your refund policy" — these hit cache at absurd rates. We're seeing 50–80% cache hit rates on our support surfaces, which means literally half of those requests cost us nothing. Here's the version I run in production. Nothing fancy, just a Redis layer in front of the model call: import hashlib import json import time def cached_chat(model: str, messages: list, ttl: int = 3600) -> dict: key = hashlib.md5( json.dumps({"model": model, "messages": messages}).encode() ).hexdigest() entry = cache.get(key) if entry and time.time() - entry["time"] < ttl: return entry["response"] response = client.chat.completions.create( model=model, messages=messages ) cache[key] = {"response": response, "time": time.time()} return response The semantics matter. I normalize whitespace, lowercase the input, and strip out timestamps before hashing, otherwise nothing ever matches. For semantic caching — "did the user ask something similar even if not identical" — I embed the input with a cheap embedding model and cosine-search the last 24 hours of answers. That added another 15% cache hits on top of the exact-match layer. A subtle but important point: caching also removes tail-latency variance. Your p99 gets dramatically better when a third of requests don't even leave your cache server. That's a UX win you don't get on a spreadsheet. I used to think prompt engineering was about clever wording. Turns out prompt engineering is also prompt accounting. Every token in your system prompt is a token you pay for, on every request, forever. I had a RAG feature where the system prompt had grown to about 2,000 tokens of instructions, examples, and context. At 10,000 requests a day, on DeepSeek V4 Flash at $0.25/M output, that input overhead alone was bleeding cash. Here's the compression pattern I rolled out: def compress_prompt(text: str, target_ratio: float = 0.5) -> str: if len(text) < 500: return text summary = call_model( "Qwen/Qwen3-8B", f"Summarize this in {int(len(text) * target_ratio)} chars: {text}" ) return summary Compressing that 2,000-token system prompt down to 400 tokens saved $0.024 per request. Multiply by 10,000 daily requests: $240/day, which annualizes to $87,600/year. From a single prompt. I genuinely had to double-check the arithmetic. The trick is that the compression itself uses a cheap model, so the meta-call costs almost nothing. You're trading a fraction of a cent of compute for permanent savings on every downstream call. At scale, this is one of the cleanest ROI plays in the whole playbook. This one burned me early. I had a pipeline that processed user feedback by calling the model once per comment. A user with 50 comments triggered 50 separate API calls. Each call paid full price on the input tokens because each one had to re-ship the system prompt. The fix is the obvious one: batch. # Before — N round trips, N system-prompt overheads for comment in comments: response = client.chat.completions.create( model="deepseek-v4-flash", messages=[{"role": "user", "content": comment}] ) # After — 1 round trip, 1 system-prompt overhead batch_prompt = "\n".join(f"[{i}] {c}" for i, c in enumerate(comments)) response = client.chat.completions.create( model="deepseek-v4-flash", messages=[{ "role": "system", "content": "Classify each comment. Return JSON list of {id, label}." }, { "role": "user", "content": batch_prompt }] ) The savings here are typically 10–20% on top of everything else, but the bigger gain is throughput. You go from N sequential round trips to one, and your wall-clock time drops proportionally. For any background processing job — bulk classification, nightly summarization, batch enrichment — this is non-negotiable. Let me sketch what the production stack actually looks like, because no single strategy lives in isolation: Tiered router classifies each incoming request into a complexity bucket. Cache layer sits in front of the model. Exact-match first, semantic fallback second. Compressed prompts get loaded from a versioned prompt store, already trimmed. Cheap model handles the easy 80%, mid-tier handles the next 15%, premium handles the 5%. Batch aggregator collects async work and flushes every N seconds or M items. The whole thing talks to models through a single unified endpoint. That's the architectural decision that ties this all together and keeps me out of vendor jail. I'm running everything through https://global-apis.com/v1 with the OpenAI Python SDK, which means I can swap models, switch providers, or A/B test a new vendor by changing a string in my config. No rewrites. No SDK migrations. No panic when one provider has a regional outage. Here's the production-shaped version with all the layers wired up: python from openai import OpenAI import hashlib, json, time client = OpenAI( api_key="sk-global-xxxxx", base_url="https://global-apis.com/v1" ) CACHE = {} def cache_key(model, messages): return hashlib.md5( json.dumps({"model": model, "messages": messages}).encode() ).hexdigest() def call_with_cache(model, messages, ttl=3600): k = cache_key(model, messages) hit = CACHE.get(k) if hit and time.time() - hit["t"] < ttl: return hit["r"] resp = client.chat.completions.create(model=model, messages=messages) CACHE[k] = {"r": resp, "t": time.time()} return resp def production_generate(user_input: str) -> str: # Compress long user input compressed = compress_prompt(user_input) # Tier 1 r1 = call_with_cache("Qwen/Qwen3-