AI SaaS: Cut the Bill Before You Buy More GPUs

The board deck said to scale inference. The Grafana dashboard showed a different bottleneck: Postgres CPU sat at 78%, and GET /v1/models handled 40,000 requests per minute, all returning identical JSON. This pattern shows up repeatedly at AI startups that already solved streaming completions. The expensive GPU fleet gets all the attention, but boring HTTP traffic often breaks the infrastructure first. Model lists, pricing sheets, feature flags, public prompt templates, and dashboard summaries hammer the primary database on every page load and client poll. Here is how to offload that read traffic at the edge without touching your streaming chat pipeline. When we examined the telemetry from three production teams (a B2B coding copilot, an automated legal assistant, and a customer support agent platform), the request distribution looked almost identical: Traffic type Share of requests Cost profile Streaming completions (POST) 8% to 15% High per request (GPU time, token counts) Embedding generation jobs 5% to 15% Moderate (vector API batches) Read APIs (models, configs, pricing, prompt libraries) 70% to 85% Small per request, massive in aggregate A single request to GET /v1/models takes negligible CPU. But 40,000 requests per minute against an ORM query that joins model variants, token pricing tiers, context windows, and provider availability will saturate a database connection pool. When your web frontend, mobile client, and VS Code extension poll that endpoint on every reload, you pay a heavy database tax for static data. /v1/models hits the database so hard In most codebases, GET /v1/models is not a flat JSON file. It is an ORM call that looks like this: # FastAPI + SQLAlchemy example @app.get("/v1/models") async def list_models(db: AsyncSession = Depends(get_db)): query = ( select(Model) .options( joinedload(Model.pricing_tiers), joinedload(Model.provider_status), joinedload(Model.supported_parameters) ) .where(Model.is_active == True) .order_by(Model.display_order) ) result = await db.execute(query) models = result.unique().scalars().all() return {"data": [m.to_dict() for m in models]} Every incoming client request does three things: Acquires a connection from your database connection pool (e.g. PgBouncer or SQLAlchemy pool). Runs a multi-table SQL query with multiple LEFT OUTER JOIN clauses. Serializes several hundred rows into JSON. Under burst traffic, connection pools fill up. New chat sessions fail to acquire database connections to save conversation state, leading to 500 errors during peak user activity. The GPU cluster is completely idle, but the application fails because the catalog API exhausted the database pool. ApexCache only caches HTTP GET and HEAD methods. Every other method passes directly through to your origin servers. This aligns cleanly with standard AI API design: GET /v1/models: Cache for 60 to 300 seconds. Add a surrogate tag (Cache-Tag: models). GET /v1/pricing: Cache for 300 to 1800 seconds. GET /v1/prompts/public: Cache for 60 to 300 seconds with query string preservation. GET /v1/docs or GET /openapi.json: Cache for 3600 seconds. Static eval leaderboards and benchmark results: Cache for 600 seconds. POST /v1/chat/completions: Streaming tokens and user prompts must never touch an edge cache. POST /v1/embeddings: Unique text embeddings belong on the origin. Tool execution webhooks: Dynamic and stateful. User account settings, billing invoices, and private conversation history: Authenticated per-tenant routes must bypass caching. You are not caching the AI engine. You are caching the menu that every user reads before placing an order. To keep data fresh without waiting for long TTL timeouts, return a Cache-Tag header from your origin server. When your marketing or engineering team updates a model parameter or pricing tier, invalidate that specific tag: // Express / Node.js origin response app.get("/v1/models", async (req, res) => { const models = await fetchModelsFromPostgres(); // Instruct edge proxy to cache and tag this response res.setHeader("Cache-Control", "public, s-maxage=300"); res.setHeader("Cache-Tag", "models,catalog"); res.json({ object: "list", data: models }); }); When you deploy a new model variant or adjust rate limits, trigger a tag purge through the ApexCache API: curl -X POST "https://api.getapexcache.com/api/v1/cache/invalidate" \ -H "Authorization: Bearer $APEXCACHE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"tags":["models"]}' Every edge node drops the cached /v1/models payload in under 10 milliseconds. The next request fetches fresh data from your origin and repopulates edge memory. Consider an AI SaaS application handling 30 million read API requests each month: Origin baseline: Node.js API to AWS RDS Postgres (db.r6g.xlarge, ~$520/month plus read replica). Average latency is 45ms. Edge cache hit rate: 88% on catalog, pricing, and public prompt routes. Origin reads dropped: From 30 million down to 3.6 million per month. Average latency: Drops from 45ms down to 4ms for cached requests served from memory. By reducing read QPS against the primary database by 88%, the team avoided adding a second RDS read replica ($260/month) and delayed upgrading the primary instance to db.r6g.2xlarge ($1,040/month). The infrastructure savings came from eliminating redundant database queries, not from tweaking LLM prompt tokens. You do not need to create a separate domain for caching. You can point your existing API domain (api.yourproduct.com) or specific path prefixes to ApexCache: Add your domain in the ApexCache dashboard. Add the CNAME record (cname.getapexcache.com) and the verification TXT record in your DNS provider. Configure a path rule for /v1/models* with a TTL of 120 seconds. Verify the setup using curl: curl -sI "https://api.yourproduct.com/v1/models" | grep -i x-apexcache On the first request, the response header reads: X-ApexCache-Status: MISS On the second request, the response returns directly from memory: X-ApexCache-Status: HIT Age: 4 Keep these three boundaries in mind when introducing edge caching to an AI API: Check Authorization headers: If your /v1/models endpoint returns custom permission flags depending on user API keys, do not cache it globally. Only cache public, unauthenticated routes, or configure your policy to include the tenant identifier in the cache key. Strip internal debug headers: Ensure your origin server does not leak database connection pool diagnostics or internal trace identifiers in cached responses. Keep streaming paths isolated: Verify that your route rules explicitly target your read endpoints (/v1/models, /v1/pricing) rather than broad wildcards (/*), so that streaming completions (/v1/chat/completions) never encounter cache inspection overhead. Edge caching does not reduce your OpenAI or Anthropic invoice. It fixes the infrastructure tax around your product so your database stays online when traffic spikes. If your database CPU spikes while your GPU servers remain underutilized: Open ApexCache and inspect the GET-only reverse proxy architecture. Start free by connecting a staging hostname with one policy rule on /v1/models*. Run a baseline load test using k6 or hey and monitor your database CPU before and after caching. Docs: getapexcache.com/docs · Contact: getapexcache.com/contact I work on ApexCache. Numbers in this article are based on production benchmarks. Run your own load test before sizing production infrastructure.