The Same Model Can Cost 14x More Depending on Who Serves It

You picked a model. You are not done. On a routing platform, one model name maps to many providers, and they do not agree on price. Of the 405 paid models on the platform, 182 are served by two or more paying providers. Across median spread 1.87x, widest 14.47x, and 46% vary by 2x or more. Same weights. Same model card. Same API call. Up to fourteen times the invoice. Three variables move independently underneath a single model name: Precision. A provider serving fp4 is not serving the same thing as one serving bf16, even though Uptime. 89.3% and 99.9% are both "available" on a status page. One of them wakes you up. Where the data goes. 71 of those 182 models — 39% — are served from providers in more than one If you have a data residency obligation, the model id tells you nothing and the provider None of this is hidden. It is just spread across separate pages, so nobody assembles it before The latency field in the public model data is null. That is the honest answer from an aggregator — So we called 329 models on a paid API with one fixed prompt and recorded time-to-first-token and t0 = time.perf_counter() ttft = None with client.stream("POST", URL, json={"model": mid, "stream": True, "messages": [{"role": "user", "content": PROMPT}], "max_tokens": MAX_TOKENS, # Reasoning models will spend the whole budget thinking # and return empty content. Turn it off for a speed test. "reasoning": {"enabled": False}}) as r: for line in r.iter_lines(): if not line.startswith("data: "): continue if ttft is None: ttft = (time.perf_counter() - t0) * 1000 # first token, milliseconds Two properties of this data will quietly break a harness: Reasoning budgets consume the whole allowance. With reasoning enabled, a model can return empty content after spending every token you granted. You measure nothing and still pay for the call. Undisclosed prices arrive as -1. If you sum projected costs to skip expensive models, those down and every model passes the "too expensive" check. Clamp at zero We show our measurement next to OpenRouter's own p50–p99 percentiles, which aggregate thousands of We also grade Korean-language quality on seven axes — register, knowledge of Korean institutions, Two axes collapse across almost the whole field: Axis A rate Honorifics 8.5% Knowledge of Korean institutions 9.4% Terminology 31.2% Format compliance 47.6% Register 53.3% This is a failure mode developers should recognise, because it is the same shape as a silent And it does not correlate with the things you would use as a proxy. A 2023 model, gpt-3.5-turbo-16k, scores a perfect 3.00, above most 2026 flagships. Release date, parameter count All of it is open, no key required: # per-provider price, precision, uptime and HQ for one model curl .../api/models/deepseek/deepseek-v3.2/endpoints # latency and throughput percentiles, per provider curl .../api/models/deepseek/deepseek-v3.2/perf # our measurements and grades curl .../api/speed curl .../api/korean # includes per-axis detail, not just the band Base URL is the Space's app URL: https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard 425 models, 329 measured for latency, 330 graded for Korean, next to price, provider, precision and Choosing a model is one decision. Choosing an endpoint is a second one, and it moves your bill by up If your procurement checklist stops at the model name, it stops one step early. Full writeup with methodology: https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard