FAQ: Five Things a 200 OK From a Free Model Endpoint Does Not Tell You

Last month a green checkmark lied to me. An API call returned 200 OK. The model that answered was not the model I asked for. Nothing in my logs said so. I now keep a provenance log for every endpoint I touch. It started as paranoia. It is now the first file I open when something looks wrong. Below are five claims I used to repeat. Each one cost me real debugging time. The status code describes the HTTP layer. It does not describe the model layer. Gateways accept a model string and route wherever they choose. Ask yourself: did the response body echo my model id back? Even when it does, that field is a claim made by the server. It is not proof of anything. So log both sides. Store requested_model and served_model in one record. When they diverge, you are looking at an alias, a fallback, or a silent routing change. They are two different budgets, and they fail in different ways. One is an entitlement: how much inference you may spend. The other is compute: where your process actually runs. An entitlement usually fails loudly. You get 429, 402, or a stream that stops mid-sentence. A compute allocation fails quietly. The box sleeps, the disk fills, the port closes, the clock drifts. Do not merge them into one mental model. Do not merge them into one retry policy either. One needs backoff. The other needs a health check. Environment drift is not a vibe. It is a diff you failed to capture. My laptop has a different Python minor version and a different PATH order than any remote machine. Record the boring facts with the request: interpreter version, working directory, and the env var names you depend on. Never log the values. Names are evidence. Secrets are liabilities. python -c "import sys, platform; print(sys.version, platform.platform())" node -v && uname -a && pwd Run those on both sides and diff them. Two minutes, no guessing. No. Rate limits are a control-flow problem. A 429 tells you your program's timing is wrong, not your wallet. A retry loop without jitter creates a thundering herd of one. Here is the shape I use. It is a template, not a certified client. import random, time def with_backoff(call, attempts=4): rec = None for i in range(attempts): rec = call() # call() logs every attempt itself if rec["status"] != 429: return rec wait = min(2 ** i, 30) time.sleep(wait + random.uniform(0, 0.5)) return rec Notice that call() still writes a log line on every attempt. A retry you cannot count is a retry you cannot explain. This is exactly backwards. When you pay, the invoice is your record. When you do not pay, the invoice disappears. Your own append-only file becomes the only artifact that says what ran, when, and against which model. That file is cheaper than an incident review. This script targets any OpenAI-compatible chat endpoint. Set three env vars and run it. Treat header names as provider-specific; I have not tested it against every gateway. """probe.py - append-only provenance record for one chat call.""" import json, os, time, requests BASE = os.environ["MC_BASE_URL"].rstrip("/") KEY = os.environ["MC_API_KEY"] MODEL = os.environ["MC_MODEL"] LOG = os.environ.get("PROVENANCE_LOG", "provenance.jsonl") def probe(prompt="Reply with the single word: ok") -> dict: started = time.perf_counter() resp = requests.post( f"{BASE}/chat/completions", headers={"Authorization": f"Bearer {KEY}"}, json={"model": MODEL, "max_tokens": 8, "messages": [{"role": "user", "content": prompt}]}, timeout=60, ) served = None try: served = resp.json().get("model") except ValueError: pass # HTML error page; keep the raw status instead rec = { "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "requested_model": MODEL, "served_model": served, "status": resp.status_code, "latency_ms": round((time.perf_counter() - started) * 1000), "request_id": resp.headers.get("x-request-id"), "retry_after": resp.headers.get("retry-after"), } with open(LOG, "a", encoding="utf-8") as fh: fh.write(json.dumps(rec) + "\n") return rec if __name__ == "__main__": print(json.dumps(probe(), indent=2)) No Python? The same evidence fits in one shell pipeline. curl -sS -D headers.txt -o body.json \ -H "Authorization: Bearer $MC_API_KEY" \ -H "Content-Type: application/json" \ -d "{\"model\":\"$MC_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}],\"max_tokens\":4}" \ "$MC_BASE_URL/chat/completions" grep -iE '^(x-request-id|retry-after|x-ratelimit)' headers.txt jq -r '.model' body.json Field What it proves What it does not prove status The HTTP layer answered Which model generated tokens served_model What the server claims it used That the claim is true request_id A handle for a support ticket Anything about reproducibility latency_ms One sample, one path Your p50, your p99, your user's day retry_after The server asked you to wait How many attempts the SDK made One row is an anecdote. Sixty rows sorted by time are a baseline. Your situation Entitlement matters most Compute matters most Why Prompt iteration and prompt diffing Yes No You need many cheap calls CI smoke test with no secrets Yes No Short-lived, stateless Long-running worker with a queue No Yes Uptime beats burst budget Anything holding production credentials Neither Neither Free tiers are not prod Audit or compliance evidence Partly Partly Your own log is the artifact Anyone needing an uptime guarantee. Neither kind of free provides one. Teams handling regulated data on shared infrastructure. Projects where a silent model swap changes legal or safety outcomes. A provenance log records what a server claimed, not what executed. It cannot detect a quantized variant behind the same alias. It cannot prove the weights. It also adds I/O to every call, so keep it out of hot loops. I will not quote quotas, hardware, or model names for any provider here. I cannot verify them today, and yesterday's number is tomorrow's misinformation. Measure your own endpoint with the probe above. Run the probe once. Confirm the script wrote one JSONL line. Compare requested_model and served_model. Note any mismatch. Force a 429 with a burst. Confirm your backoff honored retry-after. Diff interpreter and OS facts between laptop and remote. Re-run the probe tomorrow and diff the two files with jq -S. If you want somewhere disposable to run steps one through five, I used MonkeyCode's free model access and free server option for exactly this kind of probe. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A green checkmark means a socket accepted bytes. That is all it ever meant. The rest is your log's job.