The HTTP 429 That Turned Seven Minutes Into Zero Work

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. ARGUS is a data-catalog governance swarm. Its agents use structured model calls to draft descriptions, classify sensitive fields, and review repairs before anything can be written. During a 45-entity sweep, the system spent seven minutes producing zero proposals. The report called the result review failed. The endpoint was reachable. Health probes returned 200. The logs showed only: 429 Too Many Requests The retry loop was working exactly as written. That was the bug. The model provider enforced both a short burst limit and a daily token quota. Both failures arrived as HTTP 429 with the same useful headers. Their remedies are opposites: Condition Correct response Per-minute burst limit Wait, then retry Daily token quota exhausted Stop using that endpoint for this sweep The only reliable distinction was in the response body. The provider's daily-quota response explained: on tokens per day (TPD): Limit 100000, Used 99299, Requested 1139 ARGUS never logged that text. The OpenAI-compatible client raised an httpx status exception whose default message contained the status and URL, but not the response body. The information needed to fix the failure was present on the wire and discarded at the error boundary. The account was close to its cap, not necessarily at exactly zero remaining tokens. A tiny probe could still fit under the allowance and return 200. The real structured request included a schema, column context, lineage context, and reserved output. It needed more than the remaining budget and failed. That created a misleading picture: connectivity looked healthy authentication looked healthy small calls looked healthy every useful call failed A probe only proves that the probe can run. It does not prove that the real workload fits inside the remaining quota. ARGUS treated every 429 as temporary. Each model call could wait through six backoffs. A sweep could need roughly one hundred calls. For a rate limit that clears in seconds, this is patient behavior. For a daily quota that resets tomorrow, it turns one deterministic failure into hours of waiting. The retry policy had no concept of recoverability. It knew that a request was throttled, but not whether time inside the current operation could change the result. The HTTP boundary now reads and carries a bounded copy of the response body when it raises an error. Instead of this: 429 Too Many Requests the log can say which limit was hit, how much was used, and what the failed request needed. That single change identified the root cause on the next run. The broader lesson is simple: an error abstraction should hide irrelevant transport details, but it must not erase the only field that distinguishes two operational states. I added a BudgetExhausted exception rather than routing a daily quota through the ordinary retry path. When the response identifies a long-lived token or request quota: The call fails immediately. The endpoint is retired for the current sweep. A breaker prevents the remaining agents from repeating the same doomed call. The breaker resets at the start of the next sweep. The CLI labels the totals as partial. The reset matters. A process-wide flag that survives its cause would make tomorrow's healthy quota look permanently dead. The most important change was not in the HTTP client. It was in the Arbiter. ARGUS already had a deliberate no-model mode. If the operator starts a run without a key, mechanical grounding becomes the declared review gate. A proposal that passes that gate can be approved. Budget exhaustion halfway through a modeled run is different. Falling into the same branch would silently remove the review gate at the exact moment it stopped working. The new outcome is: not reviewed: model budget exhausted It is not rejected, because no reviewer made a judgment. It is not approved, because the expected review never happened. Unreviewed proposals are never written. This distinction protects both the data catalog and the truthfulness of the report. The provider reports remaining tokens and reset timing on successful responses. ARGUS now uses those headers to pace before hitting the wall instead of discovering the wall by repeatedly crashing into it. I also reduced reserved output from 4096 tokens to 900. The structured responses are short descriptions, tag lists, and verdicts. On free tiers, reserving 4096 tokens can consume rate-limit capacity even when the reply uses far less. When multiple endpoints are configured, the retry and rotation loops are now separate: retry when the same endpoint may answer next time rotate when the endpoint will not answer during this sweep do not rotate on unrelated errors such as malformed output or HTTP 500 trip the breaker only when every configured endpoint is unavailable That last negative test matters. Rotating on every error would burn the entire provider chain because of one bad prompt. The original reset parser stripped a trailing s and called float(). Real providers return values such as: 1h30m43.2s 205ms Both values failed the old parser and silently fell back to guessed delays. The replacement parser handles hours, minutes, seconds, milliseconds, decimals, and compound durations. Behavior Before After Daily quota Six backoffs per call One classified failure Provider explanation Discarded Preserved and logged Remaining calls Repeated the same failure Short-circuited or rotated Proposal outcome review failed not reviewed Writes after exhaustion Ambiguous fallback risk Never written Sweep totals Looked complete Explicitly partial Reset header parsing Simple float only Compound duration support Provider fallback Stopped on exhausted endpoint Rotates after bounded retries The real exhausted endpoint that previously took seven minutes now produced a clear partial-run report in seconds and committed nothing. The regression suite grew dedicated coverage for: daily quota versus temporary rate limit no retry after confirmed exhaustion breaker reset between sweeps no automatic approval after mid-run exhaustion unreviewed proposals never reaching a write duration parsing for 1h30m43.2s and 205ms rotation after persistent unnamed throttling no rotation on non-throttle failures stopping clearly when every endpoint is spent HTTP 429 says the request cannot run now. It does not say whether waiting inside the current job can help. Logging more bytes is not automatically useful. Logging the one bounded provider field that distinguishes retryable from terminal failure is. Starting without a reviewer is an explicit operating mode. Losing the reviewer halfway through is a failed assumption. They should not share a fallback branch merely because both lack a model response. The sweep produced real numbers for fewer entities than requested. Presenting those totals without qualification would overstate coverage even if every individual number were correct. The best part of this fix is that it did not merely reduce latency. It stopped an infrastructure outage from being misreported as an AI judgment and preserved the review boundary when the budget disappeared. AI assistance was used during code and draft iteration. Joshua Hernandez is the sole contest entrant and is responsible for the submitted implementation and evidence.