Broken Retries: Why 'Just Try Again' Kept 5 Automation Lanes Dead for 63 Minutes

Five automation lanes went down on the same morning. Every log said the same thing: "Failed, so I ran it again." Then again. Then again. Sixty-three minutes after the first failure, all I had was the exact same failure — three more times over. When I first started writing scripts, I thought retries looked like this: for attempt in range(3): try: result = do_something() break except Exception as e: time.sleep(2) And that's correct — as far as it goes. A network blip, a server that got temporarily slow: that kind of failure really does get fixed by this. The problem is that it applies the assumption "the next attempt will succeed" uniformly to every failure. As the number of running jobs grows, failures that break this assumption will inevitably show up. In the SNS automation environment I operate, retry fixes came in from five separate lanes at once over two days, August 8–9, 2026. The symptoms all looked alike. The causes were completely different. note-autolike produced zero rewrites for three days. Last success: 2026-08-06T09:03:20Z. There was a 58-character limit validation on article titles, but the generation prompt said nothing at all about a character limit, so the model returned a 63-character title every single time. Three attempts, three 63-character titles. A fourth was never tried — it went straight into the give-up path. ai-portraits spun its wheels for 63 minutes on image generation. When a request got rejected by moderation, I had implemented a three-stage fallback that progressively softened the prompt and retried. Up to 21 minutes per concept; up to 63 minutes for three concepts. In practice, when the target content itself is what trips moderation, lowering the tier doesn't move the wall. Trying all three stages produced the same result as stage one. browser-slot.sh sees 18–34 launch attempts per hour against 3 global slots. A single run takes 14–62 minutes, so the slots are always full. The pre-fix implementation printed SKIP: global limit reached and exited 0 the instant acquire failed. The numbers left in the log: 9 runs in the 9 o'clock hour, 8 in the 11 o'clock hour, and 12 in the 12 o'clock hour never even started. One account hit skip on both of its runs that day, ending with 0 likes and 0 follows. All three had "just try again" implemented. None of them were fixed by it. Because the kinds of failure were different. A 63-character title: throw the same prompt at the same model and you get the same 63 characters back. That's a deterministic failure. A moderation block, when the content is the cause, doesn't change no matter how many stages you try unless the conditions change. A slot shortage can only be solved by waiting for someone to free a slot. In a retry implementation with no such classification, all three kinds of failure get sucked into the same "loop 3 times, then exit" handling. The result: fixable failures and unfixable failures are counted identically and come out the same exit. There's a failure mode in the opposite direction too — cases logged as "failed" that were actually transient. My watch-arb script recorded only extract_failed when it couldn't fetch a product page. Look at the dashboard the next morning and it reads as "there are 16 shops we can't fetch." In reality it was just HTTP 429 coming back, and a prober that hit the same shops slowly, one at a time, fetched them correctly. Lining up the similar incidents I confirmed over the same period: the same 429 was recorded under seven different names in the logs. Lane Name in the log Actual root cause watch-arb product page extract_failed HTTP 429 watch-arb bootstrap FATAL undici's 10-second ConnectTimeout social-autolike ig-3 coverage 0/11 → all sources failed → exit 1 429 from IG private API social-autolike ig-sug following map fetch failed → run halted The same 429 outreach-multi enrich all records error → all dropped at qualify → 0 messages total web_profile_info returning 429 constantly outreach-multi igJson unparsable json Response was 512,558 chars but sliced at 400,000 If it's recorded as extract_failed, you read it as "this shop's structure is unusual." In fact the problem was "I hit it too fast," and hitting it slowly the next day works. Because the name was wrong, I kept fixing in the wrong direction. Cleaning this up, correct retry design converges on answering three questions. (1) Will waiting fix it? (2) When do you cut it off? (3) What do you change before the next attempt? "Failed, so try again" is an implementation that omits all three. On the morning those five lanes were down together, none of the three had an answer written down. An automation environment doesn't keep running because I write code every day. It works because the environment self-corrects and keeps running while I'm asleep. From that premise, the requirements for retry design change. The requirement isn't "the next attempt will succeed after a failure" — it's "decide the next action per kind of failure." Between an implementation that has this design and one that doesn't, whether I have to be personally involved in incident recovery changes fundamentally. Here are the three questions in one diagram. A failure occurred │ ▼ ┌─────────────────────────────────────┐ │ (1) Will waiting fix it? │ │ │ │ Slot shortage → wait for one to free│ │ Network drop → back off and retry │ │ 429 → wait, preferring Retry-After │ │ Expired login → waiting won't fix it│ │ Validation error → waiting won't fix│ │ Moderation → depends on stage (below)│ └─────────────────────────────────────┘ │ waiting won't fix it ▼ ┌─────────────────────────────────────┐ │ (2) When do you cut it off? │ │ │ │ Attempt cap or >3h since intake → stop│ │ Still attempts left → go to (3) │ └─────────────────────────────────────┘ │ still attempts left ▼ ┌─────────────────────────────────────┐ │ (3) What do you change? │ │ │ │ Same input → same result if deterministic│ │ Attach the error → recoverable failures recover│ │ Tier downgrade (vertical) → soften and retry│ │ Swap the task (horizontal) → change the task itself│ └─────────────────────────────────────┘ The three sections aren't processed in order — the combination that applies changes with the kind of failure. For a network drop, (1) alone is enough. A deterministic validation error skips (1) and enters at (3). A moderation block involves all of it: skipping (1), the threshold in (2), and the way you change things in (3). Here's the core I added in the browser-slot.sh fix. BROWSER_SLOT_WAIT_SEC="${BROWSER_SLOT_WAIT_SEC:-600}" wait_for_slot() { local waited=0 while ! try_acquire_slot; do local jitter=$(( RANDOM % 31 + 15 )) # 15〜45秒ランダム sleep "$jitter" waited=$(( waited + jitter )) if [[ $waited -ge $BROWSER_SLOT_WAIT_SEC ]]; then log "RESULT waited=${waited}s SKIP: slot timeout" return 1 fi done return 0 } Make the retry interval a random 15–45 seconds. With a fixed interval, every job waiting for a slot piles in at the same instant (thundering herd). Adding random jitter offsets jobs from each other so they spread out naturally. I made the change only after confirming by measurement that a fixed interval serialized all the jobs and made congestion worse. Don't hold the guard for the whole wait. Hold the lock only for the instant of the acquire check, and always release it afterward. Get this wrong and every waiting job holds a lock against every other, and they all stall. Leave waited=600s on the RESULT line in the log. "Gave up after waiting 600 seconds" and "discarded immediately" are different events, and if you can't distinguish them you can't evaluate anything after the fact. BROWSER_SLOT_WAIT_SEC=0 matches the pre-fix immediate-skip behavior exactly. More than 30 launchd jobs share this script, so I made it possible to switch over gradually while preserving backward compatibility. On the other hand, there are cases that look transient and are in fact fixed by an immediate retry. I had a job that failed three days running with Cloudflare's Authentication error [code: 10000], which looked like "OAuth expired → a human has to intervene." But when I actually ran five probes in both a shell and a minimal launchd environment, all of them exited 0. Re-reading the logs from the failure days, 0.5 seconds after an access token reissue, a different endpoint returned 200 with that same token. It was a transient 401 right after token reissue, and the real root cause was that there was not a single retry. Failure rate: 3 days out of 9 (33%). Snap-judging "it's an auth error, so it's a human task" piles machine-fixable things onto a human queue. Take ai-portraits' tier ladder. Moderation block errors have a moderation_stage field. Ones that stopped at output have a track record of getting through with a tier downgrade. Ones that started at input failed all 9 attempts over the same period. for tier in [1, 2, 3]: result = generate(concept, tier=tier) if result.get("moderation_stage") == "input": # input段 → Tierを変えても壁は動かない、残りをbreak log(f"input-stage block, skip remaining tiers for this concept") break if result.get("moderation_blocked"): # output段 → Tier降格で通る可能性がある continue if result.get("success"): return result Decide from the first-stage error body whether climbing down the ladder is worth anything. Just breaking on moderation_stage == "input" turns 63 minutes of spinning into 0. In the same spirit, gsheets_retry.py is designed to raise immediately on 401/403/404. RETRYABLE_HTTP = {429, 500, 502, 503, 504} def sheets_call_with_retry(fn, max_attempts=3): delay = 2 for attempt in range(max_attempts): try: return fn() except HttpError as e: if e.resp.status not in RETRYABLE_HTTP: raise # 401/403/404 は即 raise — リトライで隠さない if attempt == max_attempts - 1: raise time.sleep(delay) delay *= 3 # 2秒 → 6秒（合計最大8秒） except (socket.timeout, ConnectionError, TimeoutError): if attempt == max_attempts - 1: raise time.sleep(delay) delay *= 3 Decide "never exceed 8 seconds total" first. If you set exponential backoff by attempt count, it quietly eats the caller's slot time. Fixing an upper bound in seconds first and back-calculating the attempt count is the safe order. If you implement "give up" using attempt count alone, you can't handle failures on the time axis. In kotonoha's job queue I had an incident where a job sat pending for 39 hours. Even when it broke off at the cap of 6 attempts, the requeue succeeded, so a failure was never raised, monitoring never fired, and the user's screen stayed on "generating." The post-fix predicate looks like this. function shouldGiveUpForQuota({ quotaHits, createdAtMs, resumeAtMs }) { const THREE_HOURS_MS = 3 * 60 * 60 * 1000; return ( quotaHits >= 2 || // 2回目の上限 resumeAtMs > createdAtMs + THREE_HOURS_MS // 受付から3時間超 ); } Place the cutoff on both "attempt count" and "absolute time since intake." Putting the actual values from the incident day (intake 2026-08-19T00:36:57Z, resume 06:37:00Z) into the tests as a boundary case gives you a machine guarantee that the same incident can never get through again. Here's the note-autolike case. The reason rewrites were zero for three days was an implementation that just threw the identical prompt three times. Make the error machine-readable and attach correction instructions to attempts from the second onward. // エラーに code / actual / limit を載せる throw new ValidationError(`タイトルは1〜58字にしてください（実際: ${actual}字）`, { code: "title-too-long", actual, limit: 58, }); // 2回目以降は修正指示を付けて投げる function rewriteRetryInstruction(error) { if (error.code === "title-too-long") { return [ `前回の出力は次の理由で不採用でした：`, `タイトルが${error.actual}字あり、上限の${error.limit}字を超えています。`, `同じ内容・同じ構成のまま、その点だけを満たす形へ直して、生JSONのみを再出力してください。`, `タイトルは絵文字込みで${error.limit}字以内（前回は${error.actual}字）に必ず収めてください。`, ].join("\n"); } } async function askClaudeWithRetry(prompt, retries = 3) { let currentPrompt = prompt; for (let i = 0; i < retries; i++) { try { const result = await callClaude(currentPrompt); validateRewrite(result); return result; } catch (e) { if (e instanceof ValidationError) { // 決定論的な失敗 — スリープなし、修正指示を付けて即再試行 currentPrompt = prompt + "\n\n" + rewriteRetryInstruction(e); } else { // プロセス系の失敗 — 30秒待つ await sleep(30_000); } } } } Retry deterministic failures immediately, with no sleep. Waiting 2 seconds on a validation error just throws 2 seconds away. If you mix "wait" and "fix and resubmit" into the same retry, you pay wait time even on fixable failures. Branch on the kind of failure and choose where the sleep goes. The correction instructions you return to the LLM should be "human-language instructions plus the actual strings," not "machine identifiers." Passing "title-too-long" through as-is loses three times in a row. Passing "the title is 63 characters, exceeding the 58-character limit; keep it within 58 characters" recovers. This is a fact I had already demonstrated in a different script. I'll stop the full picture of the three sections here. Next I'll break down how each section breaks and how to fix it, case by case, drawn from the five-lane simultaneous outage. The three sections designed above are, at the moment you write them, nothing but "code that should work." Intermittent failures have no reproduction method, so by default there's no way to confirm whether the retry actually traverses that path. My first four implementations ended there. I looked at git diff, felt like I'd confirmed it, and called it done. What actually worked was bundling fault injection into the same commit as the implementation itself. // watch-arb/collector/lib/api.mjs let _failOnceRemaining = parseInt(process.env.WA_API_FAIL_ONCE ?? '0', 10); async function req(path, opts = {}) { if (_failOnceRemaining > 0) { _failOnceRemaining--; throw new TypeError('fetch failed'); // 合成の障害、本番では絶対に発火しない } // ... 本来のfetch処理 } The counter is module-scoped and shared across all req calls. Only for the first N calls after process start, it throws a synthetic TypeError before calling fetch. The completion criteria are a set of two, and I always confirm both. ① Launch with WA_API_FAIL_ONCE=2 and see api retry attempt=1/3 and 2/3, then a full run to exit 0 with no FATAL ② With the env var unset, not a single retry log line appears Having ② means you can demonstrate "the fault injection hasn't leaked into production" with the same command. Here's the log I actually got. [INFO] api retry attempt=1/3 path=/api/collector/shops reason=fetch failed [INFO] api retry attempt=2/3 path=/api/collector/shops reason=fetch failed [INFO] 32件を ingest して exit 0 I rolled this pattern out from watch-arb to zaiko-radar, lily-line-funnel, and social-autolike — four in total. The rule changed to: "I wrote the retry" doesn't count as done unless it's "I watched it run with fault injection." When you write a retry, the first thing to decide isn't "how many attempts" but "what gets retried." // 判定を1関数に集約する function isTransientStatus(status) { return status === 429 || (status >= 500 && status <= 504); } async function req(path, opts = {}) { for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) { try { const res = await fetchWithTimeout(path, opts); if (!res.ok) { if (!isTransientStatus(res.status)) throw new HttpError(res.status); if (attempt === MAX_ATTEMPTS) throw new HttpError(res.status); await sleep(backoff(attempt)); continue; } return await res.json(); } catch (e) { if (!isNetworkError(e)) throw e; // ネットワーク例外以外は即throw if (attempt === MAX_ATTEMPTS) throw e; await sleep(backoff(attempt)); } } } Throwing immediately on 401/403/404 isn't a shortcut — it's design that prevents a different kind of incident. Wrap a permission error in a retry and you throw the same request three times and get the same failure three times. The log records only "failed 3 times," and the real cause — "no permission" — gets buried. This is the inverse of the "429 recorded under seven names" I documented in transient-failure-recorded-as-permanent.md. Same structure as a 429 masquerading as extract_failed: 401/403 masquerades as "sometimes slow." The reason to pull isTransientStatus out into a single function is that when the judgment is scattered across multiple call sites, each site grows a subtly different definition. The constant set RETRYABLE_HTTP = {429, 500, 502, 503, 504} in gsheets_retry.py comes from the same idea: it confines the knowledge of "which statuses are retryable" to one place. In the commit right after I implemented withRetry in kotonoha, three side-effect fixes went in. I thought I had "added a feature"; what I had actually done was "change how exceptions propagate." Before the change, purge() continued on to the subsequent "clean local work directories older than 30 days" step even when a 5xx came back, because the implementation moved to the next line even when res.ok was false. Once withRetry was in and it threw the instant a 5xx appeared, the cleanup code became unreachable. The real damage was a quiet degradation of the kind you only notice if you manually clean periodically: "work directories accumulate 30 days' worth." async function purge(jobId) { try { await uploadResult(jobId); } catch (e) { log(`purge upload failed: ${e.message}`); // ここで throw しない — 置き場の失敗は記録して続行する } await cleanWorkDir(); // 例外の有無にかかわらず必ず走らせる } The other one was setContentSafely, where the gap between the first and second retry was 0 seconds. Retrying immediately under the same conditions right after a failure caused by resource pressure gives the same result as long as the pressure remains. Earlier I wrote "retry deterministic failures immediately with no sleep" — this is the opposite side of that. For failures caused by resource pressure, a retry is meaningless unless you wait. Attempt counts are set asymmetrically according to "what happens when it fails." report() (result reporting) leaves the user's screen stuck on "running" forever if it fails, so attempts=5; claim() (reservation acquisition) gets attempts=3. A uniform attempt count ignores the asymmetry of cost and damage. Timeouts are also split by the weight of the operation: putObject and getStream get 120 seconds, getText and listKeys get 30 seconds. A fetch with no timeout configured creates the worst un-retryable state of all: hanging forever. There's a constraint I wasn't conscious of until I implemented this. To make something retryable, the input has to be re-executable. // NG: ストリームは一度消費したら再生できない async function retryWithStream(stream) { for (let i = 0; i < 3; i++) { await fetch(url, { body: stream }); // 2回目はストリームが空 } } // OK: Buffer か string に限定する async function retryWithBuffer(buffer) { for (let i = 0; i < 3; i++) { await fetch(url, { body: buffer }); // 何度でも同じデータを送れる } } I restricted the body passed to putObject to Buffer or string. If you don't leave the type constraint in a comment, a caller later passes a stream and you get a bug where "the retry is working but it fails every time." Pass a stream into a retry and every attempt after the first sends an empty body. No error surfaces; it just fails every time. In outreach-multi's DM inbox parsing, an unparsable json error appeared every day. I suspected a broken selector or a changed JSON structure and kept fixing for three days. The reality: the inbox-fetching code had a line reading text.slice(0, 400_000). It was an upper bound written to protect memory when a huge response came in. When I measured the actual size of the IG DM inbox, it was 512,558 characters. Since the limit is 400,000, it gets cut off every single time. Truncated JSON fails JSON.parse every single time. Success rate: 0%. The reason I stared at the same failure for three days is that I had forgotten about the text.slice line and read it as "parsing is failing → the structure broke." Nor had the symptom started "today." It quietly started failing on the day the inbox size crossed 400,000 characters, and one day I noticed the running total was 0. The fix was to complete JSON.parse inside page.evaluate and return only the first 300 characters for diagnostics when it fails. The design rule I took from this: "if you place a limit, log the fact that the limit was hit." text.slice silently truncates and tells no one. Even when a protective constant has turned into a permanent failure the moment its value was exceeded, nothing is left in the log. The unfollow processing in unfollow-core.js had a mechanism where three failures put an account into a 7-day hold (quarantine). The problem is "what happens when it comes out of quarantine." A revived account goes back into the candidate list. Fail three more times, and it goes into another 7-day hold. This spins forever. There were 2,605 candidates, and I puzzled over the numbers scanned: 8 / unfollowed: 2 for two weeks. Almost none of the candidates were being processed. When I investigated "why isn't this progressing," the answer was "the same accounts keep entering and leaving quarantine, and nothing moves forward." The log said circuit-break. It's contradictory for circuit-break to be functioning while the execution count doesn't rise. I could have noticed much sooner, but I was reading it as "circuit-break is working = it's protected," so I never questioned it. The fix was to physically delete from the ledger any account that reached a cumulative failure count (UNFOLLOW_GIVEUP_FAILS = 6, twice the quarantine failure threshold). A "hold" only accumulates; it never drains. Unless you place a terminator that discards, the ledger keeps filling up with accounts that can't be processed. That said, before discarding, you first have to confirm "is my own processing working correctly?" There's an inverse incident where the session had merely expired and you discard all the healthy accounts. The principle I wrote on the kotonoha page — "check your own health before discarding" — applies here too. A kotonoha job sat stuck on "generating" for 39 hours. Monitoring never fired once. The job was never recorded as a failure. The cause: "a retry that doesn't hold the attempt count in state can't have a cutoff placed on it." When a job hits a quota limit, requeue puts it back to pending. The returned job carried no counter for "which attempt is this." A 7th-attempt job and a 1st-attempt job are indistinguishable from the implementation's point of view. Since requeue succeeds, no failure is raised. It never hits a monitoring threshold. The user's screen keeps displaying "generating" indefinitely. The fix came in two stages. First, "hold the attempt count in state." Then, "place the cutoff on both attempt count and absolute time since intake." When I put the shouldGiveUpForQuota function shown earlier through tests with boundary cases, I included the actual values from the incident day (intake 2026-08-19T00:36:57Z, scheduled resume 06:37:00Z) as one case. That gives a machine guarantee that the same incident can never get through again. With a cutoff on attempt count alone, you let through "only the 1st attempt, but resume is 12 hours away." With time alone, you let through "resume is soon, but the limit hits back to back." This incident got caught on the first attempt, on the time-condition side. It's a case where the value of doubling up was measurable in real data. "The worker waits" and "give up on that job" are separate decisions. A quota is a failure that waiting fixes, so the worker waits. But that isn't grounds for making a user wait more than three hours. com.lily.line-pdca stopped three days running with Cloudflare's Authentication error [code: 10000]. Since it was an auth error, I judged "OAuth expired, a manual wrangler login is needed." I also suspected a missing environment variable. Classifying it as "a case requiring human GUI operation" before verifying was the mistake. When I actually ran wrangler whoami, kv key list, and deploy --dry-run in both a shell and a minimal launchd environment, all exited 0. Even the output byte counts were identical (1,516 bytes). Re-reading the logs from the failure days, KV keys returned 401 0.5 seconds after an access token was reissued. With that same token, /user 0.5 seconds later returned 200. It was a transient 401 right after token reissue, and the real root cause was that there was not a single retry. The failure rate was 3 days out of 9 (about 33%). It doesn't happen every day, but it does happen once every three days. That frequency was itself part of why it "looked like a human task." The criterion that changed my judgment was actually hitting the endpoint to confirm "would the same procedure pass if I ran it again right now?" Don't classify something as "unfixable, period" based only on the kind of error (401, Authentication error). Same pattern as the "429 recorded under seven names" documented in transient-failure-recorded-as-permanent.md — this time the name "auth error" made a transient failure look like a human task. Since this incident, the rule changed to: when an auth error appears, run one probe before deciding. The premise behind "don't pile machine-fixable things onto a human work queue" is "I hit it with a machine and checked." Snap-judging without checking is no different from declaring "it won't be fixed" without ever trying. Here are the structural patterns behind the individual cases above. Only the "name of the place I got stuck" differs; the root is the same pattern. Compressing a failure into "a single name" when recording it watch-arb's product pages were recorded as extract_failed, the bootstrap run as FATAL, and the DM inbox as unparsable json. All three had HTTP 429 as the actual root cause. Organizing this phenomenon, recorded in transient-failure-recorded-as-permanent.md, the same 429 was left under 7 different names within the same period. When the HTTP status disappears at the naming step, you can no longer judge whether it's "waiting will fix it" or "the structure needs fixing." The fix is one line. Before recording, classify: 429/503 → rate_limited / 401/403 → auth_error / everything else → extract_failed. This three-way branch is the precondition sitting in front of the wait/give-up/feed-back branch. Throwing away the error body Because ai-portraits' generate.py discarded codex exec's stdout/stderr, the reason no image was produced was recorded only as missing. If you can see neither moderation_blocked nor safety_violations: ["sexual"], you can't judge whether a tier downgrade will recover it or whether you should cut it off immediately at the input stage. In an implementation that doesn't retain errors, the three sections of "wait / give up / feed back" can't exist in the first place, so error retention is a design precondition, not a feature addition. Trying every tier without checking moderation_stage Concepts that stopped with moderation_stage: input were tried through all stages from T1 to T3 and lost all 9 attempts. Ones that stopped with moderation_stage: output have a track record of passing with a tier downgrade (partial 01_feet_up_window at T2, 03_legs_on_bed_topdown at T3). A single word in T1's moderation_stage field determines whether climbing down the ladder is worth anything, yet trying every stage without reading the field wastes up to 21 minutes per concept — up to 63 minutes for three. Judge "is it worth climbing down based on the first-stage error body" before "how many safe-side fallback stages are there." Wiring a fallback without measuring the destination's capacity Before adding a fallback in social-autolike's follow-source-followers, I first measured that the UI modal's following side grows to 408 entries in 63 seconds while the followers side stops at 12. That's why I adopted only the side with capacity as the fallback. Wire it without measuring and all the traffic flows to a destination with no capacity, producing "I added a fallback but the success rate didn't change." outreach-multi's pk resolution likewise builds in the known wall of IG cutting off around 70 entries, limiting to 30 per run. Counting a permanent failure against the "one more try" budget In outreach-multi's DM sending, when no メッセージ button (DM-disabled setting) came back, it rode the exhausted after 3 attempts path and consumed 3 attempts' worth of send budget per recipient. The other party's account settings don't change no matter how much we retry. Read the reason field in sent.jsonl, put recipients containing no メッセージ button / profile gone / このページは存在しません into a permanentlyFailed Set, and place the shouldSkip() check ahead of exhausted. This shares a root with the "inverse incident of recording a transient failure as permanent" mentioned earlier: the design of failure classification is needed before the design of retry counts. Continuing to hit at the same rate after a 429 social-autolike's ig-3 fired 17 sources within a few seconds at 0.3–0.5 second intervals and produced zero results over 14 hours. If you don't change the interval after receiving a 429, you pile up the same requests during the limit and extend your own throttling. The fix recorded in retry-and-giveup-design.md includes a latch that cuts off that process's API calls after 3 consecutive 429s, plus a design that drops concurrency to 1 for any shop that has seen a 429 even once. Slowing down the rate itself comes as a set with waiting before retrying — not instead of it. Retries stacked in two layers watch-arb's ingestShop had its own 3-attempt loop, and req() had a 3-attempt loop as well. Worst case 3×3 = 9 attempts, with "retry attempt 3/3" appearing twice in the log, making it unreadable where the limit is and which attempt you're on. Even with fault injection in place, you can no longer tell "which layer is running." The fix was to delete ingestShop's own loop and consolidate onto the single layer in req(). The caller just throws immediately. Counting fallback successes toward the cutoff counter The latch that "cuts off API calls after 3 consecutive 429s" was also counting cases where the UI fallback succeeded. Even with an escape route, it stopped at 3 items. The cutoff counter should increment only "when every path failed." Cases that succeeded via the fallback shouldn't be logged against the main failure count. Judging "couldn't fetch (abnormal)" and "had no time to fetch (a normal cutoff)" by the same zero count When follow-source-followers uses up its time budget on source fetching alone and ends with zero candidates, it falls into the existing "all sources failed → halt to prevent mis-operation" path. "Couldn't fetch" and "just ran out of time" are separate events. Judge them by the same zero count and you get false alarms every day and lose the ability to tell real failures apart. Prepare a separate exit called time-budget to distinguish them. Hardcoding the give-up predicate inside the catch block When the cutoff conditions are written inside a catch, you can't put them into boundary-case tests. Pull them into a pure function and you can add one case with the actual values from the incident day, giving a machine guarantee that the same condition never gets through again. As long as the logic lives inside the catch, the test passes in a state where "the case just happened not to be exercised." Error messages carrying no machine-readable information Throw a ValidationError with only a string and you can't extract "what was wrong" from code at retry time. The reason throwing the same prompt from the second attempt onward gives the same result is that "feed back" isn't implemented; put on a structure like code: "title-too-long", actual: 63, limit: 58 and you can assemble correction instructions dynamically. When returning to the LLM, pass a human-language instruction and the measured values, not the identifier ("title-too-long"). Passing the identifier through as-is loses three times in a row. Here are the rules distilled from the five-lane simultaneous outage. There are a lot of them, but every one is something I added because I actually got stuck. ① Classify failures by HTTP status and exception type before recording them 429/503 → rate_limited / 401/403 → auth_error / everything else → processing error. Just placing this three-way branch first changes the meaning of the "list of failing shops" shown on the dashboard. After adding this classification, watch-arb also split the wording of shops.last_error into "couldn't fetch due to rate limiting" and "can't extract the structure." ② When you write a retry, put fault injection in the same commit let _failOnceRemaining = parseInt(process.env.WA_API_FAIL_ONCE ?? '0', 10); // 起動後の最初のN回だけ合成のエラーを投げる The completion criteria are a set of two. With =2, retry attempt=1/3 and 2/3 appear and the run completes at exit 0; with it unset, not a single retry log line appears. Without ②, you can't confirm with the same command that "the fault injection hasn't leaked into production." ③ Place the cutoff on both "attempt count" and "absolute time since intake" Attempt count alone lets through "only the 1st attempt, but resume is 12 hours away." Absolute time alone lets through "resume is soon, but the limit hits back to back." The kotonoha incident got caught on the first attempt, on the time-condition side (intake 2026-08-19T00:36:57Z, resume 06:37:00Z, outside intake+3 hours). A case where the value of doubling up was measurable in real data. ④ Decide the total wait ceiling in seconds first, then back-calculate the attempt count For gsheets_retry.py, I fixed "never exceed 8 seconds total" in the request text first, then back-calculated 2s → 6s (3 attempts). If you fix exponential backoff by attempt count first, the caller's slot time gets quietly eroded. ⑤ Throw immediately on 401/403/404 — don't hide them in a retry if e.resp.status not in RETRYABLE_HTTP: raise # 権限エラーは即 raise Wrap a permission error in a retry and only "failed 3 times" remains, burying the real cause of "no permission." Same structure as the "429 masquerading as extract_failed" recorded in transient-failure-recorded-as-permanent.md — this time a 401 masquerades as "sometimes slow." ⑥ Use moderation_stage to judge whether climbing down the ladder is worth it before trying tiers if result.get("moderation_stage") == "input": log("input段の拒否。残りTierをbreak") break Adding one line turns 63 minutes of spinning into 0. Ones that stopped at output are worth running the ladder for. Ones that stopped at input get discarded concept and all, and you move on. ⑦ Consolidate retry layers into one place If the caller has its own loop and the library also retries, you get up to N×M attempts. When you find duplication, delete the outer loop and consolidate onto the single layer in req(). ⑧ Measure the destination's capacity before wiring it up Before adding a fallback, confirm by measurement "how many entries this destination returns" and "where it jams." Routing all traffic to a destination with no capacity won't change the success rate. ⑨ Record permanent failures in a persistent list once and exclude them Failures rooted in the other party's settings, like no メッセージ button / profile gone, should be excluded before they consume retry attempts. Place the shouldSkip() check ahead of exhausted. ⑩ Actually hit it to check "would it pass if I ran it again right now?" before classifying it as a human task I snap-judged Cloudflare's Authentication error as "OAuth expired" and left it in a human queue for three days. Running probes in both a shell and a minimal launchd environment, all exited 0. Don't classify something as "unfixable, period" from the name of the error alone. Check, then judge. ⑪ Log the fact that a limit constant was hit text.slice(0, 400_000) just silently truncated. Even when a protective constant becomes the trigger for a permanent failure, nothing is left in the log. Complete JSON.parse internally and return the first 300 characters for diagnostics only when it fails, and the fact that "it got cut off" survives in the record. ⑫ After introducing retries, check how the range of exception propagation changed Adding withRetry can make required downstream processing unreachable where the behavior changed from "swallow" to "throw." A retry isn't a feature addition; it's a change to how exceptions propagate. Where there's something downstream that "must happen even on failure," wrap it individually. ⑬ For failures caused by resource pressure, wait before retrying With setContentSafely's retry interval at zero seconds, it fails again under the same conditions right after failing under high load. Immediate retry with no sleep is correct for deterministic failures, but on the flip side, a retry is meaningless unless you wait for resources. Choose where the sleep goes based on the kind of failure. ⑭ Pull the give-up predicate into a pure function and put the incident day's actual values into the tests Extract it into shouldGiveUpForQuota({quotaHits, createdAtMs, resumeAtMs}) and add intake 2026-08-19T00:36:57Z / resume 06:37:00Z as one case, and the same incident can never get through again. A predicate hardcoded inside a catch block can't have its boundary cases tested. ⑮ Leave the fact that you waited, gave up, or cut off on a single log line waited=600s SKIP: slot timeout and SKIP: global limit reached (immediate discard) are different events. If both look like the same "SKIP," you can't distinguish afterward whether "slot contention increased" or "it was discarding immediately all along." Emit the wait time you invested as a number in the log. When you write "failed, so try again," that code has no answers to three questions. Will waiting fix it? When do you cut it off? What do you change before the next attempt? An implementation missing these three burns resources in front of a deterministic failure and stalls. What I noticed when I lined them up on the morning five lanes went down together: whether I have to be personally involved in incident recovery is almost entirely determined by whether this design is present. With it, the script decides its own next action per kind of failure. Without it, the same failure just gets recorded three times, and the next morning it's stuck in the same place. An automation environment runs every day not because I write code every day. It works because failures are classified correctly and the machine recovers on its own. From that premise, retry design isn't "a feature you add later" — it belongs in the first implementation. "I wrote the retry" counts as done when it's "I watched it run with fault injection." What's the failure in your own automation that's been recorded under the wrong name the longest? Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code. Portfolio · X · GitHub*