I Built a Demo to Finally Understand Idempotency - Here's What Happened

Last Friday I posted something on LinkedIn about idempotency. I'd hit the concept while reading about API design and it wouldn't let me go. The scenario is so specific and so consequential that once you understand it, you start seeing the problem everywhere. Here's the scenario: a user hits "Pay". The connection drops. They retry. Without protection, they just got charged three times. I could explain it in a post. But I learn by building - so I built a demo to prove it works, live, in an interactive test console. Here's everything from the rabbit hole. An operation is idempotent if doing it once produces the same result as doing it many times. Pressing a lift button is idempotent. It doesn't matter how many times you press 7 - the lift still goes to floor 7 once. The tenth press doesn't summon ten lifts. Sending M-Pesa money is not naturally idempotent. Send KES 500 to Brian three times, Brian gets KES 1,500. Each request is treated as a brand new instruction. In APIs, the HTTP methods have idempotency rules baked in by design: Method Idempotent? Why GET ✅ Yes Fetching data doesn't change anything PUT ✅ Yes Setting a value to X is the same whether done once or ten times DELETE ✅ Yes Deleting something that's already deleted is still deleted POST ❌ No Creating things — each call creates a new thing POST is the problem child. And POST is exactly what payment endpoints use. A user clicks "Pay". Your server receives the request, starts processing the charge, then the network hiccups. The client gets a timeout - not a success response, not an error. Just silence. So it retries. Standard behaviour for any well-built client. Your server had already processed the first charge. Now it processes the second. And the third. The user's card has been charged three times for one purchase. They'll call support. You'll spend two hours in a refund process. If it happens at scale, it's a chargeback nightmare. The POST /charge endpoint with no protection looks like this: @app.post("/charge") def charge(payload: ChargeRequest): charge_id = str(uuid.uuid4()) # new ID every single time charges_db[charge_id] = { "charge_id": charge_id, "amount": payload.amount, "currency": payload.currency, "status": "success", } return charges_db[charge_id] Run it three times with the same request body. You get three different charge_id values. Three separate charges in the database. The solution: give the client a unique key for each intended operation, and send that key with every retry. The server uses it to recognise: "I've seen this request before - here's the result from the first time." POST /charge/idempotent Headers: Idempotency-Key: pay_brian_1001_attempt_1 Body: { "amount": 500, "currency": "KES" } Retry it three times with the same Idempotency-Key. You get the same charge_id back every time. The charge only runs once. Every repeat just replays the stored result. The protected endpoint: @app.post("/charge/idempotent") def charge_idempotent(payload: ChargeRequest, idempotency_key: str = Header(...)): # Already seen this key — return the stored result if idempotency_key in idempotency_store: return idempotency_store[idempotency_key] # First time seeing this key — process and store charge_id = str(uuid.uuid4()) result = { "charge_id": charge_id, "amount": payload.amount, "currency": payload.currency, "status": "success", "idempotency_key": idempotency_key, } idempotency_store[idempotency_key] = result charges_db[charge_id] = result return result The logic is two lines at its core: If we've seen this key before -return what we stored. If we haven't - process it, store it, return it. Every subsequent call with the same key hits line 1 and returns immediately without touching the payment processor. I built this as a small FastAPI project so you can watch the difference happen in real time, not just read about it. Repo: github.com/Navashub/idempotency-demo FastAPI gives you an interactive test console at /docs - Swagger UI ,, out of the box. No Postman, no curl, no separate tool. Just open the browser. To see the problem: Clone the repo and run it: git clone https://github.com/Navashub/idempotency-demo cd idempotency-demo pip install -r requirements.txt uvicorn main:app --reload Open http://127.0.0.1:8000/docs Expand POST /charge, click Try it out, hit Execute three times with the same body. Watch the charge_id field in the response. It changes every time. Three different values. Three charges. To see the fix: Expand POST /charge/idempotent, add any string as the Idempotency-Key header — something like test-payment-001. Hit Execute three times with the same key. The charge_id stays identical on every response. One charge, three attempts, same result. That live comparison — charge_id changing vs staying the same - is a different kind of understanding than reading a description of it. 1. The idempotency key belongs to the client, not the server. The client generates it before sending the first request and keeps it for all retries of the same intended operation. The server never generates it. This is important - if the server generated it, retries would get fresh keys and the protection would be useless. 2. The key should be scoped to one operation, not one session. A key like user_123_payment_456 ties a specific payment attempt to a specific user. A generic key like my-key would incorrectly replay an old charge if someone reused it for a different payment. 3. In-memory storage is for demos only. My idempotency_store = {} disappears when the server restarts. A real implementation stores keys in Redis (with a TTL - typically 24 hours) or a database. The lookup needs to be fast and survive server restarts. 4. This pattern is everywhere once you know it. Stripe, M-Pesa, Flutterwave, Paystack - every payment API has an idempotency key parameter. GitHub's API uses it. Twilio uses it. It's not an advanced concept - it's a standard tool that you'll reach for every time you build an endpoint that creates something. 5. The cost of not knowing this is real. Duplicate charges, duplicate orders, duplicate emails. It's the kind of bug that doesn't show up in tests (you usually don't retry in tests) and only surfaces in production, at scale, when the network misbehaves. github.com/Navashub/idempotency-demo The repo has two endpoints and a README that walks through the same scenario. Clone it, run it, open /docs, and hit Execute on both endpoints a few times. The visual difference between a changing charge_id and a stable one is the whole lesson in one browser tab. If you want to extend it: add Redis as the idempotency store instead of the in-memory dict. That's the production step. The logic stays identical — the storage layer is the only thing that changes. I'm an AI Engineer in Nairobi. I write about what I'm building and learning - because explaining it is how I understand it.