The Webhook Bug That Passed Every Test and Every Code Review

When a subscriber disappears from a list, nobody gets an error. The campaign sends, the contact count is one fewer than expected, and unless someone is watching very carefully, you might never know you lost a subscriber who had nothing to do with that send. This is about a bug I found in my own code that did exactly that. It lived in a bounce handler, invisible to every test I wrote and every review pass I ran — including one where I explicitly sat down to look for security issues. This is Part 3 of a series on building a multi-tenant email service with FastAPI and Resend. Part 1 — Building the service: one Resend account, multiple customer domains, contacts, segments, campaigns. Part 2 — Making sends safe to retry: idempotency keys, per-recipient status tracking, the chunk key that was never actually retried. Part 3 (this one) — A webhook handler that looked correct and wasn't. The full working code is on GitHub: srinivaspavuluri/resend-fastapi The service manages email sending for multiple customers — think of each customer as a separate business using the same platform. The relevant tables: Customer — one row per tenant, each with a verified sending domain Contact — email addresses, scoped to a specific customer via customer_id Campaign — a send from a customer to their contacts CampaignRecipient — one row per contact per campaign; stores the resend_email_id that Resend assigns to each individual email When a campaign sends, every contact gets a CampaignRecipient row: db.add(CampaignRecipient( campaign_id=campaign_id, contact_id=contact.id, resend_email_id=result.resend_email_id, status="queued", )) That resend_email_id matters. It comes back in every webhook event Resend fires. Auto-unsubscribing on bounce is important for sender reputation. Resend monitors bounce and complaint rates on your account — if they climb too high, your sending ability gets restricted. So when a bounce arrives, you want to unsubscribe that contact immediately and automatically. My SECURITY.md noted this under "What's Done Well": Bounce/complaint auto-unsubscribe — protects sender reputation, scoped correctly. That credit was wrong. The bounce handler received an event from Resend with a payload like this: { "type": "email.bounced", "data": { "email_id": "ae2faec7-...", "to": ["user@gmail.com"] } } The original unsubscribe logic took the to field — the recipient email address — and used it to find the contact to unsubscribe: # The original approach to_emails = data.get("to", []) if isinstance(to_emails, str): to_emails = [to_emails] await db.execute( update(Contact) .where(Contact.email.in_(to_emails)) # no customer_id filter .values(is_subscribed=False) ) Find contacts with that email. Set them to unsubscribed. Simple. The problem is what "contacts with that email" actually means. The Contact table looks like this: class Contact(Base): id = Column(String, primary_key=True) customer_id = Column(String, ForeignKey("customers.id"), nullable=False) email = Column(String, nullable=False) is_subscribed = Column(Boolean, default=True) There is no unique constraint on email across the whole table. It's unique per customer, not globally. The same address — user@gmail.com — can legitimately exist as a Contact row under Customer A and Customer B. That's not a design flaw; it's the point of multi-tenancy. These are two separate businesses. Their subscribers don't know about each other. But the original bounce handler didn't know or care about any of that. It queried Contact.email.in_(to_emails) with no customer_id filter. If both Customer A and Customer B had user@gmail.com in their contact lists, and Customer A's email to that address bounced, the handler would unsubscribe both contacts — Customer A's and Customer B's. Customer B just lost a subscriber they never sent to. No error raised. No log entry about the wrong tenant. The contact row just flips to is_subscribed = False and the next campaign quietly skips them. I had tests covering the bounce handler. They verified that after a bounce event, the right contact ended up unsubscribed. Every test passed. But every fixture used a single customer. One customer, one contact, one campaign, one send. With only one customer in the database, there's no ambiguity about which Contact row a raw email query hits — it gets the right one every time. The bug is invisible in that setup. It only appears when you have two customers with an overlapping contact email, and that's a production condition, not one I thought to simulate. I never wrote the test that would have caught this because I never asked "what happens if the same email exists under a different customer?" The tests weren't wrong. They verified exactly what I asked them to verify. I just didn't ask the right question. Before writing Part 2, I ran a deliberate security pass over the codebase. I went handler by handler, route by route, and wrote up findings in SECURITY.md. The bounce handler made the "What's Done Well" list. The note said it was "scoped correctly." What I was actually reviewing during that pass was the signature verification — Resend signs every webhook with a Svix HMAC signature, and I was checking whether the verification logic was sound. I was asking "can this endpoint be spoofed or replayed?" That's the right question for a webhook security review. But it's not the only question. I wasn't asking "does this update the right rows?" because the handler looked like it did. You read Contact.email.in_(to_emails) and you think: yes, that's the email address that bounced, that's the contact to unsubscribe. It's only wrong if you're holding the full multi-tenant data model in mind at the same time — and in a review pass covering a dozen findings, I wasn't. The bug passed because I asked one security question thoroughly and missed a different one entirely. The fix uses data that was already there. Every CampaignRecipient row stores a resend_email_id. The bounce event carries that same ID. That row links back to a specific contact_id — one that belongs to exactly one customer, because it came from one campaign, which belongs to one customer. So instead of querying contacts by raw email, derive the contact through the recipient row: elif event_type == "email.bounced": email_id = data.get("email_id") if email_id: await db.execute( update(CampaignRecipient) .where(CampaignRecipient.resend_email_id == email_id) .values(status="bounced", updated_at=now) ) # Derive contact_id from CampaignRecipient, not from raw email. # The same email can exist under multiple customers — matching on email # with no customer_id filter would unsubscribe the wrong tenant's contact. contact_id_subq = ( select(CampaignRecipient.contact_id) .where( CampaignRecipient.resend_email_id == email_id, CampaignRecipient.contact_id.isnot(None), ) .scalar_subquery() ) await db.execute( update(Contact) .where(Contact.id == contact_id_subq) .values(is_subscribed=False) ) await db.commit() resend_email_id → CampaignRecipient.contact_id → Contact.id. Each step follows a relationship that's already scoped to the right tenant. There's no raw email lookup anywhere. Cross-tenant access is structurally impossible because there's no point in the chain where the query spans customers. Email addresses feel like natural lookup keys. They're unique in the human sense — one person, one address — and they're right there in the webhook payload. When a bounce arrives, your first instinct is: find the contact with that email, unsubscribe them. But in a multi-tenant system, "unique in the human sense" and "unique in your database" are different things. An email address is unique per customer. Globally, across all customers, it isn't. When you use a value that's globally unique in the real world but only tenant-scoped in your data model, you're querying at the wrong level of isolation. The query won't error. It'll return rows — plausible-looking rows — and update ones you never intended to touch. The correct instinct is to follow the relationship chain that the data model already enforces. resend_email_id → CampaignRecipient → contact_id works because every link in that chain belongs to one customer. The relationship scope is the tenant scope. You don't need to add a customer_id filter anywhere because the chain never leaves the right tenant. This is the kind of bug that doesn't announce itself. Nothing crashes. No test fails. No security scanner flags it. You find it by asking a question that your normal review process wasn't designed to ask: does this query respect the tenant boundaries my data model defines? This bounce handler bug was the headline finding, but the same review pass surfaced three others: A cascade gap. The endpoint for deleting a customer claimed to delete all their associated data, but the ORM relationship had no cascade="all, delete" set, and the SQLite dev environment never had PRAGMA foreign_keys = ON active. Campaign and recipient rows were silently orphaned on customer delete in dev. Postgres would have enforced the foreign key constraint and actually deleted them. Fixed with the cascade option on the relationship. A dead route. GET /{customer_id}/campaigns/{campaign_id} was defined in two different routers. Registration order meant one copy was always unreachable. Removed it. A replay window. The Svix signature verification had no timestamp freshness check, meaning a captured valid payload and signature could be replayed at any point in the future. Added a five-minute tolerance check before the HMAC comparison. None of these three are as consequential as the bounce handler bug. The cascade gap is a data hygiene problem in dev. The dead route is harmless. The replay window is a real vulnerability, but exploiting it requires capturing a valid signed payload first. The bounce handler bug was uniquely dangerous because the failure mode was completely silent — no crash, no error, just data quietly drifting in the wrong direction. The next piece in this series is a smaller fix with a disproportionate teaching value. The idempotency check at the top of the send endpoint — the one that short-circuits if a campaign with that key already exists — returns a raw 500 from an unhandled IntegrityError instead of the clean 409 it's supposed to return. One exception catch to fix it. But where you put that catch, and why it has to be there specifically, surfaces something worth understanding about the relationship between application-level error handling and database-level constraints. That's Part 4.