Part 5 of 7 — InvocaCare Architecture series

Some bugs announce themselves. This one didn't. It showed up as an occasional, seemingly random e2e test failure — a tenant left behind after a test that should have cleaned up after itself, an orphaned USER# row in DynamoDB with no matching tenant METADATA record. Easy to write off as test flakiness. Easy, and wrong.

Here's the investigation, because the actual root cause turned out to be more interesting — and more instructive — than the first three explanations I considered and ruled out.

The symptom

user/event-handler.ts listens for a TenantCreated event and provisions the founding admin user via Cognito's AdminCreateUser. Straightforward, one call, should run exactly once per tenant. Except every so often, an e2e test's cleanup step would find a USER# row with no tenant behind it — a user that outlived the tenant it belonged to, or was created twice.

My first theory was DynamoDB read-consistency lag — a plausible, boring explanation, and the kind of thing you can talk yourself into believing without checking, because it doesn't require finding anything actually wrong with your own code. I checked anyway.

Following the actual evidence

I pulled CloudWatch Logs for two of the seven orphaned tenant IDs and looked at what really happened, in order. Both showed the identical shape: a TenantCreated invocation creates the admin user, then throws before it reaches the next step (updateTenant({ownerUserId}) — no success log for that call). Then, 56 to 61 seconds later, the same X-Ray trace ID re-invokes the handler and creates a second Cognito user for the same tenant.

Same trace ID is the detail that mattered. That's not EventBridge redelivering an event — that's Lambda's own built-in asynchronous-invocation retry, kicking in on the same invocation, independent of whatever retry policy is or isn't configured on the EventBridge target. In one case, a TenantDeleted event — "all users deleted for tenant" — landed chronologically between the two TenantCreated attempts. The tenant was being deleted while the automatic retry was still in flight.

That last part explains why this was so easy to misdiagnose as read-consistency lag: it looks like a timing race between reads, when the actual cause is a write — an unconditional retry — happening on a Lambda's own schedule, without asking whether the world it's about to act on has changed underneath it.

What this actually revealed

Widening the investigation past these two tenants surfaced that this wasn't a one-off — it was systemic:

• No EventBridge target in the codebase had a RetryPolicy or DeadLetterConfig. Every target — the user lifecycle handler, all three billing targets — was running on AWS's silent default: up to 185 retry attempts over up to 24 hours, failures dropped once that window closes, nothing to alert on.
• No event consumer guarded against reprocessing. TenantCreated's handler unconditionally called createUser, and Cognito mints a fresh sub on every call — so a retried delivery doesn't fail safely, it succeeds twice. The Stripe webhook handler had the identical gap: no check against Stripe's own event.id, despite Stripe's docs explicitly stating webhook redelivery is expected behavior, for up to three days, and explicitly recommending deduplication on event.id.
• This had already been the stated design intent and just never got built. An earlier ADR's context section describes tenant-lifecycle handlers as having "independent retry via DLQ per subsystem" as a deliberate property. It was written down as the intended architecture. It was never actually wired up in the infra code.
• There was also one partial precedent already in place: the billing service passed a Stripe idempotency key on outbound Stripe API calls, which makes those calls safe to retry — but does nothing for the inbound DynamoDB and Cognito writes a retried Lambda invocation performs around that call. Half the problem had a fix; the other half didn't.

And there's a second, independent retry layer hiding in this story that's easy to miss: Lambda's own async-invocation retry and EventBridge's target-level RetryPolicy are two separate, stacked mechanisms, one on the function and one on the event target. Fixing only the EventBridge side — the more commonly-discussed one — would have left Lambda's own retry (with a backoff measured in roughly a minute, not "up to 24 hours") free to keep producing the exact bug I was chasing.

The fix, and what I ruled out

The narrow, obvious first fix — an if (existing) return guard in the handler, mirroring a pattern already used elsewhere in the codebase — was tempting and genuinely wrong as the only fix. TenantCreated's user-creation step doesn't have a clean existing-record shape to check before the first write; and even where a check is possible, it has to be independently reasoned about and correctly reimplemented in every handler, forever, with no structural guarantee the next handler gets it right. It also doesn't stop reprocessing from re-running side effects that already happened before the guard's own check runs — a partial-completion replay slips right past it.

I also considered rolling a bespoke DynamoDB dedup table rather than adopting a library for it, and ruled that out too: AWS Lambda Powertools' idempotency utility was already a pinned dependency in this codebase for logging, metrics, and tracing, maintained by the same AWS team, and it correctly handles the genuinely hard parts of this problem — two concurrent deliveries of the same event both starting before either has recorded a result, TTL-based cleanup, distinguishing in-progress from complete from expired. Building that myself would have meant re-solving problems a well-maintained library had already solved, for a "savings" that was really just avoiding one new dependency I could already trust.

So the actual fix has two parts, doing two different jobs: every EventBridge target now gets an explicit, bounded RetryPolicy and a dedicated DLQ, provisioned through one shared infra helper so new targets get this by construction instead of by someone remembering to copy the config. And every Lambda consuming an at-least-once source — EventBridge events, Stripe webhooks — is wrapped with Powertools' idempotency utility, keyed on that source's own stable delivery identifier: event.id for EventBridge, Stripe's own parsed event.id for webhooks. The existing "don't act on a since-deleted tenant" guard stayed in place too — it protects a different invariant than delivery-deduplication does, and both are worth keeping.

The part I didn't go looking for

While touching the infra file for this fix, I found something unrelated: the main table had no TTL configuration in Pulumi at all, despite the rate limiter writing expiresAt attributes expecting DynamoDB to sweep them automatically. Those rows were never actually expiring. Not part of this bug, not caused by this bug — just sitting there, findable because I was already in the file. I tracked it as its own task rather than folding it into this fix, since conflating an unrelated bug with the one you're actually solving is how a clean fix turns into a change nobody can review properly — but it got fixed in the same PR, because the cost of fixing it right there was close to zero.

Why this one mattered

The instructive part isn't the fix — retry policies and idempotency keys are well-understood patterns. It's that the first explanation (read-consistency lag) was plausible, convenient, and wrong, and the only way to find that out was to pull the actual logs and follow the trace ID instead of accepting the story that required the least additional work from me. "At-least-once delivery" is a phrase every distributed-systems engineer can recite. Building the system so a specific, provable retry — 56 seconds later, same trace ID, mid-flight tenant deletion — can't quietly produce a duplicate user is the part that actually costs something to get right