The Bug That Wasn't Where I Thought It Was: Chasing Down Duplicate Users in an At-Least-Once World
An occasional orphaned user row looked like DynamoDB read-consistency lag. It was actually Lambda's own async-invocation retry — a second mechanism stacked underneath EventBridge's, firing about a minute later on the same trace ID and creating a duplicate Cognito user for a tenant already mid-deletion
Part 5 of 7 — InvocaCare Architecture series
Some bugs announce themselves. This one didn't. It showed up as an occasional, seemingly random e2e test failure — a tenant left behind after a test that should have cleaned up after itself, an orphaned USER# row in DynamoDB with no matching tenant METADATA record. Easy to write off as test flakiness. Easy, and wrong.
Here's the investigation, because the actual root cause turned out to be more interesting — and more instructive — than the first three explanations I considered and ruled out.
The symptom
user/event-handler.ts listens for a TenantCreated event and provisions the founding admin user via Cognito's AdminCreateUser. Straightforward, one call, should run exactly once per tenant. Except every so often, an e2e test's cleanup step would find a USER# row with no tenant behind it — a user that outlived the tenant it belonged to, or was created twice.
My first theory was DynamoDB read-consistency lag — a plausible, boring explanation, and the kind of thing you can talk yourself into believing without checking, because it doesn't require finding anything actually wrong with your own code. I checked anyway.
Following the actual evidence
I pulled CloudWatch Logs for two of the seven orphaned tenant IDs and looked at what really happened, in order. Both showed the identical shape: a TenantCreated invocation creates the admin user, then throws before it reaches the next step (updateTenant({ownerUserId}) — no success log for that call). Then, 56 to 61 seconds later, the same X-Ray trace ID re-invokes the handler and creates a second Cognito user for the same tenant.
Same trace ID is the detail that mattered. That's not EventBridge redelivering an event — that's Lambda's own built-in asynchronous-invocation retry, kicking in on the same invocation, independent of whatever retry policy is or isn't configured on the EventBridge target. In one case, a TenantDeleted event — "all users deleted for tenant" — landed chronologically between the two TenantCreated attempts. The tenant was being deleted while the automatic retry was still in flight.