Two Tools, One Rule: Why Dev Tooling Goes Through the Service Layer (Except When It Doesn't)
A raw-SDK cleanup script once deleted zero records it should have deleted, because it reimplemented entity shape independently of the real write path. The fix: dev tooling now calls the same service functions production does — with one deliberate, structurally-separate exception for state the service layer can't reach
Every backend eventually needs a way to conjure up test data fast — a tenant, some users, a subscription — without going through the real UI flow every time. The obvious way to build that is a script that writes directly to the database and the auth provider: fast, simple, no dependency on the app's own service layer. InvocaCare had exactly that script. It's also what caused a real bug that quietly deleted zero records when it should have deleted several, and traced back to the actual design mistake underneath it.
The convenient shortcut, and why it drifted
The original cleanup.sh reimplemented tenant/user teardown directly against DynamoDB and Cognito via the AWS SDK — its own understanding of what a tenant record and a user record look like, independent of whatever tenantService/userService actually write. That independence is exactly the problem. Production's tenant lifecycle is deliberately asynchronous: tenantService.registerTenant writes the tenant's METADATA row and emits TenantCreated; two separate Lambda subscribers pick that event up and asynchronously provision the SUBSCRIPTION row and the founding admin's USER# row. That choreography is intentional — failure isolation across Stripe, Cognito, and DynamoDB as independently-owned systems that shouldn't be able to block each other.
A raw-SDK script has to reimplement that entity shape by hand, and hand-reimplementations drift. The concrete failure: an investigation into cleanup.sh deleting zero records for a test-data prefix that clearly had matching tenants traced back to the script matching on METADATA.name — while the actual e2e test data's naming convention lived on the USER# row's email field, not on the tenant name at all. Real orphaned SUBSCRIPTION-only rows with no METADATA sat there too, another direct symptom of a script that constructs entity writes independently of the real write path and has no compiler or test to catch the two definitions drifting apart.
The decision: dev tooling calls the same service functions production does
The fix wasn't to patch the field-matching bug and move on — that leaves the root cause in place for the next entity-shape change. The actual decision: manual dev tooling and e2e test helpers create/read/update/delete data by calling the real tenantService/userService/billingService/item DAL functions in-process — the exact same functions the production Lambda handlers call — instead of writing DynamoDB items and Cognito users directly.
That's dev-cli.ts: a thin [--flags] grammar (tenant create, user create, billing update --recomputeSeats, and so on) that delegates to shared test fixtures, which call the real service layer. One shared flag dictionary keeps every entity/verb combination spelling the same concept the same way — --tenantId always means the same thing, everywhere. There's no separate "test data shape" to keep in sync with the real one, because there's only one shape: whatever the service layer actually produces.
An alternative considered and rejected: build a synchronous, in-memory EventBridge double for tests, so registerTenant's event emission triggers the billing/user handlers immediately in the same call stack instead of asynchronously. That would make dev/test tooling deterministic without touching raw AWS SDK calls at all. It didn't ship, for a reason worth being honest about even in a decision that "won": it would mean tests exercise a different event-dispatch mechanism than production actually uses, which is its own kind of drift — just moved from the entity-shape layer to the choreography layer instead of eliminated.
The deliberate exception: repair-orphans.sh
Going all-in on service-layer calls creates a real gap: what do you do about state the service layer can't reach — records already left behind by a historical bug, a crashed test run, or manual intervention? That's exactly what repair-orphans.sh is for, and it's kept structurally separate from dev-cli.ts on purpose, not merged into one CRUD surface. Its own header comment says why directly: it operates on raw DynamoDB/Cognito primitives with no app-layer types or business rules, and it's deliberately not TypeScript — so running it always feels different from the safe path, a small but real guardrail against reaching for it out of habit.
Its scan verb is read-only and reports exactly the shapes of inconsistency that matter: METADATA-only tenants, SUBSCRIPTION-only tenants, orphaned USER# rows, and Cognito users with no matching DynamoDB row. Its delete verb is where the real design care shows up. A full wipe defaults to protecting TENANT#SYSTEM — the SystemAdmin's own tenant — and a comment in the code explains a bug that was caught and fixed before it caused damage: an explicit --protect-pk used to replace that default protection instead of adding to it, meaning a bare --protect-pk TENANT#foo would silently strip SystemAdmin's own tenant of its protection while protecting someone else's. It now always adds to the default.
The subtler fix sits in how DynamoDB protection and Cognito protection stay in sync. Protecting a tenant's DynamoDB rows by PK does nothing for its Cognito identities unless something explicitly connects the two — so before the Cognito deletion phase runs, the script looks up every protected PK's USER# row emails and folds them into the Cognito-side protected-email list for that run. Skip that step and you get exactly the kind of split-brain inconsistency the whole tool exists to clean up: a protected tenant's DynamoDB rows survive a wipe while its own Cognito users get deleted anyway, or the reverse.
Why two tools, not one
It would be simpler, on paper, to have one CRUD tool that does everything. The actual constraint runs the other way: correctness for the normal path (deterministic test data, always shaped exactly like production data) and correctness for the exception path (surgical repair of state the normal path structurally cannot produce or reach) are different problems, and conflating them is how you end up with a single tool that's slightly wrong at both jobs. dev-cli.ts never has to think about orphaned, half-provisioned state, because service-layer calls can't produce it. repair-orphans.sh never has to worry about staying in sync with evolving entity shapes as a "convenience," because it isn't pretending to be the safe path — its own comments say so, out loud, every time someone opens the file.
