Part 6 of 7 — InvocaCare Architecture series

Infrastructure-as-code sells itself on a simple promise: the code is the truth, preview shows you the diff, apply makes reality match it. Most of the time that promise holds. This is about the time it didn't — when Pulumi's own bookkeeping about a live AWS resource silently diverged from the resource itself, and the tool kept confidently reporting success while doing nothing, or worse, refusing to do anything at all.

The setup

InvocaCare's Cognito User Pool has three custom schema attributes — custom:tenantId, custom:userRole, custom:tenantTier — that carry tenant identity through every request (the mechanism a companion post on this blog covers in detail). Cognito's API makes schemas immutable after pool creation: UpdateUserPool rejects any payload containing it, even one that matches what's already live byte-for-byte. That's an AWS API constraint, not a Pulumi one, and it should be a non-issue — Pulumi's job is to notice a resource needs no change and skip it.

It didn't skip it. It crashed.

First failure: pulumi refresh corrupts its own state

Running pulumi refresh against a stack with this User Pool hit a provider-level bug (Terraform-bridge AWS provider) in how it reconciles the schemas field — a Set-typed attribute — against AWS's full returned schema, which includes both the built-in Cognito attributes and the custom ones. The refresh doesn't corrupt the actual AWS resource; Cognito's live user pool is fine throughout. What gets corrupted is Pulumi's own recorded state for that field. Once that happens, every subsequent UpdateUserPool call fails with cannot modify or remove schema items — including calls that have nothing to do with schemas at all, because Cognito's update API treats the whole payload atomically.

The tempting fix — ignoreChanges: ['schemas'] — isn't one. It hits a different known bug in the same provider (Set-typed fields don't survive ignoreChanges correctly either), so it trades one corruption mode for another.

The actual mitigation is to never let a blanket refresh/preview/up touch that resource in the first place — scope it out explicitly:

pulumi refresh --stack  --yes \
  --exclude 'urn:pulumi:::invocacare::aws:cognito/userPool:UserPool::UserPool'

If the state is already corrupted, recovery is manual and unforgiving: export the stack (pulumi stack export), hand-edit inputs.schemas and outputs.schemas back to the real two-attribute value — cross-checked against aws cognito-idp describe-user-pool's live output, not guessed — and trim an internal delta-tracking array (outputs.__pulumi_raw_state_delta.obj.ps.schemas.arr.el) to match the same element count, or the provider throws a different error on the very next diff. Take a full backup before touching any of it, and verify the edit with a real semantic diff between the two JSON exports — a raw text diff on Pulumi state produces enormous, meaningless noise from formatting alone.

Second failure: a "successful" apply that changed nothing

The sibling bug is quieter and more dangerous, because it doesn't announce itself as a bug at all. pulumi up logs ~ aws:cognito:UserPool UserPool updated (1s), exits zero, and the next pulumi preview shows no diff — every signal says the change landed. It didn't. The provider was diffing against its own recorded outputs, and those outputs never actually got refreshed with what UpdateUserPool really persisted on the AWS side. Three specific fields are affected: lambdaConfig, emailConfiguration, adminCreateUserConfig — and this matches a filed, known upstream issue, not a one-off.

The tell is a pulumi preview that keeps showing the identical diff on one of those three fields across several deploys that each individually reported success. The only way to be sure is to stop trusting Pulumi's own diff and check AWS directly:

aws cognito-idp describe-user-pool --user-pool-id  --region  \
  --query "UserPool.{AdminCreateUserConfig:AdminCreateUserConfig,EmailConfiguration:EmailConfiguration,LambdaConfig:LambdaConfig}"

If live AWS still shows the old values despite a run that reported success, that confirms it's this bug and not a propagation delay. Unlike the schemas corruption, this one doesn't need state-file surgery — those three fields' internal delta entries are just presence markers, not duplicated literal values, so a corrected apply is safe. The fix is: push the exact values the code declares directly via the AWS CLI (Cognito replaces these as whole nested objects per call, so every field being fixed has to go in the same call), then re-run pulumi up --target scoped to just that resource's URN — never refresh again, same danger as before. Because AWS genuinely has nothing left to change at that point, that apply is the one that finally gets recorded correctly, and pulumi preview showing a real, stable zero-diff is the actual confirmation, not the exit code of the previous up.

One more trap worth naming: while chasing this, it's worth checking every Lambda ARN referenced inside lambdaConfig for staleness too. A resource rename can leave a Cognito trigger pointing at a physical Lambda name that no longer exists — and that failure mode won't surface until the specific trigger actually fires. PreTokenGenerationConfig breaking real sign-ins only shows up after sign-up succeeds, so a failure at the sign-up step in a test run can mask a second, unrelated broken trigger sitting one step further down the flow.

What this actually cost, and what it bought

Neither of these bugs is something application code, tests, or CI could have caught — they live entirely in the gap between what a cloud provider's API actually did and what your IaC tool's local bookkeeping believes it did. That gap doesn't show up in a diff review or a code review; it shows up as a deploy that reports success while quietly lying about it, discoverable only by cross-checking the tool's story against the platform's own truth.

The response wasn't to stop trusting Pulumi generally — it's still the source of truth for the other 90% of this infrastructure, correctly. It was to write down, precisely, the one resource where that trust needs a manual verification step (describe-user-pool after any lambdaConfig/emailConfiguration/adminCreateUserConfig change) and a hard rule (never blanket-refresh a stack with this User Pool). That's the actual deliverable from an incident like this: not a patch, because there's no patch to write for someone else's provider bug — a documented, specific distrust of one narrow surface, sized to exactly the blast radius of the actual failure mode, so the next engineer who touches this pool doesn't have to rediscover it the hard way.