DreamLake

Deployment index safety

2026-09-15 — staging application deployed; gates remain off

The approved cb73cee application revision deployed through run34938027728. The ordinary staging fast-forward preserved concurrent work. Additive reconciliation verified 232 contracts and created zero indexes. ECS settled on revision 163, image digest a84bcad9c82bc736fe236a064f3b4574aae45eedd3b051493316682c46de8bb4; the public health endpoint returned200/statusok. The 06:56:13 UTC full-schema check found no missing indexes, blockers, duplicates, parallel arrays or changes from the reviewed migration state.

Direct task-descriptor parsing confirms private runs and KMS epoch migration are both explicitly false on163, with private origins absent. Revision162 explicitly had KMS migrationfalse and private flags absent/defaultoff; an earlier empty nested-query projection was ambiguous and is not retained as evidence. No production deployment or private-run enablement occurred. Sanitized deployment/flags/health/index receipt.

Reviewed staging index migration

PR490 merged at cb73cee after independent review and full CI on cb9761f. The explicitly scoped staging migration completed: two canonical Bindr indexes created and verified; only Bindr_projectNodeId_name_key dropped. Full-schema verification at 06:35:22 UTC found all 232 contracts satisfied, zero additions/blockers/duplicate groups/parallel arrays, and no index differences outside the three reviewed changes. Staging remained on task revision162/source cd490e0; no application deployment or production database mutation occurred.

Sanitized before/plan/apply/after receipts include empty active-deployment checks before and after. Those checks and repeated source/index verification are not an atomic lock against external manual DDL. The old manual db:ensure-indexes / Source reset helper still drops historical Source indexes and was not invoked. Source/Connection container startup verifies only; it does not call that helper.

Earlier inventory and implementation boundary

Tracks Vault #241, staging reconciliation #387 and master #247. Reconciliation PR481 merged; this follow-up replaces unconditional deployment schema push and destructive index repair with explicit index planning and additive application. It preserves concurrent Env/Tasks changes.

A read-only inventory of settled staging revision 162, source cd490e0, checked all 87 schema collections / 232 index contracts. No duplicate groups or parallel-array blockers were found. Two canonical Bindr indexes need adding, and the older Bindr_projectNodeId_name_key conflicts with retired-name reuse. The generic synchronizer refuses this extra unique constraint; it never silently drops or repairs it. This is staging evidence, not a production inventory or a deployment claim.

The separate Bindr migration pins the reviewed database fingerprint, task definition, source digest and plan hash. It checks exact old-index metadata, creates and verifies the canonical unique (projectNodeId, name, deletedAt) and nonunique parentId indexes, then drops only the exact old constraint. Unfamiliar metadata or data refuses mutation. Stop concurrent deployment/index writers for this explicit operation: MongoDB's name-based drop cannot atomically compare the previously inspected index metadata.

Operator commands

Run from dreamlake-server. The inventory reads the current settled ECS task's connection privately; it emits index metadata and aggregate counts only. Credentials never appear in arguments or receipts.

shell
pnpm exec tsx scripts/schema-index-preflight.ts --staging-current \
  > /tmp/staging-index-inventory.json
# Exit 2 means inspect the reported blockers; this command makes no changes.

pnpm exec tsx scripts/migrate-bindr-name-index.ts --plan \
  --target-receipt /tmp/staging-index-inventory.json \
  > /tmp/bindr-index-plan.json

# Run only the reviewed plan, with concurrent index/deployment writers excluded.
pnpm exec tsx scripts/migrate-bindr-name-index.ts --apply-reviewed \
  --target-receipt /tmp/bindr-index-plan.json

pnpm exec tsx scripts/schema-index-preflight.ts --staging-current \
  > /tmp/staging-index-after.json

Dataset name index (issue #1353)

Project-level Datasets follow the same retired-name contract: the schema key is now unique (projectNodeId, name, deletedAt), replacing Dataset_projectNodeId_name_key. scripts/migrate-dataset-name-index.ts runs the same reviewed procedure as the Bindr migration: create and verify Dataset_projectNodeId_name_deletedAt_key, then drop only the exact old key. Its additions and its one drop are fixed in src/deployment/datasetNameIndexMigration.ts; it cannot drop anything else. Rows are not modified, and existing soft-deleted Datasets release their names once the old key is gone. The new key is strictly looser than the old one, so existing data cannot produce duplicates.

Until this migration runs on a database, deploying this schema there refuses at index preflight, before the application rolls out: the new key is an addition and the old key an unreviewed unique constraint. The server already running there is unchanged: create on a deleted name answers 409, and a rename onto any existing name answers 500. Run the migration first; the running server works with either key, and gains deleted-name reuse once the old key is gone.

Forward-only. As soon as a deleted name is reused, the old key cannot be rebuilt; rolling back means fixing forward, not restoring Dataset_projectNodeId_name_key. Production completed the targeted migration in the reviewed index-reconciliation operation. The post-migration inventory verified all 305 expected production indexes with no additions, blockers, duplicate groups or parallel arrays. This is production evidence only; check a fresh target-bound plan before operating on another database.

Staging uses the receipt mode of the Bindr commands above, with this script and a dataset-index-plan.json output. Any settled ECS service, including production, uses the ECS-target mode: DATABASE_URL is the database to change, and every check requires it to match, by databaseFingerprint, the database of the settled service named by INDEX_SYNC_ECS_CLUSTER, INDEX_SYNC_ECS_SERVICE and AWS_REGION, with the same settled-service checks as the deploy preflight. Supply those four variables privately in the shell; never in arguments.

shell
# 1. Read-only. Review the output: zero blockers, duplicates and parallel arrays;
#    `additions` names only the new key; `drop` is the old key (or null if done).
pnpm exec tsx scripts/migrate-dataset-name-index.ts --plan --ecs-target \
  > /tmp/dataset-index-plan.json

# 2. Apply exactly the reviewed plan, with concurrent deployments and index writers stopped.
pnpm exec tsx scripts/migrate-dataset-name-index.ts --apply-reviewed --ecs-target \
  --target-fingerprint "$(jq -r .targetFingerprint /tmp/dataset-index-plan.json)" \
  --plan-hash "$(jq -r .planHash /tmp/dataset-index-plan.json)" \
  --confirm SWAP_DATASET_PROJECT_NAME_KEY_ONLY

# 3. Read-only again: `additions` empty and `drop` null.
pnpm exec tsx scripts/migrate-dataset-name-index.ts --plan --ecs-target

Apply refuses without the exact confirmation, a well-formed fingerprint and plan hash, or when the target or index inventory no longer matches the reviewed plan; it re-verifies the target before each change. Production execution needs its own authorization, and the operator's existing read access to the ECS service and database; this procedure grants none.

Do not reuse a stale receipt after another rollout or index change. A failed or interrupted operation may already have created an index or completed the drop; obtain a fresh inventory and migration plan to resolve the outcome. Repeating a freshly planned completed migration is a no-op. There is no automatic rollback, broad drop, data cleanup or whole-schema push.

Failed workflow preflight retains the sanitized index plan as a seven-day CI artifact: schema index metadata and aggregate counts, with no document values, tenant names or connection credentials. Private-run configuration is forwarded without enabling it: DREAMLAKE_VAULT_PRIVATE_RUNS_ENABLED defaults to false; unset API/repository origins are omitted, and supplied values retain JSON escaping and fail-closed bootstrap validation.

The normal workflow independently checks its configured database against the settled ECS database, validates all expected constraints before creating any, and binds apply to the structural plan hash. Equivalent index names are retained. Unique constraints and TTL indexes affect writes and retention, so additive does not mean behavior-free or atomic; races or build failures can leave acknowledged additions.

Validation and remaining work

Forty focused tests passed across four suites, including owned real Mongo, complete-schema CLI application, idempotence, target/hash refusal, malformed/duplicate data, partial-index semantics, replacement failure, fresh-plan restart and retired-name reuse. Typecheck passed. The reviewed staging index migration and full-schema post-check are now complete. Staging application rollout, health, actual disabled flags and full-schema verification are complete. Private-run enablement and authenticated hosted feature acceptance remain separate gates; no production database changes are claimed.

Full sanitized inventory · Read-only Bindr plan · Remote delivery evidence.