DreamLake
  1. DreamLake
  2. Notes
  3. Lakeshore
  4. ML-Dash
  5. CLI

[vault/kms/populated-migration] Implementation and rollout contract

Status: implementation merged in backend359, CLI59, Python45; native CLI 0.17.0/Python 0.14.0 released and the owned staging AWS forward/outage/reverse flow passed. Evidence and remaining gates keep production, GCP, second-tenant/wrong-context and additional live dependency cases separate. Migration is disabled by default; staging enablement followed full-fleet validation. Foundation merged: workspace353, CLI57, Python43. Track Vault241.

Public contract

Personal owner only. Start accepts prefix, trusted key reference and immutable request ID; no ARN or secret arguments. Add dreamlake vault -p alice/research kms migrate --key research-key --request-id migration-001, kms resume migration-001 --limit 50, kms status migration-001; Python client.vault.kms.migrate(prefix=..., key_ref=..., request_id=...), .resume(request_id=..., limit=50), .status(request_id=...). Start persists intent and changes future-write policy only after probe and a guarded commit. It never runs an unbounded batch implicitly. Status contains counts/checkpoints/errors without names, ciphertexts, KMS identifiers or secret material. The existing operation status route returns either activation or migration metadata; dedicated migration routes also remain available. Request-ID collisions across kinds must reject rather than replay a different operation.

managed is a reserved explicit target resolved from operator default configuration, never client-supplied key material. New custom references remain tenant-bound operator config. One configured provider is supported. AWS keys remain restricted to its configured region. GCP accepts operator-allowlisted CryptoKey locations; cross-location live acceptance is unverified. Existing exact boundary may change; an ungoverned populated prefix may acquire its own boundary. Parent/child overlap still conflicts. One active migration per intersecting prefix; migration reversal starts only after current completion, never rewinds a checkpoint.

Schema and invariants

Each migration gives the boundary a fresh random encryption epoch, preventing a return to an earlier key from accepting an old write. Envelope-bearing entries and write/HOTP operations carry internal encryption epoch. Legacy missing epoch is null. Internal fields must be excluded from all metadata/list/receipt serializers, including nested write snapshots. Operation records hold immutable tenant/prefix/target-reference/target-epoch, state, startedAt/completedAt and committed counts. Each encrypted record stores its target epoch inside the internal envelope. That marker is a durable checkpoint: a committed row is excluded from the next pending query, including after restart. Overlapping workers use exact old-envelope CAS and increment counts in the same transaction, so repeated work cannot double-count or silently skip rows.

Start probes outside transaction, then takes existing tenant guard, confirms current policy and no active overlapping migration, advances boundary epoch and records operation atomically. Existing data stays readable using retained historical provider keys. New writes must seal with fresh target epoch and commit only if the epoch still governs. A stale pre-start writer must fail closed instead of restoring old ciphertext. Every save path must participate, including ordinary update, idempotent update, HOTP activation/generation and retirement/restore. Metadata-only retirement/restore should modify its fields with exact identity/revision predicates without replacing a migrated envelope. Generic put is forbidden as a migration primitive.

Bounded work

Scan entries, write receipts, HOTP receipts independently in deterministic name/ID order, scoped in Mongo by tenant and prefix before limit. Include retired/expired-but-retained records. Open old envelope and seal unchanged plaintext using the existing context and target boundary outside transaction. Never generate an OTP, increment its counter or change a credential's logical revision. Commit replacement under tenant guard with entry identity, logical revision, old envelope and old epoch CAS; receipt identity and old envelope/epoch CAS similarly. If data was changed or purged, reread or move to a later reconciliation pass rather than overwrite. The envelope epoch and committed count update in the same transaction. On provider error, preserve the failed position and return a redacted retryable outcome; do not claim successful completion or fall back to another key.

After every bounded resume, take the guard and query all three for any non-target epoch. If any remain, the next resume selects them again; no positional cursor can hide a raced or failed row. Otherwise mark operation complete atomically. Writes after completion are already epoch-enforced. TTL deletion may remove a receipt during the scan; that is a safe no-op, never extend receipt retention. Purge and binding operations continue without logical credential rotation; stale retirement/restore must not restore the old envelope.

Recovery and old-key retention

Each request and resume is restartable from persisted metadata. Lost response is reconciled using the same operation ID. No implicit background loop or counter advancement. Completed means active database ciphertext dependencies covered, not backup re-encryption or remote credential revocation. Historical keys remain configured until actual retained backups and recovery windows no longer require them. No key deletion API is included. Removing a required old key or losing target permissions pauses progress with redacted failure; restored permission allows same-operation resume.

Required verification

Real Mongo+HTTP, synthetic KMS transport first: populated entries including retired/expired, receipt-only prefix, active HOTP and unexpired issuance receipt replay, idempotent write replay, host binding revision stability, restart between batches, lost committed resume response, competing collectors, inserted/updated/retired/restored/purged rows, start racing old-envelope writers, reverse/same-key epoch cases, tenant and scoped-key denial before KMS, selected-key denial/recovery without fallback. CLI/Python parity with immutable intent and bounded progress. Then dedicated disposable AWS key/role live denial/recovery; GCP project/API/bootstrap remains separately required. Do not fault shared staging/production keys or claim hosted acceptance from fixture evidence.

Paired usage

shell
dreamlake vault -p alice/research kms preview --key research-key
# Persist account/server/prefix/key/request ID before starting.
dreamlake vault -p alice/research kms migrate --key research-key --request-id research-move-001
dreamlake vault -p alice/research kms resume research-move-001 --limit 50
dreamlake vault kms status research-move-001
# Repeat resume explicitly until state is completed. Keep historical keys available.

The operator flag DREAMLAKE_VAULT_KMS_MIGRATION_ENABLED defaults to false; disabled runtimes omit migration routes and report the capability as false. Deploy epoch-aware code to every writing backend instance first, verify the fleet, then explicitly enable the flag. An older writer can ignore epochs and restore old ciphertext, so mixed-version writes are not a supported rollout. No hosted rollout has occurred.

Candidate scans use tenant/name/ID/epoch indexes and a five-second server query deadline. A batch limit bounds returned ciphertext and KMS work, not every index key examined; a sparse remainder can require scanning more index keys. Preview counts are live, not a snapshot or reserved total.

An explicit prefix on resume is a server-checked assertion against the original operation; a mismatch performs no KMS work or mutation. Omitting it retains ID-only recovery. Python uses optional prefix= with the same behavior. Status may also assert a prefix before returning metadata.

  • [vault/kms/fleet-rollout] Verify every live writer runs the reviewed epoch-aware revision with migration disabled, then enable migration and run isolated acceptance. Disabling the flag pauses API access but does not remove policies or undo progress. Never roll back to a pre-epoch writer after migration starts.

Completion timestamps are causally ordered as the later of the worker clock and persisted start time. This handles clock skew but does not measure precise elapsed duration.

The ECS deployment workflow forwards GitHub Environment variables DREAMLAKE_VAULT_KMS_MIGRATION_ENABLED (default false), DREAMLAKE_VAULT_KMS_PREFIX_KEYS (default empty registry) and DREAMLAKE_VAULT_KMS_ALLOWED_KEY_IDS (default primary key only). JSON registry strings are encoded as strings in the action input. This wiring does not create cloud grants or enable the flag; hosted environment readback remains part of rollout acceptance.