DreamLake

Core app S3 prefixes

This inventory records core app key builders and their owning implementations. Before this inventory, storage conventions were scattered across service files, Memos design docs, deployment configuration and individual feature guides. There is no runtime registry that enumerates every key or every customer bucket. Update this page when introducing a bucket selector, prefix builder or retention rule. Source paths below are relative to the workspace repository.

Bucket selection and observed configuration

Use the configured S3_BUCKET for ownership-transfer snapshots, alongside private core app objects. A production Memo read and a read-only AWS audit observed amzn-s3-dreamlake-bucket in us-east-1 on October 15, 2026. The AWS audit at approximately 07:03–07:06 UTC found bucket versioning unset (unversioned), default AES256 encryption, and all four S3 public-access blocks enabled. Its only lifecycle rule aborted incomplete multipart uploads after seven days; it had no current-object or noncurrent-version expiration rule. Bucket policies and credential grants were not audited, and the inspection performed no object writes. These are dated observations, not guarantees about future configuration. Verify deployment variables and policies before an operational change.

Catalog storage uses separate selectors and may use different buckets. A source default such as dreamlake-artifacts is not evidence that the deployed app uses that bucket. Per-resource catalog bindings override birth-location defaults; ownership/project changes must not reconstruct storage paths from a new slug.

FamilyBucket resolution, first configured value winsRegion resolution
Core Memos, attachments, media, speech, project files, transfer snapshotsS3_BUCKET (required for writes)AWS_REGION, then us-east-1
Artifactspinned binding, otherwise ARTIFACTS_BUCKET, then dreamlake-artifactsARTIFACTS_REGION, AWS_REGION, us-east-1
WorkflowsWORKFLOWS_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifactsWORKFLOWS_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1
Annotationspinned binding, otherwise ANNOTATIONS_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifactsANNOTATIONS_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1
Envs and Scenespinned binding, otherwise ENVS_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifactsENVS_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1
Asset librariesLIBRARIES_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifactsLIBRARIES_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1
Managed DreamDB SourcesSOURCE_ASSETS_BUCKET; legacy connector defaults to dreamlake-source-assets, Source v1 startup requires an explicit bucketSOURCE_ASSETS_REGION, AWS_REGION, us-east-1
External Sources / execution Storage definitionsauthorized Connection/Storage configurationconfigured provider region/endpoint

Core S3 clients use server base credentials and support AWS_ENDPOINT and S3_FORCE_PATH_STYLE where implemented. Catalog families additionally have family-specific endpoint/credential settings; consult their credential builders rather than assuming one shared identity. Never expose those credentials in docs.

Prefix inventory

ns below means a namespace slug; namespaceId means its stable opaque ID. Stored keys and catalog bindings remain authoritative for existing objects. A prefix is a storage layout, not an authorization check or Project affiliation.

Family and key shapeOwning source under dreamlake-server/src/Access and retention boundary
Memo root notes/<ns>/<noteId>; immutable body .../projections/<sha256> and legacy .../revisions/<hash>services/notesStorage.tsAuthenticated Memo API or explicit sharing; native RTC operations own edits, S3 body objects are projections. Do not expire live bodies/history by age.
Memo saved versions, merge baselines and timeline under the stored Memo root: versions/, rtc-baselines/<tokenHash>.msgpack, timeline/<time>--<identity>.jsonservices/noteVersions.ts, notesRtcBaselines.ts, notesTimeline.tsRevision/merge/history recovery data; follow owning service cleanup, not blanket root expiry.
Memo files notes/files/<namespaceId>/<noteId>/<randomHex> and publication derivativesservices/noteFiles.ts, noteFilesStorage.ts, noteFileJournalStorage.tsNote-authorized attachment API; immutable publication and recovery manifests govern deletion.
Media notes/media/<randomId>services/notesMediaStorage.ts, routes/notesMedia.tsAnyone holding the media URL can read; do not store transfer snapshots here.
Speech speech-cache/v1/<encodedNamespaceId>/<encodedNoteId>/<hash>.ndjsonservices/speechCache.tsRecomputable cache, bounded by service limits; no blanket automatic TTL is established by this inventory.
Debug reports debug/notes/<ns>/<noteId>/<time-random>.jsonservices/notesStorage.ts (putDebugReport)Sensitive diagnostic/CRDT payloads; server-authorized access, no public ACL. Audit retention separately.
Artifacts users/<ns>/<artifactId>/... or pinned binding prefixservices/artifactCredentials.ts, artifactRead.tsDreamDB refs/chunks inside the dataset; scoped credentials and authorized read broker. Ownership changes preserve pinned storage location.
Workflows workflows/<ns>/<name>/...services/workflowSpecCredentials.ts, workflowSpec/s3Backend.tsDreamDB dataset; scoped credentials and workflow access policy.
Annotations annotations/<ns>/<name>/... or pinned bindingservices/annotationStorage.tsDreamDB dataset, scoped resource credentials.
Envs and Scenes envs/<ns>/<name>/... or pinned bindingservices/envStorage.ts, routes/envsPublic.tsScenes reuse Env storage; public visibility/share policy controls reads. No separate universal scenes/ prefix.
Libraries libraries/<ns>/<name>/files/<relativePath>services/assetLibraryStorage.ts, assetLibraryManifest.tsScoped upload credentials; manifest-driven reconcile owns deletion. Reserved files/.dreamlake/ contains manifest/vectors; legacy files/assets.json remains readable.
Project uploads files/<projectNodeId>/<hash>; legacy BSS keys can differservices/projectUploadPresign.ts, routes/assets.tsProject permission and exact prefix containment; deletion uses cleanup tasks. Do not infer namespace from path or rewrite legacy keys.
Managed Sources sources/<sourceId>/<relativePath>sources/connectors/dreamdb.ts, sources/v1/bootstrap.tsSource authority and configured managed bucket; external S3 Sources retain customer-selected bucket/root and Connection identity.
Transfer ledgers ownership/transfers/v1/<sourceNamespaceId>/<operationId>/execution/<sha256>.jsonownership/transferSnapshots.ts (snapshot-capable release)Server-private immutable checksummed ledgers; terminal-only cleanup after retention, described below.

Check the current implementation when reading legacy data: keys are persisted and transfers may retain an original prefix. Catalog dataset internals (refs, chunks and manifests) are owned by DreamDB, not a second hand-written key schema.

Ownership snapshot convention

The snapshot-capable backend writes whole execution-ledger revisions to the core bucket under the dedicated ownership/transfers/v1/ prefix. Stable source namespace and operation IDs isolate one transfer; SHA-256 addresses immutable JSON bytes. The journal stores bucket, region, key, checksum, byte count and optional S3 version ID. The server verifies bytes before adopting the reference. New S3 ledgers omit original Memos' derived searchText, retain RTC-prepared replacement text, preserve canonical preview hashes, and verify the compact form's integrity. Existing documents, files and history stay in place; permanent ownership proofs remain in MongoDB after snapshot cleanup (backend PR1163). New snapshot admission uses OWNERSHIP_TRANSFER_SNAPSHOTS_ENABLED, disabled by default until all executors are upgraded or drained. Disabling admission still allows recovery and cleanup of already pinned snapshots. Metadata endpoints require current source-namespace ownership; receiving owners do not receive S3 locators. Objects request AES256 server-side encryption and no public ACL. Retain the metadata/audit record after deleting snapshot bytes.

Cleanup is server-controlled, not an age-only S3 lifecycle rule: active operations, held execution leases and recovery-required operations must retain their snapshots. Terminal operations retain snapshots for at least seven days from their terminal update. An explicit operation cleanup enumerates its whole prefix, including superseded/orphan revisions and version/delete-marker history. Do not configure unconditional expiry under this prefix: a stalled operation may still need its original snapshot long after creation.

Use the matching CLI build to inspect ownership snapshot <operation-id> and preview ownership snapshot-cleanup <operation-id>; --execute requests actual cleanup. A CLI installation does not prove the compatible backend is deployed. See the CLI ownership reference for the complete command contract.

Other storage boundaries

Infrastructure stack state and version history in the core app live in PostgreSQL (dreamlake-server/src/infrastructure/store.ts), not a shared S3 prefix. The optional AWS jump-worker Terraform template has its own operator configured S3 backend and state_key; that is separate from the app's Infrastructure Stack HTTP backend.

Code snapshot registrations store an archiveKey and selected Storage/backend (dreamlake-server/src/routes/declarations.ts). S3 archives, execution results and run outputs can use operator-defined Storage bindings or external execution services. They have no universal core-app bucket/prefix. GitHub-backed snapshots store a ref instead of S3 bytes. BSS episode data and ML-Dash experiment data also have service-specific storage contracts; this inventory does not authorize sweeping their buckets or claim it enumerates their dynamic object layouts.

Maintenance and cleanup rules

Keep new prefixes versioned when their persisted schema needs migration. Use stable IDs for new ownership/recovery data, explicit resource authorization, checksums for immutable bytes and exact slash-delimited containment. Never use prefix spelling as a substitute for database authority. Resolve stored bindings before defaults, and document exceptional retention with the owning service.

Before deleting, identify the owning catalog/journal, live references, recovery requirements, bucket versioning and any legal/operational retention. A delete marker is not removal of old S3 versions. This page records application behavior; live IAM policies and lifecycle configurations require separate verification.