Core app S3 prefixes
This inventory records core app key builders and their owning implementations. Before this inventory, storage conventions were scattered across service files, Memos design docs, deployment configuration and individual feature guides. There is no runtime registry that enumerates every key or every customer bucket. Update this page when introducing a bucket selector, prefix builder or retention rule. Source paths below are relative to the workspace repository.
Bucket selection and observed configuration
Use the configured S3_BUCKET for ownership-transfer snapshots, alongside
private core app objects. A production Memo read and a read-only AWS audit observed
amzn-s3-dreamlake-bucket in us-east-1 on October 15, 2026.
The AWS audit at approximately 07:03–07:06 UTC found bucket versioning unset
(unversioned), default AES256 encryption, and all four S3 public-access blocks
enabled. Its only lifecycle rule aborted incomplete multipart uploads after
seven days; it had no current-object or noncurrent-version expiration rule.
Bucket policies and credential grants were not audited, and the inspection
performed no object writes. These are dated observations, not guarantees about
future configuration. Verify deployment variables and policies before an
operational change.
Catalog storage uses separate selectors and may use different buckets. A source
default such as dreamlake-artifacts is not evidence that the deployed app uses
that bucket. Per-resource catalog bindings override birth-location defaults;
ownership/project changes must not reconstruct storage paths from a new slug.
| Family | Bucket resolution, first configured value wins | Region resolution |
|---|---|---|
| Core Memos, attachments, media, speech, project files, transfer snapshots | S3_BUCKET (required for writes) | AWS_REGION, then us-east-1 |
| Artifacts | pinned binding, otherwise ARTIFACTS_BUCKET, then dreamlake-artifacts | ARTIFACTS_REGION, AWS_REGION, us-east-1 |
| Workflows | WORKFLOWS_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifacts | WORKFLOWS_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1 |
| Annotations | pinned binding, otherwise ANNOTATIONS_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifacts | ANNOTATIONS_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1 |
| Envs and Scenes | pinned binding, otherwise ENVS_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifacts | ENVS_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1 |
| Asset libraries | LIBRARIES_BUCKET, ARTIFACTS_BUCKET, dreamlake-artifacts | LIBRARIES_REGION, ARTIFACTS_REGION, AWS_REGION, us-east-1 |
| Managed DreamDB Sources | SOURCE_ASSETS_BUCKET; legacy connector defaults to dreamlake-source-assets, Source v1 startup requires an explicit bucket | SOURCE_ASSETS_REGION, AWS_REGION, us-east-1 |
| External Sources / execution Storage definitions | authorized Connection/Storage configuration | configured provider region/endpoint |
Core S3 clients use server base credentials and support AWS_ENDPOINT and
S3_FORCE_PATH_STYLE where implemented. Catalog families additionally have
family-specific endpoint/credential settings; consult their credential builders
rather than assuming one shared identity. Never expose those credentials in docs.
Prefix inventory
ns below means a namespace slug; namespaceId means its stable opaque ID.
Stored keys and catalog bindings remain authoritative for existing objects.
A prefix is a storage layout, not an authorization check or Project affiliation.
| Family and key shape | Owning source under dreamlake-server/src/ | Access and retention boundary |
|---|---|---|
Memo root notes/<ns>/<noteId>; immutable body .../projections/<sha256> and legacy .../revisions/<hash> | services/notesStorage.ts | Authenticated Memo API or explicit sharing; native RTC operations own edits, S3 body objects are projections. Do not expire live bodies/history by age. |
Memo saved versions, merge baselines and timeline under the stored Memo root: versions/, rtc-baselines/<tokenHash>.msgpack, timeline/<time>--<identity>.json | services/noteVersions.ts, notesRtcBaselines.ts, notesTimeline.ts | Revision/merge/history recovery data; follow owning service cleanup, not blanket root expiry. |
Memo files notes/files/<namespaceId>/<noteId>/<randomHex> and publication derivatives | services/noteFiles.ts, noteFilesStorage.ts, noteFileJournalStorage.ts | Note-authorized attachment API; immutable publication and recovery manifests govern deletion. |
Media notes/media/<randomId> | services/notesMediaStorage.ts, routes/notesMedia.ts | Anyone holding the media URL can read; do not store transfer snapshots here. |
Speech speech-cache/v1/<encodedNamespaceId>/<encodedNoteId>/<hash>.ndjson | services/speechCache.ts | Recomputable cache, bounded by service limits; no blanket automatic TTL is established by this inventory. |
Debug reports debug/notes/<ns>/<noteId>/<time-random>.json | services/notesStorage.ts (putDebugReport) | Sensitive diagnostic/CRDT payloads; server-authorized access, no public ACL. Audit retention separately. |
Artifacts users/<ns>/<artifactId>/... or pinned binding prefix | services/artifactCredentials.ts, artifactRead.ts | DreamDB refs/chunks inside the dataset; scoped credentials and authorized read broker. Ownership changes preserve pinned storage location. |
Workflows workflows/<ns>/<name>/... | services/workflowSpecCredentials.ts, workflowSpec/s3Backend.ts | DreamDB dataset; scoped credentials and workflow access policy. |
Annotations annotations/<ns>/<name>/... or pinned binding | services/annotationStorage.ts | DreamDB dataset, scoped resource credentials. |
Envs and Scenes envs/<ns>/<name>/... or pinned binding | services/envStorage.ts, routes/envsPublic.ts | Scenes reuse Env storage; public visibility/share policy controls reads. No separate universal scenes/ prefix. |
Libraries libraries/<ns>/<name>/files/<relativePath> | services/assetLibraryStorage.ts, assetLibraryManifest.ts | Scoped upload credentials; manifest-driven reconcile owns deletion. Reserved files/.dreamlake/ contains manifest/vectors; legacy files/assets.json remains readable. |
Project uploads files/<projectNodeId>/<hash>; legacy BSS keys can differ | services/projectUploadPresign.ts, routes/assets.ts | Project permission and exact prefix containment; deletion uses cleanup tasks. Do not infer namespace from path or rewrite legacy keys. |
Managed Sources sources/<sourceId>/<relativePath> | sources/connectors/dreamdb.ts, sources/v1/bootstrap.ts | Source authority and configured managed bucket; external S3 Sources retain customer-selected bucket/root and Connection identity. |
Transfer ledgers ownership/transfers/v1/<sourceNamespaceId>/<operationId>/execution/<sha256>.json | ownership/transferSnapshots.ts (snapshot-capable release) | Server-private immutable checksummed ledgers; terminal-only cleanup after retention, described below. |
Check the current implementation when reading legacy data: keys are persisted and transfers may retain an original prefix. Catalog dataset internals (refs, chunks and manifests) are owned by DreamDB, not a second hand-written key schema.
Ownership snapshot convention
The snapshot-capable backend writes whole execution-ledger revisions to the
core bucket under the dedicated ownership/transfers/v1/ prefix. Stable source
namespace and operation IDs isolate one transfer; SHA-256 addresses immutable
JSON bytes. The journal stores bucket, region, key, checksum, byte count and
optional S3 version ID. The server verifies bytes before adopting the reference.
New S3 ledgers omit original Memos' derived searchText, retain RTC-prepared
replacement text, preserve canonical preview hashes, and verify the compact
form's integrity. Existing documents, files and history stay in place;
permanent ownership proofs remain in MongoDB after snapshot cleanup
(backend PR1163).
New snapshot admission uses OWNERSHIP_TRANSFER_SNAPSHOTS_ENABLED, disabled by
default until all executors are upgraded or drained. Disabling admission still
allows recovery and cleanup of already pinned snapshots. Metadata endpoints
require current source-namespace ownership; receiving owners do not receive S3
locators. Objects request AES256 server-side encryption and no public ACL. Retain the
metadata/audit record after deleting snapshot bytes.
Cleanup is server-controlled, not an age-only S3 lifecycle rule: active operations, held execution leases and recovery-required operations must retain their snapshots. Terminal operations retain snapshots for at least seven days from their terminal update. An explicit operation cleanup enumerates its whole prefix, including superseded/orphan revisions and version/delete-marker history. Do not configure unconditional expiry under this prefix: a stalled operation may still need its original snapshot long after creation.
Use the matching CLI build to inspect ownership snapshot <operation-id> and
preview ownership snapshot-cleanup <operation-id>; --execute requests actual
cleanup. A CLI installation does not prove the compatible backend is deployed.
See the CLI ownership reference for the complete command contract.
Other storage boundaries
Infrastructure stack state and version history in the core app live in
PostgreSQL (dreamlake-server/src/infrastructure/store.ts), not a shared S3
prefix. The optional AWS jump-worker Terraform template has its own operator
configured S3 backend and state_key; that is separate from the app's
Infrastructure Stack HTTP backend.
Code snapshot registrations store an archiveKey and selected Storage/backend
(dreamlake-server/src/routes/declarations.ts). S3 archives, execution results
and run outputs can use operator-defined Storage bindings or external execution
services. They have no universal core-app bucket/prefix. GitHub-backed snapshots
store a ref instead of S3 bytes. BSS episode data and ML-Dash experiment data also
have service-specific storage contracts; this inventory does not authorize
sweeping their buckets or claim it enumerates their dynamic object layouts.
Maintenance and cleanup rules
Keep new prefixes versioned when their persisted schema needs migration. Use stable IDs for new ownership/recovery data, explicit resource authorization, checksums for immutable bytes and exact slash-delimited containment. Never use prefix spelling as a substitute for database authority. Resolve stored bindings before defaults, and document exceptional retention with the owning service.
Before deleting, identify the owning catalog/journal, live references, recovery requirements, bucket versioning and any legal/operational retention. A delete marker is not removal of old S3 versions. This page records application behavior; live IAM policies and lifecycle configurations require separate verification.