DreamLake

Connect Lakeshore to DreamLake

Use this guide when a Lakeshore server is already running. You need its HTTPS URL, a Lakeshore namespace, an access token for that namespace, and the DreamLake CLI.

If you are hosting the server yourself, first follow Setting Up Lakeshore Service. If someone else operates it, ask them for the URL, namespace, and token.

Connecting saves a server connection in DreamLake. It does not start the server or enroll workers. The Python SDK quick start can use the service directly without this connection.

Unified connections

The new Connect a Lakeshore form and dl lakeshore connect use the same connection API. Choose Execution / management (management) or Queue management (queues). These are allowed uses of one connection model, not separate service registrations. New connections encrypt their token in the connection record; they do not create or expose a shared Vault secret.

Both modes support personal and organization namespaces under existing DreamLake connection permissions. Queue mode verifies a namespace access token and uses the existing restricted proxy route set. It cannot enroll hosts, run tracked execution, open terminals or call producer/submit. Management mode supports those paths only when the caller and supplied token have permission; it does not grant admin rights. The connection check is not a capability grant.

Existing records retain their IDs, bindings and credentials. Missing accessMode fields are interpreted using the old record contract. Existing Vault-backed queue mounts remain restricted and continue using Vault. The old queues mount command/API remains personal-only and Vault-backed for compatibility. New CLI and web clients use the same APIs: management creation uses POST /namespaces/:slug/lakeshores, and queue creation uses POST /namespaces/:slug/lakeshores/connect-v1. Both store new credentials encrypted in the record.

Access mode is immutable in this release. Queue connections allow verified token replacement only. Existing management editing, enrollment, run and terminal behavior remains supported. No credential migration or automatic token upgrade occurs. dl launch keeps its existing direct-to-Lakeshore behavior with --server. With --mount, it submits through a management connection instead: POST /namespaces/:slug/lakeshores/:id/launches (lakeshore:use) checks a small configured snapshot, stores and registers it in that connection's code-staging storage, and submits the standard launch invocation with the stored credential; GET …/launches/:invocationId (lakeshore:read) reads its state and result, for launches submitted this way only. Queue connections are refused. See launch through a mount.

Experimental: Source-staged snapshots. A deployment can stage the snapshot in DreamLake instead of the connection's code-staging storage. The default is still code-staging; this mode is off until an operator enables it, either for listed namespaces with LAKESHORE_LAUNCH_SOURCE_STAGING_NAMESPACE_IDS (a JSON array of namespace ids) or for every namespace with LAKESHORE_LAUNCH_SOURCE_STAGING_ENABLED=true, and only after every worker on the affected queues runs dreamlake-lakeshore with the Source loader. LAKESHORE_LAUNCH_SOURCE_STAGING_QUEUE_NAMES (a JSON array of queue names) further limits it to launches submitted to those queues; unset or empty means no queue filter, and an invalid list or [] matches no queue. Launches to other queues keep code-staging. Older workers refuse these launches (archiveKey missing) instead of running them. In this mode, with the same request and CLI:

  • The snapshot is written once to snapshots/<sha256>.tar.gz in a private Source named launch-code-<user id> in the launch's namespace, in DreamLake's managed storage, so no bucket, S3 credential or control-plane storage is involved. Each user has their own: it is created by that user's first launch, and creating a Source needs namespace Owner or Admin, so a member without that role cannot make the first Source-staged launch. Once it exists, it is reused, and normal Source permissions apply: EDIT to store a snapshot, USE for the worker to read it. A same-name Source not created by that user, or one that is not private, active and managed-only, is never written to (409 launch_code_source_conflict). The snapshot stays until you delete it or the Source.
  • The submitted job carries a job-code read ticket in its arguments, which the control plane gives only to the worker that claims it. DreamLake receipts and the control plane's invocation read APIs never return it. When the worker runs the job, it exchanges the ticket at POST /internal/launch-code/read-capability for a 5-minute download of exactly that object, so a launch can wait in the queue for hours. Each exchange re-checks the submitter's lakeshore:use and Source access, that the mount still exists, and that the job is running on a worker, not cancelled, and unchanged since submission. The ticket stops working seven days after submission. The worker verifies the full sha256 before unpacking.
  • Turning the flag off stops new Source-staged launches only; jobs already queued can still fetch their snapshot. There is no fallback to code-staging if Source storage fails. Repeating a launch with the same invocation ID replays it only if the queue, script, arguments and snapshot are identical; otherwise it is 409 invocation_conflict.

See the CLI connection reference for commands and compatibility requirements. The remaining mount instructions are the retained compatibility workflow.

Separate fabric, runtime and queue access

Fabric — Add Host: use host enrollment to establish machine identity, connectivity and management, including terminals and remote commands. No task queue or queue mount is required. Current enrollment requires an authorized execution connection. Create or select it on the owner's Lakeshores page with the server URL, remote server namespace and a credential explicitly authorized to mint Host enrollment tokens. An ordinary queue-access token is not sufficient evidence of that authority.

Runtime — run a worker: configure a task-consumer process on the machine to serve the intended queue. See queues, worker lifecycle and the Nymph daemon reference. Verify worker readiness and a completed task. The same host can support development and worker processes; enrollment alone does not configure queue consumption.

Client queue access — queue mount: the steps below save queue access in DreamLake. A mount neither registers a host nor starts a worker. Connection health is not proof of enrollment authority or task execution.

Connection type (legacy), authMode: token and an online connection check do not establish enrollment-mint permission. Check the deployed service's supported authorization contract with its administrator. A namespace token needs explicit Host enrollment authority where delegated enrollment is supported; older servers may allow only admin minting. Do not distribute a global admin credential to members or substitute it into a personal connection. A denied upstream mint can surface as a downstream enrollment failure even after connection validation.

For a pending enrollment, preserve its connection ID, server namespace, host name and machine identity while an administrator reconciles authorization and operation state. Renaming or recreating resources does not grant the missing permission.

The execution connection's DreamLake owner must match the namespace at the start of the host name: alice/dev/laptop needs an alice-owned connection, while an acme org connection is used for authorized acme/... hosts. The remote Lakeshore namespace is a separate value. Supply the connection ID with dl hosts enroll --lakeshore-id when selecting explicitly.

A queue-only connection cannot authorize host enrollment, even when online: it returns HTTP 409 queue_mount_execution_unsupported. A connection ID owned by another DreamLake namespace can return lakeshore_not_found. Preserve the intended host ownership when correcting the connection.

The remaining steps cover optional queue access, not development-machine registration.

1. Get an access token

Already have a token? Continue to step 2.

For server administrators: after logging the Lakeshore CLI into the intended server and namespace, mint a separate token for DreamLake:

bash
lakeshore auth status
export LAKESHORE_ADMIN_TOKEN='your-server-admin-token'
lakeshore admin tokens create dreamlake-mount
unset LAKESHORE_ADMIN_TOKEN

Use the same admin token configured on the server. Save the printed client token; it is shown once. Keep LAKESHORE_URL unset so the CLI uses the server and namespace from your saved login.

2. Connect the server

With the DreamLake CLI installed:

bash
dl login
dl queues mount --uri https://lakeshore.example.com --prefix lab --lakeshore-namespace lab

Replace the URL and Lakeshore namespace with the values from your administrator. Paste the access token at the hidden prompt. DreamLake verifies the connection and saves the credential in Vault.

--prefix lab is the connection's name in DreamLake. --lakeshore-namespace lab is the namespace on the server; the two names can differ.

The queue mount belongs to a personal DreamLake namespace. Use --namespace your-personal-namespace explicitly when needed. Current queue mounts do not support organization owners; an organization Vault does not change this restriction. Use the unified connection API above for new organization connections once deployed.

3. Check the connection

bash
dl queues mounts
dl queues inspect lab
dl queues list lab
dl queues workers lab
dl queues jobs lab

inspect should report a healthy connection. Empty queues, workers, and jobs are normal before you enroll workers or submit work.

Disconnect later

bash
dl queues unmount lab

This removes the DreamLake connection. It retains Vault entries and leaves the server running.

Rollout of encrypted queue connections

New queue-only creation uses POST /namespaces/:slug/lakeshores/connect-v1 after capability discovery. It never falls back to the legacy management create endpoint. The application defaults creation to disabled. The standard production EKS deployment automatically upgrades all API readers, verifies the exact image and old-Pod drain, then enables LAKESHORE_ENCRYPTED_QUEUES_ENABLED=true and verifies the activation rollout. Subsequent compatible releases preserve an already-verified enabled fleet. No separate manual activation is needed for that workflow; other deployment targets must verify their reader fleet before enabling creation. This switch does not disable existing connections or the old queues mount API. An old backend cannot enforce the policy on new encrypted queue records, so rollback below this version requires first disabling those records under a recovery plan; switching off creation alone does not make such a rollback safe.