DreamLake

Connections

A Connection is a credential DreamLake holds for a namespace to reach another system: an S3-compatible bucket, a Hugging Face account, Dropbox. A Source with an external layer reads through one. DreamLake stores the credential encrypted and never shows it again — not in the web app, not in the CLI, not to the assistant.

A Connection is not the same thing as:

  • a Lakeshore connection — a link to a Lakeshore service (its URL and namespace) used for compute;
  • host enrollment — how a computer joins DreamLake with Nymph. Enrolling a machine does not need a Connection or an S3 bucket (Connect to DreamLake).

Who can manage them

Only the namespace's credential managers — the owner of a personal namespace, or an owner of an organization — can see or change its Connections. Anyone else sees nothing: an id they are not allowed to read looks exactly like one that does not exist.

WhoDiscover and listCreate or reconnect with a secretCredential-free create
You, signed in (web, CLI, SDKs)YesYesYes
The DreamLake assistant, in your chatYesYou enter the secret in the browser; the assistant never sees itYes, with your approval of writes
Background or delegated agent runsNoNoNo
Service accounts and machine tokensNoNoNo

Connections cannot be granted to an agent service account; Project and host grants do not include them.

What you can connect

CLI and SDK support depends on the installed release. Check dl connection --help and the documentation for your SDK version before using the examples below; a client method alone does not establish support in the connected server.

What a server accepts is declared by that server, and can differ between deployments. Ask it: the web form reads the list, and so do dl connection connectors, client.connections.connectors(namespace) in Python and connections(namespace).connectors() in TypeScript. Each way in has a mode:

ModeYou provideExample
Typed fieldsA credential (some fields secret) and non-secret configurationS3 access key; S3 role; Hugging Face token
Browser sign-in (OAuth)Nothing typed — you approve at the providerHugging Face, Dropbox
NoneOnly configuration, no credentialPublic GitHub repositories, public Hugging Face

For S3, there are two ways in. Both ask for a profile — for example Amazon S3, Backblaze B2 or Cloudflare R2; the profiles offered are the ones this deployment has configured — and, for some profiles, a region or account id.

  • Access key — an access key id and secret access key, and an optional session token. Works with every S3 profile the server offers.
  • Role — Amazon S3 only. DreamLake assumes an IAM role in your AWS account instead of holding a long-lived key. You give it the role ARN (stored as readable configuration) and an external id (stored as a secret). Before creating the Connection, the role's trust policy must allow DreamLake's AWS identity to assume it, with an sts:ExternalId condition equal to that external id, and the role needs read access to the buckets you will import. DreamLake does not display its AWS identity in the app; get it from whoever operates your DreamLake deployment. Backblaze B2, Cloudflare R2 and other S3-compatible profiles refuse the role way in.

Create one

In the web app, open Connections in your namespace (/<namespace>/connections), choose + new connection, pick the service and the way in, and fill in the form. DreamLake validates the credential first; if it is refused, no Connection is created. Check on a Connection then reports available when the provider accepted the credential — which proves the credential and the Connection's endpoint, not access to any particular bucket — or unverified when this way in has no check that works without a target. Whether a bucket and prefix can be read is decided when you import a Source.

From the CLI, pass secrets from a file or stdin, never as an argument:

bash
dl connection connectors
dl connection create --label "Team bucket" --connector s3 --auth-kind access-key \
  --credential-file credential.json --private-config-file config.json

For the role way in, the credential file holds {"externalId": "..."} and the configuration file holds {"profileId": "aws", "region": "us-east-1", "roleArn": "arn:aws:iam::123456789012:role/dreamlake-read"} with --auth-kind role. The CLI Connections guide covers every command, OAuth, reconnecting and deleting. The Python SDK has the same operations on client.connections.

By asking the assistant: say which storage you want to connect. The assistant reads what this server accepts, then gives you a link to the Connections page. That page shows the namespace and what will be created, with the service and way in already chosen; you enter the secret there, signed in. The link works only for you, only in that namespace, once, for 30 minutes. When you have finished, the assistant reads back the new Connection's id and status — never the credential. You can decline the request on that page, or ask the assistant to cancel it. Replacing an expired key works the same way, from the Connection's own page.

Never paste a credential into a chat; the assistant refuses tool calls that carry one.

Use it from a Source

Import a Source that reads through the Connection, in the web app's Source import or with:

bash
dl source import --connector s3 --connection <connection-id> \
  --target '{"bucket": "team-data", "prefix": "episodes/"}'

The Connection decides which endpoint the Source reads from; the target names only the bucket and prefix. A Connection that a Source still uses cannot be deleted — disable it to stop access, or detach the Source first.