DreamLake

EC2 Provider

The managed aws-ec2 provider executes cloud operations on the user-configured launch server using that server's instance role. Use CLI 0.71.0 or later and the matching backend launch-server implementation. Verify deployment and live readiness before launching; acceptance of this route remains pending in this guide.

Configure a managed EC2 provider

Enroll the selected server as a Nymph Host in the provider's namespace. Install Python 3 and boto3, enable IMDSv2, and give its dedicated instance role the intended EC2 permissions and network access. The adapter verifies its STS account and role before calling EC2; worker instance profiles are separate identities.

Save a provider declaration such as aws-east.json:

json
{
  "version": 1,
  "name": "aws-east",
  "kind": "aws-ec2",
  "config": {
    "region": "us-east-1",
    "accountId": "123456789012",
    "launchServer": {
      "enrollmentId": "1123456789abcdef01234567",
      "expectedRoleArn": "arn:aws:iam::123456789012:role/user-launch"
    }
  }
}

Save the intended image and machine shape in a JSON Layer file, provider.jsonl. Replace the image ID with an AMI in the selected region and configure the intended subnet and security groups before a real launch.

json
{"instanceType":"t3.large","imageId":"ami-0123456789abcdef0","volumeGiB":100}
bash
dl providers register team --file aws-east.json --launch provider.jsonl --request-id aws-east-01
dl providers preflight team/PROVIDER_ID --json
dl providers instances launch team/PROVIDER_ID --count 1 --request-id worker-01 --json
dl providers instances list team/PROVIDER_ID --json

Registration and preflight do not call EC2 or prove server readiness. For an existing provider, use the conditional launch-server binding with its current revision to preserve its ID and launch layers.

The backend reserves the original launch operation before cloud effects and sends a fixed adapter through authenticated Nymph exec. EC2 launch, start, stop and terminate all require the configured server; missing or invalid authority never falls back to service credentials. Lifecycle actions check instance ownership, and each distinct request gets its own ownership observation.

Retain request IDs across uncertain responses and inspect dl providers operations find team --request-id worker-01 --json. A recoveryOperationId identifies an additional remote-execution receipt for dl providers operations show team/OPERATION_ID --json. Pending or unknown dispatch is reconciled without automatically resubmitting the mutation. After a definitive failed rejection, repair the cause, verify no resource was created, and deliberately use a new ID. See permission repair.

Provisioning an instance does not by itself establish Nymph or terminal readiness. The separate protected Host bootstrap option supports a single instance; check enrollment and Host status before treating it as ready for jobs or SSH. See the CLI provider reference for enrollment options and lifecycle commands.

Legacy implementation reference

--launcher EC2 implements AWS instance provisioning. The legacy control plane owns the AWS SDK client and the decrypted credentials — the CLI never talks to EC2 directly. The default dispatch is daemon: the instance's userdata installs a nymph daemon that long-polls for work.

PhaseWhat happens
ProvisionRunInstances via @aws-sdk/client-ec2, with your script base64-encoded into UserData.
LaunchWhatever the userdata script does — normally install and start the daemon.
Default dispatchdaemon

Where the legacy launch happens

There are two launches here, and only the first one touches credentials.

Who actsCredentials
1. Provision the boxcontrol plane calls RunInstancesAWS keys, held by the control plane
2. Get work to itthe nymph long-polls outboundnone — the daemon's own bearer token

Step 2 is the same on every launcher. Step 1 is where EC2 differs from SLURM and Kube: on those two a nymph can do the submitting itself (runner/slurm.rs, runner/kube.rs), so the credential stays inside the cluster. There is no ec2 runner in nymph — runner/mod.rs registers process, docker, gvisor, slurm, kube, exec, and mock, and nothing else. This describes the legacy implementation, not authority to provision from the control plane.

The control plane holds long-lived AWS keys

aws_credentials resolves from the Secret store into an { accessKeyId, secretAccessKey } pair inside the control-plane process. The CLI genuinely never sees them — but "the CLI never sees them" is a weaker claim than "they never leave your account," and only the first one describes this legacy route. Do not provision a new IAM user or broaden service IAM to keep this path working. Use the required launch-server boundary for new launches.

Kwargs

These are the keys the EC2 launcher reads out of the merged provider + RunConfig kwargs. Everything else round-trips on the row but does not reach the SDK call.

FieldWhere it usually livesMeaning
regionproviderAWS region for the SDK client.
aws_credentialsproviderLegacy { accessKeyId, secretAccessKey } input. The implementation can fall back to the server SDK chain when omitted; that fallback is prohibited for new launches.
key_nameproviderEC2 key-pair name → KeyName.
security_groupproviderOne security group id → SecurityGroupIds. Use security_group_ids for a list.
subnet_idprovider→ SubnetId.
iam_instance_profile_arnprovider→ IamInstanceProfile.Arn.
image_id (or ami, or ImageId)modeRequired. The launch throws without it.
instance_type (or InstanceType)modeDefaults to t3.micro.
Spot, AZ pinning, and DryRun are not wired

The launcher builds a fixed RunInstances parameter set. There is no spot_price, no availability_zone, and no dry handling — setting those kwargs stores them on the row and changes nothing about the call.

Every instance is tagged lakeshore=true, namespace=<ns>, and provider=<name>, plus any tags you pass on the launch body. User tags that collide with those three reserved keys are dropped rather than duplicated, which EC2 would reject.

Seeding from an AWS profile

The following documents existing import behavior. Do not seed an operator profile or Vault IAM user into a provider as a launch-server substitute.

The legacy importer reads ~/.aws/credentials and ~/.aws/config:

bash
lakeshore aws discover                # what the CLI can see locally
lakeshore providers add aws-dev --from-aws-profile sandbox

--from-aws-profile <name> seeds aws_credentials.accessKeyId, aws_credentials.secretAccessKey, aws_credentials.sessionToken (when present), and region (when the profile or its config section sets one), and infers --launcher EC2. Layer --kwarg on top for the rest:

bash
lakeshore providers add aws-dev \
  --from-aws-profile sandbox \
  --kwarg key_name=lakeshore-ops \
  --kwarg security_group=sg-0a1b2c3d4e5f \
  --kwarg iam_instance_profile_arn='arn:aws:iam::…:instance-profile/lakeshore-worker'

SSO profiles are rejected. Any sso_* key in either file flags the profile, and the legacy launcher needs static keys. This limitation does not authorize creating or copying static keys to bypass the required launch server.

Tab completion on --from-aws-profile <TAB> lists the local profile names — see Completion.

Seeded credentials are stored inline

--from-aws-profile writes the keys straight into the provider kwargs. Moving them into a Secret changes storage, not the launch execution boundary. Neither inline nor Secret-backed credentials authorize new control-plane launches.

bash
# The aws_keypair plaintext format is exactly `<access_key_id>:<secret_access_key>`.
printf '%s:%s' AKIA… wJal… | lakeshore secrets add aws-dev-keys --kind aws_keypair
lakeshore providers update aws-dev --kwarg aws_credentials='{"$secret":"aws-dev-keys"}'

The launcher recognises aws_keypair secrets specially: it splits the plaintext on the first : and substitutes { accessKeyId, secretAccessKey }. See Secrets.

Example

.dreamrc tab: The same shape in a local .dreamrc:

bash
lakeshore providers add aws-us-east --launcher EC2 \
  --kwarg region=us-east-1 \
  --kwarg key_name=lakeshore-ops \
  --kwarg security_group=sg-0a1b2c3d4e5f \
  --kwarg iam_instance_profile_arn='arn:aws:iam::123456789012:instance-profile/lakeshore-worker'

lakeshore modes add gpu \
  --field provider=aws-us-east \
  --field instance_type=g5.xlarge \
  --field image_id=ami-0abcdef1234567890 \
  --field runner=docker \
  --field image=ghcr.io/lakeshore-py/cuda12.4-pytorch:2.4 \
  --field resources.gpu=1

Server-side routes

VerbPathResult
POST/v1/namespaces/:ns/providers/:name/launch202 { instanceId, state, providerName, launchedAt }
GET/v1/namespaces/:ns/providers/:name/instancesInstances tagged lakeshore=true for this namespace + provider
DELETE/v1/namespaces/:ns/providers/:name/instances/:id204

launch requires a non-empty script (a complete bash userdata blob, capped at 256 KiB) and takes optional runConfigKwargs and tags. The launcher base64-encodes the script and hands it to RunInstances verbatim — it never templates or interprets it.

instanceId is the EC2 instance id (i-0abc1234). state is normalized from the SDK's State.Name: pending → starting, running → running, shutting-down/stopping → stopping, stopped/terminated/terminating → stopped, anything else unknown. At launch time it is almost always starting — poll instances to watch it become running.

AWS errors on a known list (UnauthorizedOperation, AccessDenied, InvalidAMIID.NotFound, InsufficientInstanceCapacity, and friends) come back as 502 with { error, awsCode, message }; anything else is a 500, on the theory that it is our bug.

Launching a daemon

For the daemon dispatch path you normally do not call /launch yourself. POST /v1/namespaces/:ns/daemons/launch — the route behind lakeshore daemon launch — renders the nymph bootstrap script (download the binary, drop the unit, start it, point it at the control plane) and passes it to this same launcher as the userdata:

bash
lakeshore daemon launch --provider aws-us-east --name gpu --count 4 \
  --queue training --runner docker --keep-alive 1800 --wait

The bootstrap script's control-plane URL comes from LAKESHORE_PUBLIC_URL on the server (falling back to LAKESHORE_SERVER_URL, then LAKESHORE_URL, then localhost), so set that on a deployed control plane or the daemons will try to reach http://localhost:8080.

Smoke test

bash
lakeshore providers test aws-us-east
lakeshore providers instances aws-us-east

providers test posts a launch through the control plane. Streaming stdout from the instance back to the CLI is not implemented — check the instance over SSH, via lakeshore daemon launch-log <id> (EC2 serial console), or by polling instances.

Configure the launch server's IAM role

The user configures the launch server and grants its role only the intended provider operations and resource scope. Verify its actual runtime identity before execution. Keep that role distinct from the control-plane service role and from any instance profile attached to a worker. An instance-profile ARN in the launch payload does not select or authenticate the launch server.

Do not create operator access keys, widen the control-plane role, or move launch execution into the hosted backend to work around a missing server path. No legacy configuration here selects the managed adapter. Use the explicit config.launchServer binding above and verify the matching deployment before launching.

The earlier proposal for the control plane to assume customer roles through OIDC is not the accepted execution architecture. Short-lived credentials alone do not satisfy the user-configured launch-server boundary.