EC2 Provider

--launcher EC2 provisions AWS instances. The control plane owns the AWS SDK client and the decrypted credentials — the CLI never talks to EC2 directly. The default dispatch is daemon: the instance's userdata installs a nymph daemon that long-polls for work.

Phase	What happens
Provision	`RunInstances` via `@aws-sdk/client-ec2`, with your script base64-encoded into `UserData`.
Launch	Whatever the userdata script does — normally install and start the daemon.
Default dispatch	`daemon`

Where the launch happens

There are two launches here, and only the first one touches credentials.

	Who acts	Credentials
1. Provision the box	control plane calls `RunInstances`	AWS keys, held by the control plane
2. Get work to it	the nymph long-polls outbound	none — the daemon's own bearer token

Step 2 is the same on every launcher. Step 1 is where EC2 differs from SLURM and Kube: on those two a nymph can do the submitting itself (runner/slurm.rs, runner/kube.rs), so the credential stays inside the cluster. There is no ec2 runner in nymph — runner/mod.rs registers process, docker, gvisor, slurm, kube, exec, and mock, and nothing else. So provisioning an instance is something only the control plane can do.

The control plane holds long-lived AWS keys

aws_credentials resolves from the Secret store into an { accessKeyId, secretAccessKey } pair inside the control-plane process. The CLI genuinely never sees them — but "the CLI never sees them" is a weaker claim than "they never leave your account," and only the first one is true today. Scope the IAM user narrowly; see Setting up a dedicated IAM user.

Kwargs

These are the keys the EC2 launcher reads out of the merged provider + RunConfig kwargs. Everything else round-trips on the row but does not reach the SDK call.

Field	Where it usually lives	Meaning
`region`	provider	AWS region for the SDK client.
`aws_credentials`	provider	`{ accessKeyId, secretAccessKey }`. Omit to fall back to the SDK's default credential chain on the server.
`key_name`	provider	EC2 key-pair name → `KeyName`.
`security_group`	provider	One security group id → `SecurityGroupIds`. Use `security_group_ids` for a list.
`subnet_id`	provider	→ `SubnetId`.
`iam_instance_profile_arn`	provider	→ `IamInstanceProfile.Arn`.
`image_id` (or `ami`, or `ImageId`)	mode	Required. The launch throws without it.
`instance_type` (or `InstanceType`)	mode	Defaults to `t3.micro`.

Spot, AZ pinning, and DryRun are not wired

The launcher builds a fixed RunInstances parameter set. There is no spot_price, no availability_zone, and no dry handling — setting those kwargs stores them on the row and changes nothing about the call.

Every instance is tagged lakeshore=true, namespace=<ns>, and provider=<name>, plus any tags you pass on the launch body. User tags that collide with those three reserved keys are dropped rather than duplicated, which EC2 would reject.

Seeding from an AWS profile

If the credentials already live in ~/.aws/credentials + ~/.aws/config, skip the typing:

bash

lakeshore aws discover                # what the CLI can see locally
lakeshore providers add aws-dev --from-aws-profile sandbox

--from-aws-profile <name> seeds aws_credentials.accessKeyId, aws_credentials.secretAccessKey, aws_credentials.sessionToken (when present), and region (when the profile or its config section sets one), and infers --launcher EC2. Layer --kwarg on top for the rest:

bash

lakeshore providers add aws-dev \
  --from-aws-profile sandbox \
  --kwarg key_name=lakeshore-ops \
  --kwarg security_group=sg-0a1b2c3d4e5f \
  --kwarg iam_instance_profile_arn='arn:aws:iam::…:instance-profile/lakeshore-worker'

SSO profiles are rejected. Any sso_* key in either file flags the profile, and the launcher needs static keys. Run aws configure --profile <name>, or pass the pair explicitly with --kwarg aws_credentials.accessKeyId=… --kwarg aws_credentials.secretAccessKey=….

Tab completion on --from-aws-profile <TAB> lists the local profile names — see Completion.

Seeded credentials are stored inline

--from-aws-profile writes the keys straight into the provider kwargs. Fine for a scratch provider; move them into a Secret before the provider is shared.

bash

# The aws_keypair plaintext format is exactly `<access_key_id>:<secret_access_key>`.
printf '%s:%s' AKIA… wJal… | lakeshore secrets add aws-dev-keys --kind aws_keypair
lakeshore providers update aws-dev --kwarg aws_credentials='{"$secret":"aws-dev-keys"}'

The launcher recognises aws_keypair secrets specially: it splits the plaintext on the first : and substitutes { accessKeyId, secretAccessKey }. See Secrets.

Example

.dreamrc tab: The same shape in a local .dreamrc:

bash

lakeshore providers add aws-us-east --launcher EC2 \
  --kwarg region=us-east-1 \
  --kwarg key_name=lakeshore-ops \
  --kwarg security_group=sg-0a1b2c3d4e5f \
  --kwarg iam_instance_profile_arn='arn:aws:iam::123456789012:instance-profile/lakeshore-worker'

lakeshore modes add gpu \
  --field provider=aws-us-east \
  --field instance_type=g5.xlarge \
  --field image_id=ami-0abcdef1234567890 \
  --field runner=docker \
  --field image=ghcr.io/lakeshore-py/cuda12.4-pytorch:2.4 \
  --field resources.gpu=1

.dreamrcyaml

providers:
  aws-us-east: !providers.EC2
    region: us-east-1
    key_name: lakeshore-ops
    security_group: sg-0a1b2c3d4e5f
    iam_instance_profile_arn: arn:aws:iam::123456789012:instance-profile/lakeshore-worker

modes:
  gpu:
    provider: aws-us-east
    instance_type: g5.xlarge
    image_id: ami-0abcdef1234567890
    runner: docker
    image: ghcr.io/lakeshore-py/cuda12.4-pytorch:2.4
    resources: { gpu: 1 }

Server-side routes

Verb	Path	Result
`POST`	`/v1/namespaces/:ns/providers/:name/launch`	202 `{ instanceId, state, providerName, launchedAt }`
`GET`	`/v1/namespaces/:ns/providers/:name/instances`	Instances tagged `lakeshore=true` for this namespace + provider
`DELETE`	`/v1/namespaces/:ns/providers/:name/instances/:id`	204

launch requires a non-empty script (a complete bash userdata blob, capped at 256 KiB) and takes optional runConfigKwargs and tags. The launcher base64-encodes the script and hands it to RunInstances verbatim — it never templates or interprets it.

instanceId is the EC2 instance id (i-0abc1234). state is normalized from the SDK's State.Name: pending → starting, running → running, shutting-down/stopping → stopping, stopped/terminated/terminating → stopped, anything else unknown. At launch time it is almost always starting — poll instances to watch it become running.

AWS errors on a known list (UnauthorizedOperation, AccessDenied, InvalidAMIID.NotFound, InsufficientInstanceCapacity, and friends) come back as 502 with { error, awsCode, message }; anything else is a 500, on the theory that it is our bug.

Launching a daemon

For the daemon dispatch path you normally do not call /launch yourself. POST /v1/namespaces/:ns/daemons/launch — the route behind lakeshore daemon launch — renders the nymph bootstrap script (download the binary, drop the unit, start it, point it at the control plane) and passes it to this same launcher as the userdata:

bash

lakeshore daemon launch --provider aws-us-east --name gpu --count 4 \
  --queue training --runner docker --keep-alive 1800 --wait

The bootstrap script's control-plane URL comes from LAKESHORE_PUBLIC_URL on the server (falling back to LAKESHORE_SERVER_URL, then LAKESHORE_URL, then localhost), so set that on a deployed control plane or the daemons will try to reach http://localhost:8080.

Smoke test

bash

lakeshore providers test aws-us-east
lakeshore providers instances aws-us-east

providers test posts a launch through the control plane. Streaming stdout from the instance back to the CLI is not implemented — check the instance over SSH, via lakeshore daemon launch-log <id> (EC2 serial console), or by polling instances.

Setting up a dedicated IAM user

For a fresh AWS account, the one-time path is: sign in as root → IAM → create a programmatic user → attach a policy → create an access key → wire it locally → register with Lakeshore.

Sign in as root at https://signin.aws.amazon.com/ (with MFA).
IAM → Users → Create user. Leave console access unchecked — programmatic only.
Permissions → Attach policies directly. AdministratorAccess is the blunt option and covers everything the launchers touch. The minimum for EC2 alone is ec2:RunInstances, ec2:Describe*, ec2:CreateTags, ec2:TerminateInstances, plus iam:PassRole if you attach an instance profile.
Security credentials → Create access key → Command Line Interface.
On the success page, wire it up before navigating away — the secret is shown exactly once.

bash

aws configure --profile lakeshore-dev
aws sts get-caller-identity --profile lakeshore-dev    # should print the IAM ARN

lakeshore providers add aws-us-east --from-aws-profile lakeshore-dev

# Then move the inline credentials into a Secret (see the callout above).

Rotation

Static IAM keys never expire on their own. Rotate roughly every 90 days:

Create a second access key for the same user — AWS allows two active keys precisely so you can roll without downtime.
aws configure --profile lakeshore-dev with the new pair.
printf '%s:%s' <new-id> <new-secret> | lakeshore secrets rotate <secret-name>. Plaintext comes from --from-file <path> or piped stdin; there is no --from-file - form, so pipe it.
Smoke test: lakeshore providers test aws-us-east.
Only then deactivate the old key in the console, wait a day for stragglers to surface, and delete it.

Deactivate-then-delete is the point: a straggler on a deactivated key fails visibly and reversibly. Deleting outright makes it fail permanently and anonymously.

OIDC federation — not built

Static keys in the control-plane database do not scale to "run in the customer's AWS account." The clean answer is OIDC federation, the model GitHub Actions uses: the control plane becomes an identity provider, the customer creates an IAM role trusting it with sub/aud pinned to a namespace, and each launch calls sts:AssumeRoleWithWebIdentity for short-lived credentials. No keys at rest, per-launch audit.

None of that exists yet. It would need a JWKS + discovery endpoint on the control plane with signing-key rotation, a aws_role_arn kwarg mutually exclusive with aws_credentials, and an AssumeRoleWithWebIdentity call in the EC2 launcher instead of the current credential passthrough.

AWS IAM Identity Center is not an alternative: SSO refresh is fundamentally interactive, and a server has no browser session to renew from. OIDC sidesteps that by making the control plane the issuer rather than a consumer.