EC2 Provider
The managed aws-ec2 provider executes cloud operations on the
user-configured launch server
using that server's instance role. Use CLI 0.71.0 or later and the matching
backend launch-server implementation. Verify deployment and live readiness
before launching; acceptance of this route remains pending in this guide.
Configure a managed EC2 provider
Enroll the selected server as a Nymph Host in the provider's namespace. Install
Python 3 and boto3, enable IMDSv2, and give its dedicated instance role the
intended EC2 permissions and network access. The adapter verifies its STS account
and role before calling EC2; worker instance profiles are separate identities.
Save a provider declaration such as aws-east.json:
Save the intended image and machine shape in a JSON Layer file, provider.jsonl.
Replace the image ID with an AMI in the selected region and configure the
intended subnet and security groups before a real launch.
Registration and preflight do not call EC2 or prove server readiness. For an existing provider, use the conditional launch-server binding with its current revision to preserve its ID and launch layers.
The backend reserves the original launch operation before cloud effects and sends a fixed adapter through authenticated Nymph exec. EC2 launch, start, stop and terminate all require the configured server; missing or invalid authority never falls back to service credentials. Lifecycle actions check instance ownership, and each distinct request gets its own ownership observation.
Retain request IDs across uncertain responses and inspect
dl providers operations find team --request-id worker-01 --json.
A recoveryOperationId identifies an additional remote-execution receipt for
dl providers operations show team/OPERATION_ID --json. Pending or
unknown dispatch is reconciled without automatically resubmitting the mutation.
After a definitive failed rejection, repair the cause, verify no resource was
created, and deliberately use a new ID. See permission repair.
Provisioning an instance does not by itself establish Nymph or terminal readiness. The separate protected Host bootstrap option supports a single instance; check enrollment and Host status before treating it as ready for jobs or SSH. See the CLI provider reference for enrollment options and lifecycle commands.
Legacy implementation reference
--launcher EC2 implements AWS instance provisioning. The legacy control plane owns the
AWS SDK client and the decrypted credentials — the CLI never talks to
EC2 directly. The default dispatch is daemon: the instance's userdata
installs a nymph daemon that long-polls for work.
| Phase | What happens |
|---|---|
| Provision | RunInstances via @aws-sdk/client-ec2, with your script base64-encoded into UserData. |
| Launch | Whatever the userdata script does — normally install and start the daemon. |
| Default dispatch | daemon |
Where the legacy launch happens
There are two launches here, and only the first one touches credentials.
| Who acts | Credentials | |
|---|---|---|
| 1. Provision the box | control plane calls RunInstances | AWS keys, held by the control plane |
| 2. Get work to it | the nymph long-polls outbound | none — the daemon's own bearer token |
Step 2 is the same on every launcher. Step 1 is where EC2 differs from
SLURM and Kube: on those two a nymph can do the submitting itself
(runner/slurm.rs, runner/kube.rs), so the credential stays inside the
cluster. There is no ec2 runner in nymph — runner/mod.rs registers
process, docker, gvisor, slurm, kube, exec, and mock, and
nothing else. This describes the legacy implementation, not authority to
provision from the control plane.
aws_credentials resolves from the Secret store into an { accessKeyId, secretAccessKey } pair inside the control-plane process. The CLI genuinely
never sees them — but "the CLI never sees them" is a weaker claim than
"they never leave your account," and only the first one describes this legacy route.
Do not provision a new IAM user or broaden service IAM to keep this path
working. Use the required launch-server boundary for new launches.
Kwargs
These are the keys the EC2 launcher reads out of the merged
provider + RunConfig kwargs. Everything else round-trips on the row but
does not reach the SDK call.
| Field | Where it usually lives | Meaning |
|---|---|---|
region | provider | AWS region for the SDK client. |
aws_credentials | provider | Legacy { accessKeyId, secretAccessKey } input. The implementation can fall back to the server SDK chain when omitted; that fallback is prohibited for new launches. |
key_name | provider | EC2 key-pair name → KeyName. |
security_group | provider | One security group id → SecurityGroupIds. Use security_group_ids for a list. |
subnet_id | provider | → SubnetId. |
iam_instance_profile_arn | provider | → IamInstanceProfile.Arn. |
image_id (or ami, or ImageId) | mode | Required. The launch throws without it. |
instance_type (or InstanceType) | mode | Defaults to t3.micro. |
The launcher builds a fixed RunInstances parameter set. There is no
spot_price, no availability_zone, and no dry handling — setting
those kwargs stores them on the row and changes nothing about the call.
Every instance is tagged lakeshore=true, namespace=<ns>, and
provider=<name>, plus any tags you pass on the launch body. User tags
that collide with those three reserved keys are dropped rather than
duplicated, which EC2 would reject.
Seeding from an AWS profile
The following documents existing import behavior. Do not seed an operator profile or Vault IAM user into a provider as a launch-server substitute.
The legacy importer reads ~/.aws/credentials and ~/.aws/config:
--from-aws-profile <name> seeds aws_credentials.accessKeyId,
aws_credentials.secretAccessKey, aws_credentials.sessionToken (when
present), and region (when the profile or its config section sets one),
and infers --launcher EC2. Layer --kwarg on top for the rest:
SSO profiles are rejected. Any sso_* key in either file flags the
profile, and the legacy launcher needs static keys. This limitation does not
authorize creating or copying static keys to bypass the required launch server.
Tab completion on --from-aws-profile <TAB> lists the local profile
names — see Completion.
--from-aws-profile writes the keys straight into the provider kwargs.
Moving them into a Secret changes storage, not the launch execution boundary.
Neither inline nor Secret-backed credentials authorize new control-plane launches.
The launcher recognises aws_keypair secrets specially: it splits the
plaintext on the first : and substitutes
{ accessKeyId, secretAccessKey }. See
Secrets.
Example
.dreamrc tab: The same shape in a local .dreamrc:
lakeshore providers add aws-us-east --launcher EC2 \
--kwarg region=us-east-1 \
--kwarg key_name=lakeshore-ops \
--kwarg security_group=sg-0a1b2c3d4e5f \
--kwarg iam_instance_profile_arn='arn:aws:iam::123456789012:instance-profile/lakeshore-worker'
lakeshore modes add gpu \
--field provider=aws-us-east \
--field instance_type=g5.xlarge \
--field image_id=ami-0abcdef1234567890 \
--field runner=docker \
--field image=ghcr.io/lakeshore-py/cuda12.4-pytorch:2.4 \
--field resources.gpu=1providers:
aws-us-east: !providers.EC2
region: us-east-1
key_name: lakeshore-ops
security_group: sg-0a1b2c3d4e5f
iam_instance_profile_arn: arn:aws:iam::123456789012:instance-profile/lakeshore-worker
modes:
gpu:
provider: aws-us-east
instance_type: g5.xlarge
image_id: ami-0abcdef1234567890
runner: docker
image: ghcr.io/lakeshore-py/cuda12.4-pytorch:2.4
resources: { gpu: 1 }Server-side routes
| Verb | Path | Result |
|---|---|---|
POST | /v1/namespaces/:ns/providers/:name/launch | 202 { instanceId, state, providerName, launchedAt } |
GET | /v1/namespaces/:ns/providers/:name/instances | Instances tagged lakeshore=true for this namespace + provider |
DELETE | /v1/namespaces/:ns/providers/:name/instances/:id | 204 |
launch requires a non-empty script (a complete bash userdata blob,
capped at 256 KiB) and takes optional runConfigKwargs and tags. The
launcher base64-encodes the script and hands it to RunInstances
verbatim — it never templates or interprets it.
instanceId is the EC2 instance id (i-0abc1234). state is normalized
from the SDK's State.Name: pending → starting, running → running,
shutting-down/stopping → stopping, stopped/terminated/terminating → stopped, anything else unknown. At launch time it is almost always
starting — poll instances to watch it become running.
AWS errors on a known list (UnauthorizedOperation, AccessDenied,
InvalidAMIID.NotFound, InsufficientInstanceCapacity, and friends)
come back as 502 with { error, awsCode, message }; anything else is a
500, on the theory that it is our bug.
Launching a daemon
For the daemon dispatch path you normally do not call /launch
yourself. POST /v1/namespaces/:ns/daemons/launch — the route behind
lakeshore daemon launch — renders the nymph bootstrap script (download
the binary, drop the unit, start it, point it at the control plane) and
passes it to this same launcher as the userdata:
The bootstrap script's control-plane URL comes from
LAKESHORE_PUBLIC_URL on the server (falling back to
LAKESHORE_SERVER_URL, then LAKESHORE_URL, then localhost), so set
that on a deployed control plane or the daemons will try to reach
http://localhost:8080.
Smoke test
providers test posts a launch through the control plane. Streaming
stdout from the instance back to the CLI is not implemented — check the
instance over SSH, via lakeshore daemon launch-log <id> (EC2 serial
console), or by polling instances.
Configure the launch server's IAM role
The user configures the launch server and grants its role only the intended provider operations and resource scope. Verify its actual runtime identity before execution. Keep that role distinct from the control-plane service role and from any instance profile attached to a worker. An instance-profile ARN in the launch payload does not select or authenticate the launch server.
Do not create operator access keys, widen the control-plane role, or move launch
execution into the hosted backend to work around a missing server path. No
legacy configuration here selects the managed adapter. Use the explicit
config.launchServer binding above and verify the matching deployment before
launching.
The earlier proposal for the control plane to assume customer roles through OIDC is not the accepted execution architecture. Short-lived credentials alone do not satisfy the user-configured launch-server boundary.