Providers
Required launch authority
All workload and machine launches must execute through a launch server explicitly configured by the user. AWS launches use that server's appropriate IAM role. The user selects the server, account, region, permitted operations and resources; provider registration alone does not establish this execution authority.
Do not use or broaden the Lakeshore control-plane or other service IAM role for launches. Ambient service credentials, an operator Vault IAM user, and direct provider execution from the hosted backend are not substitutes for the launch server. The control plane coordinates requests; provider calls execute at the user-configured server boundary. Worker instance profiles and workload service accounts are separate identities and do not authorize the launch server.
Before a new launch, verify the selected server, its runtime identity and scope, and that the deployed path actually executes there. If that path is unavailable or unverified, stop the new launch or provisioning action. Do not fall back to a legacy route or extend service permissions. Preserve existing resources and unrelated service operation; read-only inspection and unrelated fixes may continue. This requirement does not authorize destroying or migrating resources.
Update launch-role permissions
An IAM role's permissions can change after creation. Attach a policy or edit a customer-managed or inline policy; changing permissions normally does not require replacing the launch server. AWS-managed policies cannot be edited, but a separate customer-managed policy can supply additional permissions. Allow IAM propagation and verify the actual operation from the configured server. See AWS's policy editing guide.
Give the role practical permissions for the provisioning workflows it owns, including networking, images and worker instance profiles. Keep its policy in the same infrastructure source as the server so routine updates remain reproducible. The launch role and the identity authorized to administer its policy are distinct: a denied operation does not itself authorize self-escalation or service-role reuse. An account credential explicitly authorized to bootstrap a launch server may create that server and role; subsequent launches must execute through the server.
When an agent encounters a permission limitation:
- Run
aws sts get-caller-identityon the selected launch server. Compare the account and assumed role with the configured binding. Record the denied action, resource, region and request ID without credential values. - Identify the enforcing policy. A missing identity-policy Allow differs from a
resource policy, permissions boundary, session policy, organization policy or
explicit Deny. Role trust controls who can assume a role;
iam:PassRolecontrols which role can be passed to a service. Kubernetes RBAC and network reachability are separate from AWS IAM. - Using the authorized policy-management identity, update the launch role's Terraform policy source, inspect the plan and apply it through its existing InfraStack. Keep unrelated roles and service infrastructure unchanged.
- Verify the previously denied action after propagation. Reconcile the original operation receipt before retrying a launch whose outcome is uncertain; retain its request ID and do not create a replacement request merely to bypass an ambiguous submission.
An added Allow cannot override an explicit Deny or exceed a permissions boundary. Use AWS's AccessDenied troubleshooting reference to locate the policy that actually limits the request. Do not respond to a denial by switching to control-plane credentials.
Managed launch-server workflow
The managed DreamLake API implements server-bound aws-ec2 and k8s providers.
Use CLI 0.71.0 or later with a backend containing the launch-server routing
and receipt fixes (implementation,
request isolation).
Source support does not establish deployment or live acceptance. Verify both
installed CLI and deployed backend before running these examples; acceptance
through this route is still pending in this guide.
- Provision a user-selected launch server with its own instance role and enroll
it as a Nymph Host in the provider's namespace. It needs Python 3,
boto3, IMDSv2 access and network access to the selected provider APIs. - Set the provider's
config.launchServerto the Host's enrollment ID and expected IAM role ARN, with the region and, optionally, matching account ID. - Register the provider and save its launch layer. For existing providers, use the conditional binding operation below to preserve their IDs, layers and resource ownership.
- Preflight the configuration, then verify actual server identity, permissions and network reachability. Registration and preflight do not make cloud calls or prove launch readiness.
- Submit a launch with a unique request ID; inspect its receipt and resource status. A receipt is not proof that the workload ran or a Host enrolled.
The backend authenticates the enrolled Host and sends a fixed provider adapter through Nymph exec. The adapter obtains instance-role credentials locally using IMDSv2 and checks the STS account and role against the binding before provider calls. It does not use ambient backend credentials or operator keys. Revoked or changed enrollment authority, missing bindings and mismatched roles reject the operation without a direct-execution fallback.
For an existing provider, first read its current revision, then bind it:
The equivalent API is POST /namespaces/:ns/lakeshore-providers/:id/launch-server
with {requestId, revision, region, launchServer: {enrollmentId, expectedRoleArn}}.
It checks ownership and revision, preserves launch layers, and increments the
provider revision. Kubernetes binding removes legacy credentialRef and
context settings. A conflicting revision requires a fresh read, not a blind
overwrite. Legacy unbound inventory can remain readable during migration; new
cloud launches and lifecycle writes require the server binding.
Recover an uncertain request
Remote dispatch can outlast an HTTP response. launch_server_operation_pending
or launch_server_outcome_unknown is a reason to inspect the existing receipt,
not submit replacement work. The CLI exposes recoveryOperationId when an
internal remote-execution receipt needs inspection:
For an EC2 instance launch (RunInstances) whose launch-server exec was
acknowledged but never read, operations show by the request's creator reads
that saved exec once through the provider's current binding and records its
terminal result. It needs the same provider update permission as the launch
itself. It never submits work, creates a bootstrap or calls the cloud, so it
still works after the original request can no longer be repeated. Other
launch-server operations (start, stop, terminate, Kubernetes) are not read;
they return as stored with reconcileError: launch_server_reconcile_unsupported.
status keeps its values;
outcome is unreconciled, unknown, rejected or committed. A finished
receipt also shows its fixed error code and, when AWS answered, a sanitized
diagnostic (code, operation, region, HTTP status, AWS request ID). rejected
means AWS refused the single attempt: a 4xx answer, or RunInstances with
InsufficientInstanceCapacity. Other server errors, timeouts and
IdempotentParameterMismatch stay unknown. A receipt that is still
unreconciled after the read carries reconcileError with the reason. Receipts
finished before this change may lack the AWS code; their outcome then stays
unknown.
Retain the original request ID and parameters while reconciling. Ambiguous mutating submissions are never automatically sent again. A definitive failed permission rejection remains failed after a policy repair: verify that no resource was created, then deliberately choose a new ID. See the CLI permission guide for bounded error diagnostics and recovery.
Legacy control-plane reference
The following sections describe existing lakeshore launcher records and
protocols. They are separate from the managed dreamlake API above and are not
a fallback for new cloud launches. Their stored credentials and direct SDK
execution do not satisfy the launch-server boundary.
A provider is the account or connection Lakeshore launches compute through — a
cloud credential, a cluster endpoint, a machine you already own. A queue points
at one via providerRef when it should grow its own workers.
Each provider picks exactly one launcher, and the control plane accepts
exactly five, spelled exactly like this. Entries carry a marker when they are not
yet finished: [ draft ] means designed but nothing built, [ dev ]
means you can configure it but it will not carry a workload end to end.
EC2, GCE and Kube are the three a queue can grow workers on. SSH registers and
runs from the CLI but has no server-side launch route; SLURM's sbatch submit
path is still a stub. A half-supported provider can exist at all because
Provider.kind is a free-form String rather than a constrained enum, and the
two lists in front of it disagree — registration accepts all five names, the
launch routes accept only three, and answer 422 for the other two.
Credentials never sit in the row as plaintext: kwargs reference them with a
{ "$secret": "<name>" } marker, and the secret is decrypted in memory inside
the launch handler only.
Each launcher's own fields, routes and setup live on its page above. The rest of this page is the cross-cutting machinery — how providers are registered, stored, merged and scoped, whichever launcher you picked.
Dispatch
Dispatch picks how work reaches the host:
direct— the launch is initiated per call from outside: an SSH session, ansbatchsubmission, akubectl apply. Nothing long-lived runs on the host.daemon— anymphdaemon on the host long-polls the control plane and pulls work. The daemon always initiates the connection, so this works behind NAT with no inbound port.
EC2 and GCE default to daemon; Kube, SSH and SLURM default to
direct. Set dispatch explicitly on a provider to override its default.
Registering a provider
Providers live on the control plane. Register them with the CLI:
Names must match ^[a-z0-9][a-z0-9-]*$. --kwarg is repeatable and dotted keys
nest (--kwarg aws_credentials.accessKeyId=AKIA… builds a nested object).
--kwargs-file <path> loads a YAML or JSON base that --kwarg then layers on
top.
All five launchers register through this one command shape — an SSH or SLURM provider is accepted and stored exactly like a cloud one. It is the launch side that is not there yet for those two.
Wire shape
The API body is { name, launcher, kwargs, dispatch? }. The database column
names differ, which matters when reading the Prisma schema:
| API field | DB column |
|---|---|
launcher | Provider.kind |
kwargs | Provider.config |
dispatch | Provider.config.dispatch |
Reads split dispatch back out and fill in the launcher default when it was
never set, so a GET always reports a concrete value.
Kwargs are an open bag — the control plane validates only that
{ "$secret": "<name>" } markers name a secret that exists in the namespace
(422 otherwise). Everything else round-trips verbatim, so a new launcher field
works without a schema change.
Editing
--kwarg deep-merges (key=null deletes a key); --kwargs-file replaces kwargs
wholesale. --launcher is immutable — supplying it is an error, so changing
launcher means remove and re-add.
The .dreamrc fallback
For solo dev the CLI also reads a local .dreamrc, resolving providers without a
control-plane round trip. Entries are YAML-tagged mappings and the tag names the
launcher class. Only !providers.SSH, !providers.SLURM, !providers.EC2,
!providers.GCE, and !providers.Kube are known tags — anything else is a hard
parse error.
.dreamrc is resolved first-hit-wins in this order: --dreamrc <path>,
$DREAMRC, ./.dreamrc walked up from cwd, $HOME/.dreamrc, a server pull
(needs auth or LAKESHORE_URL), the on-disk cache of a previous pull, then an
empty config. The server pull composes a DreamRc from the namespace's
providers and modes endpoints, so the two forms converge.
Credentials in a local .dreamrc are inline. Treat the file like a .env and
never commit it.
Referencing a provider from a mode
A mode is a stored RunConfig. It carries the per-machine fields and names
its provider by string:
The override flag is --field on modes add / modes update and --kwarg on
providers add / providers update. Both are repeatable and both nest on dotted
keys, but the names are not interchangeable.
The mode's recognised fields are backend, runner, image, tags,
resources, env, timeout_s, and provider. Anything else is stashed under
resources.__extras and folded back in on read, so unknown keys survive a round
trip. Neither backend nor runner is validated by the API: the create handler
defaults them to fabric and process and otherwise accepts any string.
In a local .dreamrc, the root of the file is the default RunConfig and
named alternatives live under modes: (alias mode:).
What happens at a legacy launch
Merge rules are the same wherever the merge happens:
- Resolve
provideron theRunConfigto the provider row. - Deep-merge the provider kwargs with the
RunConfig's launcher-shaped kwargs. Nested objects merge recursively, arrays concatenate and dedupe, and on a scalar conflict theRunConfigwins. - Runner-side config (
runner,image,resources,env,tags,timeout_s) stays on theRunConfig— the runtime layer consumes it, not the launcher. - Resolve
{ "$secret": "<name>" }markers by decrypting the namedSecretin memory. Anaws_keypairsecret is stored as<access_key_id>:<secret_access_key>and expands to{ accessKeyId, secretAccessKey }; every other kind substitutes its UTF-8 plaintext. Plaintext never leaves the handler and is never serialized back to the wire. See Secrets. - Call the cloud API, or open the SSH session.
The control plane holds the SDK clients and the decrypted credentials; the CLI
never talks to AWS, Google, or the apiserver directly. Every launched instance is
tagged or labelled with lakeshore=true, the namespace, and the provider name,
so instances listings filter cleanly.
Visibility and lifecycle
remove is a soft delete. The row keeps its history and gets a tombstone
suffix (<name> [deleted <iso>]) appended so the (namespace, name) unique
index stays intact and a later providers add <name> does not collide.
Responses strip the suffix, so callers always see the original name.
| State | Set by | Effect on listings |
|---|---|---|
hidden | PATCH .../providers/:name with { hidden: true } | Excluded by default; --hidden opts back in. |
deletedAt | DELETE .../providers/:name | Excluded by default; --deleted opts back in. Restore with POST .../providers/:name/restore. |
The table prints NAME, LAUNCHER, DISPATCH, KWARGS. Hidden rows render as
name [H] and soft-deleted rows as name [D <iso>], with the bare name still
first so grep by name keeps working.
Interactive editor
Two steps. First a picker over the listing — the first entry is a synthetic toggle that flips the hidden + deleted filter without leaving the editor, the second is quit. Then an action menu that adapts to the row: an active row offers hide and delete, a hidden row offers unhide and delete, a soft-deleted row offers restore. The list refetches between iterations, so each action shows up in the next frame. Ctrl-C exits cleanly.
Namespace and org scoping
Providers belong to a namespace. What you see depends on whether that namespace
has an orgId:
- With an
orgId— listings, name lookups, and duplicate-name checks span every namespace sharing thatorgId. Provider names must be unique across the org, and a token scoped to one member namespace works against its siblings. - Without an
orgId— the listing is unscoped: the namespace sees every provider on the control plane.
Give tenant namespaces an orgId if you want provider isolation between teams.
See Admin setup for how to set one.
Not built yet
- SLURM launch. The launch routes support only
EC2,GCE, andKube, andproviders teston a SLURM provider prints a stub message. Clearing that is what clears its[ dev ]marker. - SSH through the server. The CLI drives SSH directly, but no server-side route does, so a queue cannot grow workers on an SSH provider.
- Hybrid dispatch — go direct on the first call, leave a daemon behind for later ones. Pick one mode for now.
- Cross-provider scheduling — listing, claiming, and leasing capacity across providers. Design only.