DreamLake

Kubernetes providers and launch

A managed Kubernetes launch creates one workload Pod on an existing cluster. Provision worker machines through an InfraStack first; registering a provider does not create a cluster or add nodes. Docker builds the image, and the cluster's container runtime runs it.

Provider calls execute on the user-configured launch server, a verified enrolled Nymph Host with a dedicated IAM role. The hosted backend coordinates authenticated requests and durable receipts; it does not call EKS using service credentials or a stored Kubernetes token for new launches.

Verify release and live acceptance

This guide describes the implemented server-bound route. It requires CLI 0.71.0 or later and a backend containing the launch-server routing and receipt fixes. Live acceptance of this route is pending in this guide. Earlier direct-backend smoke runs do not establish readiness. Verify the deployed route, selected server, role and cluster access before creating workloads.

Follow the CLI Kubernetes walkthrough for InfraStack setup and commands. Node inventory and Pod observations are separate from Nymph enrollment, Job API invocation and terminal readiness.

Existing API contracts

Managed DreamLake APILegacy Lakeshore control plane
Provider kindk8s, declaration version 1Kube launcher
Launch route/namespaces/:ns/lakeshore-providers/:id/launches/v1/namespaces/:ns/providers/:name/launch
WorkloadDocker image with optional command and argsPython script in an image with /bin/sh and python3
Image entrypointPreserved when command is omittedReplaced by a shell/Python wrapper
CredentialsLaunch server instance role, verified by STS; locally signed EKS tokenControl-plane provider config or Secret store
Retry recordDurable DreamLake operation receiptOptional requestId and retained ConfigMap reservation

These are separate contracts. k8s is correct for the managed declaration; Kube is the case-sensitive legacy launcher name. Kubernetes workload launch does not enroll nodes or Pods as Nymph Hosts, create a Job API invocation, or establish terminal readiness.

Set up the launch server and infrastructure

Use the CLI Kubernetes setup walkthrough for the provider template, InfraStack state, EKS worker pools, image builds and teardown. Provision the launch server and dedicated role using an explicitly authorized bootstrap identity, then perform cluster provisioning on that server using its instance role. InfraStack state authentication is separate from AWS provisioning authority; do not copy operator AWS credentials onto the server.

Enroll the server as a Nymph Host in the provider's namespace. Its runtime needs Python 3, boto3, IMDSv2, STS and EKS API connectivity. Configure a private EKS endpoint reachable from the launch server, or an explicitly restricted endpoint. The server role needs eks:DescribeCluster and an EKS access entry with the Kubernetes authorization appropriate to the selected namespace.

Create the workload namespace before launching. Kubernetes RBAC must allow create/get/list/delete on pods, get on pods/log, and cluster-scoped get/list on nodes for inventory. The managed route needs no ConfigMap permissions: its retry reservations are stored in DreamLake. Worker instance roles and Pod service accounts are distinct from the launch server role.

The adapter verifies the expected STS role and account, obtains cluster endpoint and CA information through EKS, and signs an EKS token locally. Bound providers use cluster, region, namespace and launchServer; they do not require a Vault kubeconfig, context, static bearer token or an aws exec plugin. Legacy unbound static-credential providers may retain read-only inventory during migration, but cannot use that path for new launches or lifecycle writes.

Register and preflight a managed provider

http
POST /namespaces/team/lakeshore-providers/registrations
Content-Type: application/json
json
{
  "requestId": "kube-register-01",
  "declaration": {
    "version": 1,
    "name": "kube-dev",
    "kind": "k8s",
    "config": {
      "cluster": "kube-dev",
      "region": "us-east-1",
      "accountId": "123456789012",
      "namespace": "dreamlake-launch",
      "launchServer": {
        "enrollmentId": "1123456789abcdef01234567",
        "expectedRoleArn": "arn:aws:iam::123456789012:role/user-launch"
      }
    },
    "launch": "{\"image\":\"REGISTRY/training:VERSION\",\"resources\":{\"requests\":{\"cpu\":\"1\",\"memory\":\"1Gi\"}}}"
  }
}

Use the returned provider ID for subsequent calls. To migrate an existing provider, use the revision-checked binding operation instead of registering a duplicate; it preserves existing Pod ownership and launch layers. Registration and binding validate configuration and authority, but do not prove that STS or the cluster can be reached.

Launch configuration uses JSON Layer text: provider defaults, named layers in request order, then request overrides. POST .../:id/launch saves the provider layer; POST .../:id/launches creates a Pod. Preflight validates configuration without a cluster call or launch:

http
POST /namespaces/team/lakeshore-providers/PROVIDER_ID/preflight
Content-Type: application/json

{"layers":[],"request":"{\"nodeSelector\":{\"dreamlake.ai/tier\":\"large\"}}"}
Managed launch fieldMeaning
imageRequired image reference, accessible to the cluster; pin a digest for reproducible runs.
command, argsOptional arrays passed as Kubernetes container command/args. Omit command to keep the Docker ENTRYPOINT; no shell or Python is injected.
envLiteral string values; no secret references or envFrom in this contract.
resourcesrequests and limits, with Kubernetes resource quantities as strings.
nodeSelector, tolerationsScheduling constraints, for example a small or large worker pool.
imagePullSecretsNames of existing pull secrets in the workload namespace.
serviceAccountNameOptional existing service account. Automatic token mounting remains disabled.

The Pod has one container named workload, restartPolicy: Never, and exact provider/namespace ownership labels. Unknown launch fields, arbitrary Pod specs, volumes, host paths, privileged settings, and bootstrap scripts are not accepted.

Launch and recover

The launch request below is a protocol reference. Its deployed execution path must satisfy the launch-server requirement before a new request is submitted.

http
POST /namespaces/team/lakeshore-providers/PROVIDER_ID/launches
Content-Type: application/json

{"requestId":"training-run-01","count":1,"layers":[]}

Kubernetes requires count: 1 and no enrollment. A successful response is an operation receipt containing the Pod name and UID. committed means the launch was recorded; it does not mean the container ran successfully.

Receipt statusMeaning and next step
pendingIntent is reserved or creation may have been sent. Read the receipt and inspect the original Pod; repeat the identical request to reconcile.
failedKubernetes definitively rejected creation. Read the saved rejection status, correct the cause, and use a new request ID for a deliberate new attempt.
committedThe original Pod identity and launch snapshot are saved. Replays return that receipt even after Pod cleanup; use live instance status to check execution.

Recover a lost response with GET /namespaces/:ns/provider-operations/by-request/:requestId. Request IDs are private to the creating user and namespace. Reusing an ID for a different provider, action, or payload returns request_id_conflict.

The remote adapter also has a durable execution receipt. If dispatch is pending or its outcome is unknown, use the returned recoveryOperationId with dl providers operations show NAMESPACE/OPERATION_ID --json, and reconcile the original launch request. Do not replace an uncertain request to bypass a permission or transport error. A definitive rejection stays failed; after repairing its cause and verifying no resource exists, a new attempt needs a new ID.

Before the first cluster call, the backend durably marks creation as submitted. After that point, retries only look for the exact original Pod; they never send another create. A crash between that mark and the API call can therefore leave an uncertain operation with no Pod. kubernetes_launch_outcome_unknown requires inspection: do not automatically switch to a new request ID, since the original request may still be in flight. Changing provider configuration or retiring the provider also blocks an unsent reservation. A fresh ID represents new work and should be chosen only after resolving the earlier outcome.

Observe and clean up

OperationManaged endpoint
Live workload statusGET /namespaces/:ns/lakeshore-providers/:id/instances
Bounded logsGET /namespaces/:ns/lakeshore-providers/:id/instances/:pod/logs?tailLines=1000
Delete one workloadPOST /namespaces/:ns/lakeshore-providers/:id/instances/:pod/terminate with a new requestId
Configured cluster inventoryGET /namespaces/:ns/kubernetes-clusters
One provider's inventoryGET /namespaces/:ns/kubernetes-clusters/:id

Instance views include phase, assigned node, readiness, image, creation time and exit code when known. Logs are explicit reads of the workload container, limited to 1–1000 tail lines and 65,536 bytes; potentiallyTruncated means this is bounded output, not proof that text was omitted. Ownership and UID are checked around the log read.

Termination requires provider delete permission, records the original Pod UID, and uses that UID as a deletion precondition. A replacement with the same name cannot be deleted by replaying the old operation. Deletion is asynchronous: a committed receipt may say stopping. start and stop are unsupported for Pods.

Inventory reports nodes from configured clusters and this provider's workload Pods, including worker pools, instance types, readiness and allocatable CPU/memory. It covers configured providers only; errors report unknown freshness and incomplete data. Cluster nodes in this view are not enrolled Nymph Hosts. Pending remote reads reuse their execution receipts. Completed node and Pod components may be reused only within a bounded 60-second observation age; the combined inventory keeps the oldest collection timestamp. Retrying an HTTP read does not make old data fresh. An unavailable or pending read must not be treated as an empty healthy cluster. Provider status remains configuration-only; inventory is the live observation. Delete Pods and follow the InfraStack teardown steps to remove worker and control plane charges. Retiring a provider alone does not destroy infrastructure.

Legacy control-plane setup

The safeguards below describe the merged control-plane change. Verify that release separately from the managed DreamLake backend before relying on request reservations, ownership checks or the early daemon rejection. Deploying the managed backend does not update the legacy control-plane service.

Existing lakeshore clients use the separate Kube launcher. Its provider connection fields are kubeconfig, context, and namespace. A kubeconfig may be a server-local path, inline content, or a { "$secret": "name" } reference; without one, the legacy server uses its default kubeconfig. These rules do not apply to the managed instance-role route above. Put connection settings on the provider, not in runConfigKwargs.

bash
lakeshore providers add my-cluster --launcher Kube \
  --kwarg context=prod-east --kwarg namespace=lakeshore
lakeshore modes add k8s-cpu \
  --field provider=my-cluster --field image=python:3.12-slim \
  --field resources='{"requests":{"cpu":"1","memory":"1Gi"}}'

The legacy API runs script as Python in container user, overriding the image entrypoint with /bin/sh -c; it needs /bin/sh and python3. Its optional Pod fields use node_selector, service_account, and image_pull_secrets rather than the managed API's camelCase fields. resources, env, and tolerations remain available.

Legacy operationEndpoint
Launch PythonPOST /v1/namespaces/:ns/providers/:name/launch
List workload PodsGET /v1/namespaces/:ns/providers/:name/instances
Delete workload PodDELETE /v1/namespaces/:ns/providers/:name/instances/:id

For example, the legacy launch body is:

json
{
  "requestId": "legacy-smoke-01",
  "script": "print('Kubernetes launch completed')",
  "runConfigKwargs": {"image": "python:3.12-slim"}
}

A successful legacy launch returns 202 with instanceId (Pod name), state, providerName, and launchedAt. Listing uses the reserved lakeshore, lakeshore.namespace, and lakeshore.provider labels. Exact ownership for recovery and deletion is stored separately in annotations because label slugification is lossy. Legacy phase mapping reports both Succeeded and Failed as stopped; inspect Kubernetes status/logs to distinguish them.

An optional legacy requestId uses a stable Pod name and immutable ConfigMap reservation. Credentials need ConfigMap create/get in addition to Pod permissions. Keep reservations after deleting Pods: replay of a reserved ID with no Pod returns 409. Calls without requestId create a fresh Pod each time. Deletion verifies exact ownership annotations and UID; old Pods without those annotations require operator cleanup. Read output with kubectl logs.

POST /v1/namespaces/:ns/daemons/launch rejects Kube before creating tokens, Workers, or Pods: a signed Nymph container enrollment contract is not implemented. An already installed Nymph's separate kube runner uses kubectl and a hostPath work directory; it requires the workload and daemon to share that node and path. It is not the managed launch route or an automatic multi-node enrollment mechanism.