Kubernetes providers and launch
A managed Kubernetes launch creates one workload Pod on an existing cluster. Provision worker machines through an InfraStack first; registering a provider does not create a cluster or add nodes. Docker builds the image, and the cluster's container runtime runs it.
Provider calls execute on the user-configured launch server, a verified enrolled Nymph Host with a dedicated IAM role. The hosted backend coordinates authenticated requests and durable receipts; it does not call EKS using service credentials or a stored Kubernetes token for new launches.
This guide describes the implemented server-bound route. It requires CLI 0.71.0 or later and a backend containing the launch-server routing and receipt fixes. Live acceptance of this route is pending in this guide. Earlier direct-backend smoke runs do not establish readiness. Verify the deployed route, selected server, role and cluster access before creating workloads.
Follow the CLI Kubernetes walkthrough for InfraStack setup and commands. Node inventory and Pod observations are separate from Nymph enrollment, Job API invocation and terminal readiness.
Existing API contracts
| Managed DreamLake API | Legacy Lakeshore control plane | |
|---|---|---|
| Provider kind | k8s, declaration version 1 | Kube launcher |
| Launch route | /namespaces/:ns/lakeshore-providers/:id/launches | /v1/namespaces/:ns/providers/:name/launch |
| Workload | Docker image with optional command and args | Python script in an image with /bin/sh and python3 |
| Image entrypoint | Preserved when command is omitted | Replaced by a shell/Python wrapper |
| Credentials | Launch server instance role, verified by STS; locally signed EKS token | Control-plane provider config or Secret store |
| Retry record | Durable DreamLake operation receipt | Optional requestId and retained ConfigMap reservation |
These are separate contracts. k8s is correct for the managed declaration;
Kube is the case-sensitive legacy launcher name. Kubernetes workload launch
does not enroll nodes or Pods as Nymph Hosts, create a Job API invocation, or
establish terminal readiness.
Set up the launch server and infrastructure
Use the CLI Kubernetes setup walkthrough for the provider template, InfraStack state, EKS worker pools, image builds and teardown. Provision the launch server and dedicated role using an explicitly authorized bootstrap identity, then perform cluster provisioning on that server using its instance role. InfraStack state authentication is separate from AWS provisioning authority; do not copy operator AWS credentials onto the server.
Enroll the server as a Nymph Host in the provider's namespace. Its runtime needs
Python 3, boto3, IMDSv2, STS and EKS API connectivity. Configure a private EKS
endpoint reachable from the launch server, or an explicitly restricted endpoint.
The server role needs eks:DescribeCluster and an EKS access entry with the
Kubernetes authorization appropriate to the selected namespace.
Create the workload namespace before launching. Kubernetes RBAC must allow
create/get/list/delete on pods, get on pods/log, and cluster-scoped
get/list on nodes for inventory. The managed route needs no ConfigMap
permissions: its retry reservations are stored in DreamLake. Worker instance
roles and Pod service accounts are distinct from the launch server role.
The adapter verifies the expected STS role and account, obtains cluster endpoint
and CA information through EKS, and signs an EKS token locally. Bound providers
use cluster, region, namespace and launchServer; they do not require a
Vault kubeconfig, context, static bearer token or an aws exec plugin. Legacy
unbound static-credential providers may retain read-only inventory during
migration, but cannot use that path for new launches or lifecycle writes.
Register and preflight a managed provider
Use the returned provider ID for subsequent calls. To migrate an existing provider, use the revision-checked binding operation instead of registering a duplicate; it preserves existing Pod ownership and launch layers. Registration and binding validate configuration and authority, but do not prove that STS or the cluster can be reached.
Launch configuration uses
JSON Layer text: provider defaults, named layers in request order, then request
overrides. POST .../:id/launch saves the provider layer; POST .../:id/launches creates a Pod. Preflight validates configuration without a
cluster call or launch:
| Managed launch field | Meaning |
|---|---|
image | Required image reference, accessible to the cluster; pin a digest for reproducible runs. |
command, args | Optional arrays passed as Kubernetes container command/args. Omit command to keep the Docker ENTRYPOINT; no shell or Python is injected. |
env | Literal string values; no secret references or envFrom in this contract. |
resources | requests and limits, with Kubernetes resource quantities as strings. |
nodeSelector, tolerations | Scheduling constraints, for example a small or large worker pool. |
imagePullSecrets | Names of existing pull secrets in the workload namespace. |
serviceAccountName | Optional existing service account. Automatic token mounting remains disabled. |
The Pod has one container named workload, restartPolicy: Never, and exact
provider/namespace ownership labels. Unknown launch fields, arbitrary Pod specs,
volumes, host paths, privileged settings, and bootstrap scripts are not accepted.
Launch and recover
The launch request below is a protocol reference. Its deployed execution path must satisfy the launch-server requirement before a new request is submitted.
Kubernetes requires count: 1 and no enrollment. A successful response is an
operation receipt containing the Pod name and UID. committed means the launch
was recorded; it does not mean the container ran successfully.
| Receipt status | Meaning and next step |
|---|---|
pending | Intent is reserved or creation may have been sent. Read the receipt and inspect the original Pod; repeat the identical request to reconcile. |
failed | Kubernetes definitively rejected creation. Read the saved rejection status, correct the cause, and use a new request ID for a deliberate new attempt. |
committed | The original Pod identity and launch snapshot are saved. Replays return that receipt even after Pod cleanup; use live instance status to check execution. |
Recover a lost response with GET /namespaces/:ns/provider-operations/by-request/:requestId.
Request IDs are private to the creating user and namespace. Reusing an ID for a
different provider, action, or payload returns request_id_conflict.
The remote adapter also has a durable execution receipt. If dispatch is pending
or its outcome is unknown, use the returned recoveryOperationId with
dl providers operations show NAMESPACE/OPERATION_ID --json, and
reconcile the original launch request. Do not replace an uncertain request to
bypass a permission or transport error. A definitive rejection stays failed;
after repairing its cause and verifying no resource exists, a new attempt needs
a new ID.
Before the first cluster call, the backend durably marks creation as submitted.
After that point, retries only look for the exact original Pod; they never send
another create. A crash between that mark and the API call can therefore leave
an uncertain operation with no Pod. kubernetes_launch_outcome_unknown requires
inspection: do not automatically switch to a new request ID, since the original
request may still be in flight. Changing provider configuration or retiring the
provider also blocks an unsent reservation. A fresh ID represents new work and
should be chosen only after resolving the earlier outcome.
Observe and clean up
| Operation | Managed endpoint |
|---|---|
| Live workload status | GET /namespaces/:ns/lakeshore-providers/:id/instances |
| Bounded logs | GET /namespaces/:ns/lakeshore-providers/:id/instances/:pod/logs?tailLines=1000 |
| Delete one workload | POST /namespaces/:ns/lakeshore-providers/:id/instances/:pod/terminate with a new requestId |
| Configured cluster inventory | GET /namespaces/:ns/kubernetes-clusters |
| One provider's inventory | GET /namespaces/:ns/kubernetes-clusters/:id |
Instance views include phase, assigned node, readiness, image, creation time and
exit code when known. Logs are explicit reads of the workload container,
limited to 1–1000 tail lines and 65,536 bytes; potentiallyTruncated means this
is bounded output, not proof that text was omitted. Ownership and UID are checked
around the log read.
Termination requires provider delete permission, records the original Pod UID,
and uses that UID as a deletion precondition. A replacement with the same name
cannot be deleted by replaying the old operation. Deletion is asynchronous: a
committed receipt may say stopping. start and stop are unsupported for Pods.
Inventory reports nodes from configured clusters and this provider's workload Pods, including worker pools, instance types, readiness and allocatable CPU/memory. It covers configured providers only; errors report unknown freshness and incomplete data. Cluster nodes in this view are not enrolled Nymph Hosts. Pending remote reads reuse their execution receipts. Completed node and Pod components may be reused only within a bounded 60-second observation age; the combined inventory keeps the oldest collection timestamp. Retrying an HTTP read does not make old data fresh. An unavailable or pending read must not be treated as an empty healthy cluster. Provider status remains configuration-only; inventory is the live observation. Delete Pods and follow the InfraStack teardown steps to remove worker and control plane charges. Retiring a provider alone does not destroy infrastructure.
Legacy control-plane setup
The safeguards below describe the merged control-plane change. Verify that release separately from the managed DreamLake backend before relying on request reservations, ownership checks or the early daemon rejection. Deploying the managed backend does not update the legacy control-plane service.
Existing lakeshore clients use the separate Kube launcher. Its provider
connection fields are kubeconfig, context, and namespace. A kubeconfig may
be a server-local path, inline content, or a { "$secret": "name" } reference;
without one, the legacy server uses its default kubeconfig. These rules do not
apply to the managed instance-role route above. Put connection settings on the provider,
not in runConfigKwargs.
lakeshore providers add my-cluster --launcher Kube \
--kwarg context=prod-east --kwarg namespace=lakeshore
lakeshore modes add k8s-cpu \
--field provider=my-cluster --field image=python:3.12-slim \
--field resources='{"requests":{"cpu":"1","memory":"1Gi"}}'providers:
my-cluster: !providers.Kube
context: prod-east
namespace: lakeshore
modes:
k8s-cpu:
provider: my-cluster
image: python:3.12-slim
resources:
requests: { cpu: "1", memory: "1Gi" }The legacy API runs script as Python in container user, overriding the image
entrypoint with /bin/sh -c; it needs /bin/sh and python3. Its optional Pod
fields use node_selector, service_account, and image_pull_secrets rather
than the managed API's camelCase fields. resources, env, and tolerations
remain available.
| Legacy operation | Endpoint |
|---|---|
| Launch Python | POST /v1/namespaces/:ns/providers/:name/launch |
| List workload Pods | GET /v1/namespaces/:ns/providers/:name/instances |
| Delete workload Pod | DELETE /v1/namespaces/:ns/providers/:name/instances/:id |
For example, the legacy launch body is:
A successful legacy launch returns 202 with instanceId (Pod name), state,
providerName, and launchedAt. Listing uses the reserved lakeshore,
lakeshore.namespace, and lakeshore.provider labels. Exact ownership for
recovery and deletion is stored separately in annotations because label
slugification is lossy. Legacy phase mapping reports both Succeeded and
Failed as stopped; inspect Kubernetes status/logs to distinguish them.
An optional legacy requestId uses a stable Pod name and immutable ConfigMap
reservation. Credentials need ConfigMap create/get in addition to Pod
permissions. Keep reservations after deleting Pods: replay of a reserved ID
with no Pod returns 409. Calls without requestId create a fresh Pod each time.
Deletion verifies exact ownership annotations and UID; old Pods without those
annotations require operator cleanup. Read output with kubectl logs.
POST /v1/namespaces/:ns/daemons/launch rejects Kube before creating tokens,
Workers, or Pods: a signed Nymph container enrollment contract is not implemented.
An already installed Nymph's separate kube runner uses kubectl and a
hostPath work directory; it requires the workload and daemon to share that node
and path. It is not the managed launch route or an automatic multi-node enrollment
mechanism.