Remote Vault delivery through tracked runs

Status: Implemented first process-host slice; deployment and installed-client acceptance remain separate. Vault #241 tracks the remaining requirements. Backend delivery/pipeline, control-plane private execution, Nymph32/45 and the paired clients are implemented; UI267 is merged, not deployed. Python0.16 is published; Native CLI0.20 is published and verified; npm0.20 is also published and verified. Use the current command guide and dated evidence. This plan retains broader design requirements that are not all delivered by the initial slice.

Smallest complete slice

Reuse TrackedRun → control-plane ExecJob → Nymph tracked runner. Start from an existing, online, personally owned enrolled process host. Fetch a public HTTPS repository at an explicit full commit, prepare its locked Python environment, deliver only selected string entries/fields into private files or the child environment, run the intended command, then remove owned credential files. The UI, CLI and Python receive durable per-entry preparation/delivery/cleanup outcomes and real process status. No success is inferred from submission or saving.

The initial slice supports an operator-allowlisted Git HTTPS origin, a pinned commit, a uv.lock project and the process runner. Private repository authentication, submodules/LFS, Slurm, Kubernetes execution, arbitrary user-supplied setup commands and persistent credential installation require separate reviewed capabilities. These limits do not prevent a complete public-repository task from using private service credentials. Missing host enrollment, bootstrap access, git, uv or private tracked-run version1/output-suppression support returns an explicit preflight requirement; never silently enroll a host, borrow SSH credentials, or fall back to an older runner. The existing enrollment guide remains the handoff for missing bootstrap.

Original inspected baseline and design scope

Source evidence: workspace 8dcd2d3, Nymph 08de611, control plane 93c295a; client API inspected in the released Python0.15 tree and current tracked-run CLI sources.

The table records the original design scope, not a claim that every proposed field or retry stage exists. Current implemented interfaces and limits are documented in the command guide; preserve the distinction when extending this plan.

Public request and CLI/Python parity

Extend the existing POST /namespaces/:slug/runs; retain GET status/logs and POST cancel. A setup request is mutually exclusive with inline source.files, and setup must be absent for legacy execution. Reject unknown fields at the API, control plane and daemon. The manifest contains no values, access keys, credential-bearing URLs or interpolation expressions.

A reviewed metadata file could contain the following. IDs/revisions come from actual metadata inspection and are enforced at delivery, as supported by the merged UI267 submission flow.

{
  "repository": "https://github.com/OWNER/REPOSITORY.git",
  "commit": "FULL_40_CHARACTER_COMMIT",
  "destination": "research-check",
  "dependencies": "uv-lock",
  "mappings": [
    {"id": "api", "entryId": "REVIEWED_ENTRY_ID", "revision": 3, "valueShape": "map", "selection": {"kind": "fields", "names": ["token"]}, "output": {"kind": "env", "name": "SERVICE_TOKEN"}},
    {"id": "config", "entryId": "REVIEWED_ENTRY_ID", "revision": 2, "valueShape": "map", "selection": {"kind": "whole"}, "output": {"kind": "file", "path": "service.json", "format": "json"}}
  ]
}

Implemented submission/status/cancel syntax, requiring compatible enabled server and worker capabilities, with all DreamLake options before workload arguments:

# setup.json contains metadata only. Explicit consent delegates these selections
# to this enrolled host for this run; workload output will not be captured.
dreamlake run --target alice/research/host --enrollment-id REVIEWED_ENROLLMENT \
  --setup setup.json --allow-vault-delivery --request-id research-check-001 \
  --timeout-seconds 3600 --no-wait --uv-run python verify_service.py

dreamlake runs status alice/RUN_ID --json
dreamlake runs cancel alice/RUN_ID --json

# client is already authenticated; no library prompts or secret values here.
run = client.runs.submit(
    "alice/research/host", enrollment_id="REVIEWED_ENROLLMENT",
    kind="uv-run", argv=["python", "verify_service.py"],
    setup=reviewed_metadata, allow_vault_delivery=True,
    request_id="research-check-001", timeout_seconds=3600,
)
status = client.runs.status("alice", run["id"])
client.runs.cancel("alice", run["id"])

destination is a validated label under the daemon's configured private run root, not an arbitrary absolute path. The initial public receipt does not promise an absolute checkout path. The credential directory is a separate owned sibling of the checkout; only the task receives DREAMLAKE_SECRETS_DIR to find selected basenames. Git checkout and dependency preparation do not receive this variable or the delivered values. The task working directory remains the checkout; a file mapping is resolved under the credential directory, never under that working directory. No file from the repository can redirect a secret write. Environment targets require a single string; a whole field map requires explicit JSON-file encoding. Preserve UTF-8 bytes/BOM/CRLF for string files. Reject NUL for environment values, duplicate mappings/outputs, unsafe basenames, OTP seed/code delivery, over-limit values and protected launcher variables (PATH, HOME, dynamic-loader variables and preparation-tool configuration).

Reading a selected file in the workload

The following is workload code for the implemented private contract, not a claim that hosted submission is enabled. With the service.json mapping above, the workload opens the provided directory explicitly. Do not copy credentials into the repository or print their contents. Let a missing directory or file fail the task instead of falling back to a checkout path.

# Inside the remote workload, after preparation and selected input delivery.
: "${DREAMLAKE_SECRETS_DIR:?credential delivery directory is required}"
test -r "$DREAMLAKE_SECRETS_DIR/service.json"
# Pass a file path to a program that supports it; the secret stays out of argv.
service-client --config-file "$DREAMLAKE_SECRETS_DIR/service.json"

# Inside verify_service.py, whose working directory is the pinned checkout.
import json
import os
from pathlib import Path

credentials = Path(os.environ["DREAMLAKE_SECRETS_DIR"])
config = json.loads((credentials / "service.json").read_text(encoding="utf-8"))
# Use config in the intended service call; do not log or persist it.

service-client illustrates a workload-specific executable, not a DreamLake command. The worker owns cleanup of its separate credential directory after verified termination. The workload must not delete a parent directory or treat DREAMLAKE_SECRETS_DIR as a persistent credential installation.

Run-scoped authorization and direct worker delivery

An explicit allowVaultDelivery: true creates a bounded grant as part of the run record. It is not a vault access-key entry and creates no reusable bearer token. Only personal owner entries are eligible. Pin owner user/namespace, host, enrollment, Unix user, worker ID, enrolled Ed25519 public key, mapping ID, immutable entry ID/revision and field subset. Pin the reviewed setup fingerprint and a server-bounded deadline no later than the run deadline; default one hour, maximum the existing 24-hour run limit. No implicit renewal or namespace/team expansion.

Nymph calls a new dedicated signed route, POST /v1/daemon/runs/:id/inputs/:mappingId, directly over HTTPS to its configured DreamLake API origin. The origin is an enrollment/operator setting, never a manifest-supplied URL; reject redirects. Reuse the Ed25519 canonical method/path/body-digest/timestamp signature format and interoperability vectors from Nymph/control-plane authentication. Add an atomic nonce reservation keyed by signing key ID and nonce, retained through the full accepted timestamp window; nonce expiry never removes uncertain run metadata. The route accepts no owner bearer fallback and exposes no general vault-read interface.

Before and after KMS I/O, verify the owner is still authorized, host/enrollment/key are still active and exactly match the grant, the request generation/deadline is current, cancellation/terminal state has not revoked delivery, and the entry remains active at the exact ID/revision. Project only the selected field/string, then return a bounded no-store response directly to the worker. Wrong owner, worker, run, mapping, field, revision, expired grant or replayed signature fails without values. A replacement at the same name never inherits the grant. Re-authorizing a newer revision requires a new reviewed request, not an automatic retry. The route and HTTP middleware must never log response bodies or credentials.

Authentication canonicalization requires separate review before scaffolding is implemented. Preserve the existing eight-line lsd-v1 Ed25519 byte format: prefix, uppercase method, exact path, worker ID, derived key ID, canonical integer timestamp, base64url nonce, lowercase SHA-256 of raw body bytes. Capture bounded raw bytes before JSON parsing; never verify a reserialized body. Only POST and exact ASCII route shapes are accepted; reject query strings, percent-encoding, dot segments and alternate slash forms. Reject duplicate/array/comma-joined signature headers rather than inheriting the control-plane parser's first-header behavior. Enforce a 32-byte public key, 64-byte signature and 16-byte random nonce with canonical unpadded base64url, derived key ID, safe integer epoch seconds and bounded clock skew.

Every signed body also binds audience to the daemon's configured canonical API origin, purpose to dreamlake-run-delivery-v1, the exact run/mapping or progress action, and delivery generation. This prevents cross-origin/staging-production use of otherwise similar signed paths; the existing format does not itself sign a host header. Derive the expected worker/key from the stored run/enrollment, never a caller-selected lookup alone. Reserve a nonce only after signature and grant checks, atomically and before decryption; an unauthenticated caller cannot fill the nonce store. Dedicated routes register outside the JWT-only subtree but require this strict signature path with no optional-auth mode. Cross-language fixed vectors and real HTTP raw-body/header tests are mandatory.

Receipt state distinguishes authorization, bytes released, worker materialization and task use. Successful retrieval alone does not prove materialization or consumption. A lost response can be retried with a new signed nonce but the same stable run/mapping/generation while preparation remains authorized; only the same pinned value is eligible. This is deliberately redeliverable preparation, not a misleading one-time-read promise. Existing one-time vault access keys are neither consumed nor copied: submission requires the owner session, and the run grant is a separate explicit delegation. No value redelivery is permitted after cancellation, expiry or permit reservation. Same-attempt receipt recovery and owned cleanup acknowledgements remain metadata-only and may continue after release closes.

Worker protocol and bounded schema

A local launch_reserved journal entry is fsynced before requesting the execution permit. Within the same live attempt, a lost permit response is reconcilable. After a worker restart, a reserved launch with no definitive process/result ownership evidence becomes outcome_unknown, even if the crash might have preceded spawn; it must not automatically execute. This intentionally prefers a reviewable uncertain result over duplicate external side effects.

The reviewed auth/store scaffold uses global (keyId, nonce) uniqueness across runs, with signature-window expiry and no plaintext. This is stricter than per-run replay uniqueness. Expired rows are removable in bounded batches; pending cleanup/run journals must not be removed by nonce retention. Mapping metadata is bounded to 100 selections, 64 KiB per selected value and 1 MiB total per delivery generation. Persist only metadata for the finite run grant, not a new access-key object per worker. Input responses are never stored for replay: retries re-authorize and re-decrypt the exact revision. Apply request rate/byte bounds before KMS work.

Checkout, preparation and output boundary

Validate HTTPS repository origin against an operator allowlist, reject credentials/query/fragment/custom ports and require a full commit. Create the checkout under an exclusively created mode0700 run directory. Execute Git by argv without a shell, disable global/system configuration, hooks, credential helpers, recursive submodules and LFS filters, and use a clean HOME/environment. Fetch the explicit commit and verify checked-out HEAD exactly. Fail closed on unsupported repository features; never substitute a branch tip. Bound network/preparation time and source size.

Prepare uv.lock dependencies before any secret is delivered, using the resolved uv executable with argv sync --frozen --no-dev --no-editable. This deliberately trusts the pinned repository and its locked dependencies: sdists and the project may execute build-backend code during this pre-secret stage. Non-editable installation is not a sandbox, and a lockfile does not prevent build scripts. There is no additional free-form preparation command API. The task invocation is the resolved uv executable with argv run --no-sync --frozen --no-env-file -- followed by the reviewed workload argv (for example python verify_service.py); a leading workload option is rejected rather than treated as another uv flag. The --no-sync stage must not install or rebuild dependencies after injection. Confirm these exact flags against the installed supported uv version during capability checks; see the official command reference.

Use a clean allowlisted environment; do not pass daemon authentication, cloud credentials or earlier run secrets into Git/uv. Reject external workspace/path dependencies and inherited index credentials in this first slice. Build code can still leave code or processes on the shared Unix account; the owner must trust it to the same degree as the intended task. A stronger untrusted-build boundary would require a separately designed sandbox, not a promise from --frozen. Once preparation succeeds, materialize only selected files using exclusive no-follow opens under the separate mode0700 credential directory; files are mode0600. Environment values stay in worker memory and enter only the intended child environment. Resolve the executable before injection. Run the locked task with no implicit environment synchronization; no secret is placed in argv, run.source, ExecJob.env, source bundles or the local journal.

Output capture is disabled at the source for this secret-enabled slice. Git/dependency/task stdout and stderr use null sinks; no shell tracing, pipe buffering, progress log forwarding, arbitrary exception text or result-inline payloads. Nymph sends only fixed stage/error codes, exit status and bounded metadata. Control plane and server also reject/drop nonempty output for these jobs before storing it. UI/CLI must say this before consent; runs logs returns an explicit captureDisabled marker. General string redaction cannot guarantee that a task will not print, encode or transform credentials.

This is authorized code execution, not a sandbox guarantee: the selected repository/dependencies and the remote Unix account are trusted to consume the values. The task can copy or transmit them. Cancellation/retirement cannot recall bytes already delivered, and removing the owned files does not prove erasure of application-created copies. Do not claim protection from the intended program or other processes sharing that Unix account.

Extended outcome and restart requirements

The initial API exposes run status plus mapping pending/materialized/cleaned receipts; the richer stage/restart model below remains an extension requirement. Store bounded metadata-only stages for checkout, dependencies and each mapping: pending, checking, materialized, failed, outcome_unknown, cleaned, with fixed reason codes and observed start/end/review timestamps. Record execution separately (not_started, starting, running, terminal/unknown) and cleanup separately (pending, complete, blocked). Persist a monotonic generation/sequence and exact ownership marker; an old worker result cannot overwrite a newer retry/cancel state. No plaintext-derived hashes of low-entropy passwords are emitted as evidence.

Add owner-authenticated POST /namespaces/:slug/runs/:id/retry accepting explicit failed mapping IDs and the observed generation. It advances preparation only after reconciling the existing worker journal; successful unchanged mappings are preserved, retry never broadens the selected set. Retired/changed entries or new mappings require a new reviewed run. Worker restart may redeliver the same pinned inputs before execution, after rechecking authorization and exact owned staging paths. Preparation deadline expiry revokes the grant and schedules cleanup; no indefinite credential-bearing idle operation.

Before spawning the task, persist a single-use execution intent/permit and the local ownership journal. Replays reconcile that permit and verified PID/start identity rather than launching a second program. The spawn boundary cannot provide universal exactly-once execution: if a crash leaves it unclear whether the intended program ran, return outcome_unknown, preserve cleanup ownership, and require an explicitly new run to execute again. Do not automatically rerun user side effects. A task exit0 can be execution_succeeded while overall cleanup remains blocked; never collapse those into an all-clear success.

Cancellation immediately revokes further delivery and preparation retries, then uses existing verified process-group cancellation. It remains cancel_requested until the worker acknowledges termination; lost connectivity is not proof of stopping. Clean only the owned private credential directory after termination is verified. Persist exact directory/owner/inode markers before writing, recover them after restart, reject symlink/foreign paths, and keep cleanup_blocked evidence rather than deleting an ambiguous target. Startup recovery and a periodic bounded sweep remove expired owned staging files; journal retention outlives unresolved cleanup. Do not retire source vault entries, revoke shared SSH keys, or delete unrelated directories as run cleanup.

Required implementation and acceptance gates

1. Independently review the run grant and signed worker authentication/nonce schema before wiring decryption. Real Mongo tests must cover owner/worker/host/enrollment/field/revision denial, revocation during KMS I/O, cancellation, nonce replay, concurrent redemption and stale-generation results. Verify plaintext never reaches source/CP/database/log serializers.
2. Implement versioned CP/Nymph capability negotiation and process runner preparation/cleanup. Test Git wrong-commit and hostile checkout paths, dependency failures, exact UTF-8 files/env, output suppression (including deliberately printed/transformed sentinels), partial materialization, lost response, daemon crash before/after spawn, cancellation and owned-path cleanup refusal.
3. Supply identical CLI/Python submit/status/retry/cancel contracts and real subprocess tests. Browser desktop/mobile acceptance must review explicit selections, show actual metadata outcomes, clear stale auth, handle missing bootstrap and never fetch the values itself.
4. Run the full path with a real enrolled SSH-accessible machine, real HTTP/Mongo/CP/Nymph and KMS, an explicit synthetic repository commit and a synthetic service that independently observes the intended task using both selected file and env values. Drop delivery/result responses, interrupt preparation and cancel a running task; prove no duplicate task launch, preserved successful mappings and precise cleanup. Publish a human-runnable example with pinned revisions and remaining limits.

The contract is complete only when that actual checkout/task/credential/cleanup flow passes. A retrieval-only command, mocked transport, metadata review screen or saved vault entry cannot close the delivery requirement.

Pipeline implementation split (not enabled)

The original implementation split below guided the now-merged main/CP/Nymph work. Private execution uses uv-run-private-v1, so older runners cannot silently ignore setup and execute the ordinary logged path. Main API enablement, deployed CP support and a fresh compatible Nymph poll remain explicit runtime prerequisites; publication alone does not satisfy them.

Whole/field selections use the explicit discriminator shown above; no empty-array, wildcard or implicit field-map meaning. Entry names are display metadata resolved from the reviewed IDs, not a substitute for immutable IDs/revisions. Submission/status/cancel are implemented; a separate public per-mapping retry API remains proposed and is not part of the command guide.