# The Registry

  The registry is where a piece of code stops being a file on your laptop and
  becomes a **name** that other people can find, pin and import. DreamLake owns
  the declaration half of that: the name, the source, the signature, and the
  environment the code expects. It owns none of the execution half.

This page is the **write** direction — declaring something and giving it a
version history. The read direction is
[Importing from the Registry](/lakeshore/registry/import.md).

## Two layers, and the split is the whole design

A registered runnable is two rows, not one.

`Runnable` is the **mutable head**. It survives every edit to the function body,
and it is what a name resolves to:

```ts file="Runnable"
interface Runnable {
  id:            string
  namespaceId:   string
  name:          string    // "<module>.<qualname>", unique within the namespace
  module:        string
  qualname:      string
  kind:          string    // "udf" | "session" | "agent"; defaults to "udf"
  description?:  string    // one-line documentation

  markdown?:     string    // agent-only — the agent's INSTRUCTIONS, not its docs
  channel?:      object    // agent-only — { kind, target?, config? }

  latestVersion?: string   // denormalised pointer to the newest version
  metadata?:     object
  createdAt:     Date
  updatedAt:     Date
}
```

`RunnableVersion` is **immutable**. One row per distinct source revision,
append-only — note the missing `updatedAt` and the missing `deletedAt`:

```ts file="RunnableVersion"
interface RunnableVersion {
  id:          string
  runnableId:  string
  namespaceId: string
  version:     string    // sha256(source), LOWERCASE HEX — the content key
  source?:     string    // full source captured at decoration time
  signature:   object    // { params, returns, doc } — shape owned by the SDK
  metadata?:   object
  createdAt:   Date
}
```

The head is keyed `(namespaceId, name, deletedAt)`. The version is keyed
`(runnableId, version)`. Everything below follows from those two keys.

### `kind` is a lifetime, not a substance

`udf`, `session` and `agent` answer one question — how long the process lives:

| `kind` | Lifetime |
| --- | --- |
| `udf` | One-shot. Invoked, runs to completion, returns. |
| `session` | Long-lived, with a bidirectional pipe held open. |
| `agent` | A session plus a policy driving it. |

That is the only axis `kind` encodes. It does **not** say whether the body is
sync, async, a generator or an async generator — see
[what the registry cannot tell you](/lakeshore/registry/import.md#the-registry-cannot-say-what-shape-a-body-is).

## What makes this version *control* and not version *display*

Four properties, and all four are load-bearing.

**1 · Registration is idempotent by content and never overwrites.** Posting a
version that already exists returns **200 with the stored row**, unchanged. A
new one returns **201**. That is safe rather than merely convenient, because
`version` **is** `sha256(source)` — a row already under that key already holds
those exact bytes, so a second caller cannot silently replace what the first one
stored. There is no PUT anywhere in this collection and no way to rewrite a
version row.

**2 · `latestVersion` is a cache, and only one endpoint may move it.** It is
deliberately not settable through `PATCH`; the version-registration endpoint is
the only writer. Re-registering an *older* version hits the 200 path and returns
early, so the pointer never moves backwards. `RunnableVersion` stays the source
of truth — if that denormalised write were ever lost, the newest row is still
findable by `createdAt desc`.

**3 · Deleting the head is soft, and leaves every version intact.** `DELETE` on
a `Runnable` stamps `deletedAt` and stops there. The version rows are untouched;
they are hidden because their parent is tombstoned, not because they were
removed.

**4 · Nothing deletes a version. Not soft, not hard.** There is no endpoint that
deletes a `RunnableVersion` or a `RepoSnapshot` anywhere in the collection.
`DELETE` exists only on the three mutable heads (`Runnable`, `RunConfig`,
`Repo`). Both children are append-only immutable history keyed by content, and
hard-deleting one would orphan every invocation on the control plane that names
it.

## The routes

Every route is namespace-scoped. `:id` resolves as a **24-hex ObjectId** or as
the **dotted name** — the SDK knows the id, a human knows
`examples.embed.embed_frames`, and both work.

| Method | Route | Notes |
| --- | --- | --- |
| `POST` | `/namespaces/:slug/runnables` | Create. 409 on a duplicate name. Needs `runnable:create`. |
| `GET` | `/namespaces/:slug/runnables` | List, paginated. `?kind=` and `?module=` filters. |
| `GET` | `/namespaces/:slug/runnables/:id` | The head, including `latestVersion`. |
| `PATCH` | `/namespaces/:slug/runnables/:id` | Partial. Omitted fields are left alone. `latestVersion` is rejected. |
| `DELETE` | `/namespaces/:slug/runnables/:id` | Soft. Versions survive. |
| `GET` | `/namespaces/:slug/runnables/:id/versions` | History, newest first. **No `source`.** |
| `POST` | `/namespaces/:slug/runnables/:id/versions` | Register. 201 new, 200 existing. Needs `runnable:update`. |
| `GET` | `/namespaces/:slug/runnables/:id/versions/:version` | One version, **with `source`**. |

List responses are enveloped under a collection key — `runnables`, `versions`,
`runConfigs`, `repos`, `snapshots` — alongside the pagination fields. Errors are
always `{ error: string }` and nothing else.

> **Note:** Not `runnable:create` — the version belongs to a runnable that already exists,
>   so pushing one is an update to that runnable. In practice: an org **ADMIN** can
>   push versions, and an org **MEMBER** (READ) can list and read them but cannot
>   push. If your SDK can create a runnable but 403s on the first
>   `POST …/versions`, this is why.

## Declaring, end to end

The `dreamlake` CLI wraps each of the routes above one for one.

```bash file="declare and version"
# 1 — create the head. Create-only; a second run is a 409.
dreamlake declare runnable.json
#   POST /namespaces/<ns>/runnables → 201

# 2 — push the first version. The hash is computed from the file bytes.
dreamlake runnable version push examples.embed.embed_frames \
  --source src/v1_embed.py
#   sha256 = 9f2c…  →  POST …/runnables/<id>/versions → 201

# 3 — edit the source, push again. latestVersion moves.
dreamlake runnable version push examples.embed.embed_frames \
  --source src/v2_embed.py
#   sha256 = 41ab…  →  201

# 4 — push v1 again, unchanged. This is the point of the whole design.
dreamlake runnable version push examples.embed.embed_frames \
  --source src/v1_embed.py
#   sha256 = 9f2c…  →  200 UNCHANGED, and latestVersion stays at 41ab…

# 5 — the history, newest first, without dragging source over the wire.
dreamlake runnable version list examples.embed.embed_frames

# 6 — one version, with its source.
dreamlake runnable version show examples.embed.embed_frames 9f2c…
```

Step 4 is the sentence worth remembering: **the version string is the hash of
the source, so a second caller pushing the same bytes cannot silently replace
what the first one stored, and pushing an old revision does not rewind the
head.**

> **Warning:** The hash is computed from the file bytes and cannot be supplied. This is not
>   ergonomics — `RunnableVersion.version` must stay **byte-identical** to the
>   control plane's `Function.version`, since that identity is what lets a row here
>   and a row there agree with no translation table. A `--version` flag would let a
>   caller mint a key that the SDK will never reproduce, and the two registries
>   would fork with no error anywhere. The CLI prints the hash it computed before
>   it writes.

## RunConfig — the environment, declared separately

A `RunConfig` is the named, reusable declaration of the environment a runnable
runs inside. It is a sibling of `Runnable`, not a field on it.

```ts file="RunConfig"
interface RunConfig {
  id:           string
  namespaceId:  string
  name:         string      // label, unique within the namespace
  description?: string

  runtime:      object      // read by the RUNTIME layer
  extras:       object      // forwarded VERBATIM to the launcher
  tags:         string[]
  provider?:    string      // a LakeshoreProvider NAME, not a foreign key
  hostKey?:     string      // derived "hk1_…"; never accepted from a client

  metadata?:    object
  createdAt:    Date
  updatedAt:    Date
}
```

### `runtime` and `extras` are two audiences, and they do not feed each other

This is the thing readers get wrong.

- **`runtime`** is read by the runtime layer:
  `{ backend?, server?, runner?, image?, resources?, env?, timeout_s?, host_setup?, run_setup? }`.
- **`extras`** is handed to the launcher untouched:
  `{ instance_type?, partition?, time_limit?, n_cpu?, n_gpu?, … }`.

Setting `runtime.resources.gpu` does not fill in `extras.n_gpu`, and setting
`extras.instance_type` does not tell the runtime anything. From the schema
comment, which is the clearest statement of why there are two columns rather
than one bag:

> Keeping them in one bag is exactly what let `_RUNTIME_FIELDS` rot into a
> hand-maintained filter with zero readers; two columns make the boundary a
> schema fact.

`tags` is a typed column rather than a key in either bag because it is the one
field forwarded to **both** halves — an EC2 instance tag and a job tag are the
same string, so it cannot honestly live on one side.

`provider` is a `LakeshoreProvider.name`, **not** a foreign key. The control
plane resolves providers by name, late, and an FK here would break the moment it
does.

### `hostKey` is derived, and its exclusions are claims

`hostKey` is recomputed on every write and **never** accepted from a client — the
create body schema simply has no such property, and `PATCH` recomputes it from
the **merged** value whenever `runtime`, `extras` or `provider` is touched.

```text file="the derivation"
subset    = { v: 1, runner, image, resources, host_setup, provider, extras }
canonical = JSON, UTF-8, no whitespace, keys sorted by code point, recursively
digest    = sha256(canonical)
hostKey   = "hk1_" + base32-lower-nopad(digest[0..10])     // 20 chars total
```

Two configs with the same `hostKey` can share a worker. What is *left out* is
where the meaning is:

| Excluded | Because |
| --- | --- |
| `env` | Applied per invocation. Hashing it would cold-start every job that differs by one variable. |
| `run_setup` | Per invocation as well — it does not change what the machine *is*. |
| `timeout_s`, `tags`, `backend` / `server` | None of them change the machine. |
| `mounts` | Nothing reads a `Mount` anywhere in the stack yet. |

And `host_setup` **is** in the key, with its **order preserved** — the array is
never sorted. Running `pip install torch` before `apt-get install ffmpeg` is a
different host preparation, and pretending otherwise would let a worker prepared
one way serve a job that declared the other. Adding anything to the subset bumps
the prefix to `hk2_`.

```bash file="reading it back"
# Which configs can share a worker?
dreamlake runconfig list --host-key hk1_4d7qz2xk6m3aynbe

# The derived key on one config — you never sent this value.
dreamlake runconfig show gpu-a10g
```

> **Warning:** Four languages implement the host key with no shared code (the Python SDK,
>   nymph, this server, and the TypeScript CLI), and **the Python SDK's copy is the
>   authoritative one** — it runs where the declaration actually exists and stamps
>   `run_config.host_key` at submit time. The server's copy exists so the dashboard
>   can show a key and so a divergence is a *diffable value* rather than a silent
>   one. `src/lib/host-key.ts` says it outright: *"Never treat the value this
>   returns as a scheduling decision."*
>
>   One sharp edge that follows: a **float** anywhere in `resources` makes the key
>   uncomputable, and the server stores `null` rather than rounding. That is a
>   deliberate hard error — IEEE-754 round-tripping through msgpack and three JSON
>   serializers is not something to bet a cache key on — but it means you can end
>   up with an accepted row that has silently lost its placement column. Keep
>   `resources` values integers or strings.

> **Warning:** There is no `RunConfigVersion` model, no history table, and no snapshot of a
>   config as it was when a run used it. The GraphQL schema states the intent
>   plainly: *"Mutable by name and NOT versioned: it is a hand-edited operator
>   record, and references to it (`Invocation.runConfigRef`) are by name on
>   purpose."*
>
>   The consequence is worth saying flatly: **editing a `RunConfig` silently
>   changes what every future run does**, and there is no diff, no history and no
>   rollback. A pinned runnable version plus a mutable config is a *half* pin. If
>   you need the environment pinned too, today the honest move is to create a new
>   config under a new name rather than editing the existing one.

## Agents: `markdown` and `channel`

An agent is a `Runnable` with `kind: "agent"` and two extra columns:

- **`markdown`** — the agent's instructions, authored as Markdown. This is
  **behaviour**. `description` is the one-line documentation, and the two are
  deliberately distinct fields.
- **`channel`** — where the agent's pipe is attached. A stored *declaration*
  only; no live connection, socket or session is modelled in DreamLake.

Both are rejected with a 400 on any `kind` other than `agent`, so a caller who
typed prose into a UDF finds out rather than watching it disappear.

### `channel` is shape-checked and credential-scanned

`channel` must be **exactly** `{ kind, target?, config? }`. Any other top-level
key is a 400. Then every key inside it is walked — to a depth of 8, through
arrays — and rejected if it looks like a credential:

```text file="the credential key pattern"
/token|secret|password|passwd|api_?key|private_?key|authorization|credential|bearer/i
```

Order matters, and it surprises people: the **shape** check runs first, so
`{ token: "x" }` comes back with the *shape* error, not the credential error —
`token` is not one of the three allowed top-level keys. `{ config: { auth: { token: "x" } } }`
is the case that gets the credential error.

> **Warning:** There is no `secretCiphertext` column anywhere in `Runnable`, `RunConfig` or
>   `Repo`. Anything you store in `channel` is stored in **plaintext**, and comes
>   straight back out of every `GET` and out of the GraphQL `Runnable.channel`
>   field to any org member holding only READ. The shape check plus the key walk is
>   the one and only enforcement point there is.
>
>   The fix a caller usually wants is to reference a `Source` (for data) or a
>   `LakeshoreProvider` (for a launcher) **by name**, and let the thing that
>   already does AES-256-GCM hold the secret.

> **Note:** Every document implies it is, including the shape above. The server does not
>   enforce it — `{ target: "#eng" }` with no `kind` is accepted. Treat `kind` as
>   required in your own tooling; just do not expect the API to catch you.

> **Warning:** `markdown` and `channel` are columns on the **mutable head**. Editing an
>   agent's instructions leaves no diff, no history and no rollback — the source
>   hash covers the code and nothing else. The GraphQL schema calls this the
>   *split tripwire*.
>
>   The fix, if it turns out to matter, is a version history **on the head row** —
>   not folding the prose into the source hash. That second option looks cheaper
>   and is much worse: `RunnableVersion.version` is byte-identical to the control
>   plane's `Function.version`, so mixing prose into it would fork the identity of
>   every function in the registry to fix a problem that affects agents only.

## One database caveat that will bite you

> **Warning:** All three head models declare `@@unique([namespaceId, name, deletedAt])`, and
>   `RunnableVersion` declares `@@unique([runnableId, version])`. On MongoDB a
>   Prisma `@@unique` is **not** enforced by the query engine — it is a real
>   database index, and that index only exists once someone has run Prisma's schema
>   push against that database, by hand and out of band. **That has not been run.**
>
>   What you actually get today is a `nameTaken()` pre-read before every create,
>   which is what makes the 409 deterministic in the normal case. The `P2002` catch
>   behind it is a backstop for the race between that read and the write — and it
>   can only fire once the index exists. So **two concurrent creates of the same
>   name can both succeed** on a database without the index, which is exactly the
>   state a fresh test run and a first deploy are both in.
>
>   If you are building anything that creates declarations concurrently, serialise
>   it yourself for now.

## Try it

The starter kit's **`04_version_history`** directory walks the six steps above
against a local control plane — declare, push, edit, push, push the old bytes
back, and watch the 200 — and finishes on the versions panel in the dashboard
with a `?v=<sha256>` deep link.

    The read direction: discover, pin, import, and what is missing.

    What a `@udf` body may look like, and what a version row is capturing.

    `host_setup` is in the host key, and its order is significant.