Task queue

Strip the launch policy away and a Lakeshore queue is an ordinary task queue: producers submit work, workers claim it, results come back by id. This page is that half on its own — everything here holds whether or not a provider is attached.

Set providerRef: null and the queue is only this. Nothing autoscales, nothing is provisioned, and the queue runs on whatever workers you attached yourself — a lab machine, an allocation you already hold, your laptop. Adding a provider is covered in Elastic queues; it changes who supplies the workers, not the behaviors below.

Ids, not futures

submit returns a durable string id — a ULID — not a handle bound to your process:

inv = f.submit(x=1)          # -> "01JD2K7Q...": a string, persist it anywhere
result = q.result(inv)       # poll from this process, or any other, later

That is the load-bearing choice. There is no Future type in the SDK, no gather, no as_completed. A submitting process can exit; another can pick the id up and read the result. Reconnection is the default case rather than a recovery path.

The lifecycle

An invocation moves through a small state machine:

              submit()
                 │
                 ▼
            [ queued ]  ──── a worker claims it
                 │
                 ▼
            [ running ]
           ┌─────┼─────┬──────────┐
           ▼     ▼     ▼          ▼
     [succeeded] [failed] [killed] [timeout]

queued and running are the live states; the other four are terminal. The terminal state is what q.result(inv) resolves against — a failed invocation carries its error rather than raising at submit time.

Idempotent submits

Pass _key= and the submit becomes idempotent:

inv = f.submit(x=1, _key="nightly-2026-08-06")

Resubmitting the same key returns the existing invocation id rather than creating a second one. The key is unique per namespace + queue, so the same key against a different queue is a different job. This is the retry-safe path for anything a scheduler might fire twice.

Streaming results

A generator body does not have to finish before you can read from it. Yields become VALUE frames in a journal, terminated by an END frame, and the stream replays from a cursor:

for frame in q.stream(inv, since=cursor):
    ...

since is a 0-based sequence cursor, so a consumer that drops off can resume from where it stopped instead of re-reading the whole stream. Topic is the same journal spelled as pub/sub.

Back-pressure

The admission block on the queue is where depth limits are declared — max_depth, on_full: reject | block, rate_limit, deadline_cutoff_s.

Running a worker

The whole plane fits in one process when you want to test:

import dreamlake.lakeshore as dls

q = dls.Dispatch(":memory:")      # a full plane, in-process
dls.run_worker(q, once=True)      # drain exactly one job and return

once=True drains a single job, which is what makes a test deterministic. Drop it for a worker that keeps pulling.

Bodies and I/O

One decorator covers four body kinds — sync, generator, async, and async-generator. inspect picks the shape once at decoration; there is no mode flag and no second decorator. See Simple functions for the decorator itself and UDF patterns for the shapes side by side.

File bytes never ride the wire. Bodies read and write through dls.run.read / dls.run.write under a dls.scope(prefix) and return the keys they wrote, so the payload on the invocation stays small and the data stays in storage.