Command line programs

The second UDF type is a program you run rather than a function you import. The unit of work is an argv line, not a callable — which makes it the right shape for anything already written as a script, a binary, or a tool you did not author.

The first type, a simple function, travels as a reference the worker imports. A command line program travels as a string the worker executes. Everything else — the queue it lands in, the workers that claim it, the provider that supplied them — is identical.

Declaring the interface with params-proto

A bare script has no declared interface: the flags it accepts exist only inside its argument parser. params-proto turns the signature itself into the interface, so the flags, types, defaults and help text all come from one place.

from params_proto import proto


@proto.cli
def encode(
    source: str,                 # Input video key, relative to the workdir
    target: str = "frames/",     # Output key prefix, relative to the workdir
    fps: int = 4,                # Frames sampled per second of source video
    width: int = 854,            # Output frame width in pixels; height follows aspect
    overwrite: bool = False,     # Re-encode even when target already holds frames
):
    """Sample frames from a video and write them into the run scope.

    Reads the video at `source`, samples one frame every `1 / fps` seconds,
    resizes each to `width`, and writes them as JPEGs under `target`. Existing
    frames are left alone unless `--overwrite` is passed.

    Prints one written key per line on stdout so the caller can collect the
    result without re-listing the prefix.
    """
    ...


if __name__ == "__main__":
    encode()

Two things carry over into the CLI automatically. The docstring becomes the program description, and each inline comment becomes that flag's help text — so --help is generated from the same source the function is defined by, and cannot drift from it:

$ python encode.py --help
Sample frames from a video and write them into the run scope.
...
  --source      STR    Input video key, relative to the workdir
  --target      STR    Output key prefix, relative to the workdir    (frames/)
  --fps         INT    Frames sampled per second of source video     (4)
  --width       INT    Output frame width in pixels; height follows aspect  (854)
  --overwrite   BOOL   Re-encode even when target already holds frames (False)

Underscores in a parameter name become dashes on the command line, so batch_size is --batch-size. Booleans are flags: overwrite: bool = False gives --overwrite, and a parameter defaulting to True gives --no-<name>.

Invoking it

lakeshore exec --queue gpu "python encode.py --source raw/run01.mp4 --fps 8"

The queue is the same object a Python UDF submits to — same claim discipline, same workers, same elasticity. See Queue model.

You can pin to a specific worker instead of routing through a queue, but not both; queue and worker_id are mutually exclusive.

Declaring access

A simple function declares what it may reach on the decorator — source= and target=, patterns relative to the workdir. A command line program has no decorator, so the declaration moves to the exec body, alongside the command:

{
  "queue": "gpu",
  "command": "python encode.py --source raw/run01.mp4 --fps 8",
  "workdir": "/workspace",
  "source": ["raw/**"],
  "target": ["frames/**"],
  "mounts": ["raw-video"]
}

Same three fields, same meaning, same workdir-relative patterns — see Declaring access. A repo-wide .dreamrc supplies defaults for both UDF types, so a program that inherits everything it needs declares nothing here.

The JSON representation

Two JSON objects matter here, and they are easy to confuse.

The parsed configuration

params-proto will hand you the resolved parameters as a plain dict, which is what you serialize when you want to record what a run was actually configured with:

{
  "source": "raw/run01.mp4",
  "target": "frames/",
  "fps": 8,
  "width": 854,
  "overwrite": false
}

Reach it with dict(encode) or encode._dict. Values are resolved by precedence — CLI arguments beat direct assignment, which beats a bound context, which beats environment variables, which beats the declared defaults — so the dict reflects the run rather than the source.

The exec wire

What actually crosses to the control plane is the exec body. command is the argv line as a single string:

{
  "queue": "gpu",
  "command": "python encode.py --source raw/run01.mp4 --fps 8",
  "env": { "PYTHONUNBUFFERED": "1" },
  "workdir": "/workspace",
  "timeout_s": 900,
  "payloadInline": "<base64>"
}

The response is a 202 carrying the handle:

{ "exec_id": "01JD2K7Q...", "worker_id": "wkr_..." }

Results come back by that id, the same way an invocation's do:

POST /v1/namespaces/:ns/jobs/:id/result     { resultInline } or { resultRef }
GET  /v1/namespaces/:ns/jobs/:id/result?wait=N
  → { state, resultInline?, resultRef?, exit_code?, error? }

exit_code is the part with no analogue on the Python side — a command line program reports failure by exiting non-zero rather than by raising.

Large inputs

payloadInline is capped server-side. Above that threshold, presign a key through the payload store and send payloadRef instead — the key must begin with payloads/, which is what the presign route mints. The rule is the same one the Python side follows: bytes stay out of the wire and travel by reference.