DreamLake

Sources (DreamDB)

A DreamLake-hosted store for your own videos and structured data. Push from the terminal — each video is sliced into streamable fragments automatically.

How this differs from file Sources

DreamDB sources hold structured records in collections — videos are transcoded and sliced, scalars and embeddings are typed by a schema. File Sources hold original bytes: an external, read-only layer (a connected bucket or repo) plus a managed layer you upload into (CLI sources guide). The commands on this page only work on DreamDB sources.

Server compatibility

The syntax on this page is the shipped Python CLI. These commands also need the DreamLake service to provide the DreamDB source endpoints (creating a dreamdb-kind source and minting its write credentials). Installing the SDK does not guarantee the service you log in to offers them, and it is not confirmed here that a service with Source v1 enabled accepts dreamdb-kind sources. File Source uploads (dl source upload) are a different path and are not interchangeable with source push.

Install

bash
pip install "dreamlake[auth,dreamdb]"    # or: uv add "dreamlake[auth,dreamdb]"
brew install ffmpeg                      # apt install ffmpeg on Linux

ffmpeg and ffprobe must be on PATH for any push with video.

Run these as python -m dreamlake.cli

The standalone dreamlake binary from cli.dreamlake.ai also has a source group — for uploading original bytes into a Source's managed files — and it does not run the DreamDB commands below. python -m dreamlake.cli always reaches the Python SDK, whatever dreamlake resolves to on your PATH.

Log in

bash
python -m dreamlake.cli login            # device auth flow

Model: source → collection → data

Think database → table → rows.

LayerWhat it isCommand
SourceNamespaced, permissioned storesource create
CollectionA dataset with a schemasource collection create
DataRecords at time anchorssource push

One source holds many collections; access is granted per source.

Quick start — a folder of videos

bash
python -m dreamlake.cli source create my-videos
python -m dreamlake.cli source collection create clips --source my-videos --preset video
python -m dreamlake.cli source push ./videos --source my-videos --collection clips

--preset video = two fields (video + path), and is the default when you pass neither --preset nor --schema. push <dir> uploads every video as one record. Each is normalised to H.264 854×480 (--width/--height to change) and sliced into ~2s fragments (fixed). push only accepts a DreamDB source.

Naming a source

Source and collection names must match ^[a-z0-9][a-z0-9._-]{0,63}$. The namespace is a separate flag, --ns <slug> (alias --namespace). Omit it and the source lands in your login's namespace — the common case:

bash
python -m dreamlake.cli source create my-videos                    # your namespace
python -m dreamlake.cli source create my-videos --ns charlie-57-org # an org you own

Creating requires being the namespace owner — org members are read-only and get 403. A 409 means the name already exists.

Custom schema

Write a schema JSON, pass it to collection create:

schema.jsonjson
{
  "fields": [
    { "name": "video",    "type": "video", "mime": "h264" },
    { "name": "path",     "type": "scalar_string" },
    { "name": "category", "type": "scalar_categorical" },
    { "name": "verified", "type": "scalar_bool" }
  ]
}
bash
python -m dreamlake.cli source collection create labeling --source my-videos --schema schema.json

Field types

TypeValue in a record
videofile path — needs mime (e.g. "h264")
imagefile path (raw bytes)
embedding.npy path or inline list — needs dim
scalar_string · scalar_categoricala string
scalar_int · scalar_timestampan integer (timestamp = ns)
scalar_floata number
scalar_booltrue / false

Push structured data — manifest

A folder can't carry scalar values, so custom schemas push with a manifest: one JSON listing fields + every record. Its fields block is a schema — same file works for both steps.

bash
python -m dreamlake.cli source collection create labeling --source my-videos --schema manifest.json
python -m dreamlake.cli source push --manifest manifest.json --source my-videos --collection labeling
manifest.jsonjson
{
  "fields": [
    { "name": "video",       "type": "video",     "mime": "h264" },
    { "name": "thumb",       "type": "image",      "mime": "jpeg" },
    { "name": "clip",        "type": "embedding",  "dim": 512 },
    { "name": "path",        "type": "scalar_string" },
    { "name": "frame_count", "type": "scalar_int" },
    { "name": "duration_s",  "type": "scalar_float" },
    { "name": "verified",    "type": "scalar_bool" },
    { "name": "category",    "type": "scalar_categorical" },
    { "name": "recorded_at", "type": "scalar_timestamp" }
  ],
  "records": [
    {
      "anchor": 0,
      "video": "clips/a.mp4", "thumb": "thumbs/a.jpg", "clip": "embeds/a.npy",
      "path": "a.mp4", "frame_count": 360, "duration_s": 12.0,
      "verified": true, "category": "assembly", "recorded_at": 1750000000000000000
    },
    {
      "anchor": 3600000000000,
      "video": "clips/b.mp4", "thumb": "thumbs/b.jpg", "clip": "embeds/b.npy",
      "path": "b.mp4", "frame_count": 180, "duration_s": 6.0,
      "verified": false, "category": "electronics", "recorded_at": 1750003600000000000
    }
  ]
}
  • video / image — path relative to the manifest.
  • embedding — a .npy path, or an inline list of exactly dim numbers.
  • anchor — required per record, integer ns. For unrelated videos use index × 3_600_000_000_000 (a 1-hour slot each).

Lay files out next to the manifest: clips/ thumbs/ embeds/.

Everything is validated before anything uploads — bad manifest fails fast, never a half-written collection.

Command reference

bash
python -m dreamlake.cli login

python -m dreamlake.cli source create <name> [--ns <slug>]

python -m dreamlake.cli source collection create <name> --source <src> [--ns <slug>] \
    [--preset video | --schema <schema.json>]

python -m dreamlake.cli source push <dir> --source <src> --collection <col> [--ns <slug>] \
    [--width <px>] [--height <px>]
python -m dreamlake.cli source push --manifest <manifest.json> --source <src> --collection <col> \
    [--ns <slug>]
FlagDefault
--ns / --namespaceyour login's namespace
--preset / --schemavideo preset
--width × --height854 × 480

Planned command spelling (not shipped)

Does not run today

This is the target surface, recorded for reference only. No released dreamlake-py accepts it — use the commands above.

The planned rename makes the groups plural (sources, collections), carries the namespace as an optional prefix on the name — [<namespace>/]<name>, so a source is identifiable from a single string in a command, config file or message — in place of --ns, and adds a configurable fragment duration (--frag-duration <seconds>) to push. The ~2s fragment length is fixed until then.