DreamLake

Visualize a source

A source is where DreamLake finds your data. It can have an external layer — an S3 bucket or a Hugging Face repo connected to your namespace, read in place and never written to by DreamLake — and/or a managed layer of DreamLake-hosted files you upload with the CLI. Drop one .dreamrc file at a dataset's root and the app renders every episode in it: cameras, time series, timelines, 3D. Nothing is imported or converted to render — connected storage is read where it lives.

s3://bucket/datasethf.co/you/datasetlinksourceread in place.dreamrcevery episode — cameras · series · timeline

Visibility and sharing

SettingDirect linkPublic profile and source listings
PrivateExisting members and explicitly admitted collaboratorsHidden from visitors
UnlistedAnyone with the link can readHidden from visitors
PublicAnyone can readListed for visitors

Changing visibility requires Source settings permission. It does not change signed-in share links, editors, or Project grants. Link readers cannot upload, edit settings, or delete data. Sharing covers the whole Source, including its external read layer; connection settings and credentials remain restricted. Disabled Sources cannot be read by public or unlisted links. Previously created Sources remain Private.

Read GET /namespaces/:namespace/sources/:id for visibility, capabilities, and settingsRevision. Change visibility with PATCH /namespaces/:namespace/sources/:id/metadata:

json
{"visibility":"unlisted","expectedSettingsRevision":1}

Use the current returned revision, not the example value. A stale revision is rejected. If the response is lost, read the Source again before retrying. Creation and external import accept visibility; omission means private. The namespace list accepts visibility=private|unlisted|public. For visitors, filtering, pagination and counts include only active Public Sources. Signed-in nonmembers request visibility=public for that catalog; their default namespace listing retains its existing access-denied response so older clients can resolve shared Sources directly by name. Anonymous catalog reads need no query flag.

The built-in resource API tools discover these schemas through dreamlake_resource_api_discover (family sources). The repository's source-only local MCP adapter exposes source_get and source_visibility_set; pass the settings revision from source_get. Use a checkout containing those tools and a backend supporting this contract. There is no hosted MCP endpoint or published adapter package.

Connect your data

Linking takes two steps: a Connection holds the credential, and a Source reads through it. Create the Connection in the web app (Connections, /<namespace>/connections), with dl connection create, from the Python or TypeScript SDK, or by asking the assistant, which hands secret entry to the web form. Then import the Source in Sources (/source/<namespace>) or with dl source import --connection <id>. Credentials stay server-side, encrypted — never in files or URLs.

./my-dataseton your machineuploadaws s3 cp · hf uploadconnectbucket / repoyour storagesource/source/⟨ns⟩S3: credentials · HF: OAuthupload API — roadmap, not yet
StorageConnect withGet bytes in today
S3 bucketConnection: access key or roleaws s3 cp --recursive ./my-dataset s3://bucket/prefix
Hugging Face repoConnection: OAuth, token or publichf upload you/dataset ./my-dataset --repo-type dataset
Any source (managed files)dl source create (namespace owner)dl source upload ./my-dataset -r --source <name> --to <prefix>

Uploads land in the source's managed layer, never in your bucket or repo. An uploaded file at a path that also exists in the connected storage takes precedence in the source view, and uploading to an existing managed path replaces it silently — use a fresh prefix. Limits, permissions and overwrite rules: the CLI sources guide.

Then verify from a shell (CLI 0.45.3+) that the source holds what you think:

bash
dl source list                                    # sources you can reach
dl source browse [path] --source <name>           # one level; -r to recurse, --files for files only
dl source stat <path> --source <name>             # one path, which layer it resolves from
dl source download <path> --source <name> -o ./out # spot-check a file

Public GitHub repository support is an implementation preview, with commit-pinned reads and explicit pilot limits.

Shape the dataset

Most robot-learning exports work as-is — a published dataset renders unmodified, with one file added:

my-dataset/.dreamrcepisodes/run_a/cam_front.mp4joints.parquetrun_b/ …one file at the ROOT — no storage: blockone folder per episode · flat namesvideos H.264/AAC MP4LeRobot / zarr / MCAP exports ship as-is
Your dataset root holdsformatAnything to do first?
meta/info.json (LeRobot v2.x / v3.0)lerobotno
*.zarr.zip or a .zarr/ directory (UMI)umino
*.mcap files — Foxglove schemas or plain ROS 1 / ROS 2 bagsmcapno
model/ + episodes/*/frames.parquet + episode.jsonfolderno — a DreamLake sim-playback dataset renders as-is
anything else — videos, images, CSV, parquetfolderone folder per episode

Layout details and shape rules: prepare your data.

Make it render

The .dreamrc names which fields feed which views — that's the whole job. Open the source in the app and the dataset renders:

dataset: format: lerobotviews: - view: videoStack cameras: ['images.*'] - view: lineChart series: [{ field: action }]
.dreamrcyaml
version: 1
dataset:
  format: lerobot # lerobot | umi | mcap | folder
  episodes: auto
views:
  - view: videoStack
    cameras: ['observation.images.*']
  - view: lineChart
    series: [{ field: [action, '*'] }]

The full authoring workflow — reading the field inventory, every view and its options, a live gallery of finished files — starts here.

Single MCAP files

Selecting one .mcap file in the browser renders it as a one-episode dataset. Which config drives the render is a fixed chain — the first hit wins:

  1. <filename>.dreamrc beside the file — test.mcap looks for test.mcap.dreamrc. A config written for this file: its views render with the selected file as the one episode.
  2. The folder's .dreamrc, when its dataset.format is mcap — the unified config for a folder of recordings, applied to just this file.
  3. The source root's .dreamrc, same condition.
  4. No config at all — the file still renders: a default view generated from its Foxglove or ROS well-known schemas. Camera topics become a frame stack, point clouds a 3D pane, tf a moving robot or skeleton (below), numeric channels charts. The header marks it as generated, and copy .dreamrc hands you the exact YAML — save it as <filename>.dreamrc (step 1) and the layout is yours to edit.

The more standard the recording, the more the no-config default shows; channels outside the Foxglove / JSON / ROS schema set fall back to the metadata table with the reason stated.

Show an annotation as an Output Source

A Source can present a published annotation instead of describing files. Give its root .dreamrc exactly one key:

.dreamrcyaml
annotation: dreamlake://<namespace>/<annotation-name>

Visualize then opens that annotation in the annotation viewer: its episodes, video and labelled timeline, plus semantic search when the annotation stores search vectors.

  • Private reads: reads go through the annotation's own private access under the reader's session. A reader who may open the Source but not the annotation sees the annotation's refusal, not its data.
  • Strict format: the file must contain only the annotation key, with a dreamlake://<namespace>/<name> value. Any other .dreamrc is read as a dataset config, exactly as above.

ROS bags and robots

Plain ROS bags need no conversion — ROS 2 (rosbag2's own MCAP output) and ROS 1 alike. Images, point clouds and poses land in the same views as their Foxglove twins, and the /tf tree renders in 3D with Foxglove semantics: transforms compose down the declared parent chain, so a tf-only recording plays as a moving skeleton — joints, bones, no model required.

When the names a recording publishes — tf frame ids, or the joint names on a /joint_states topic — carry a known robot's own naming, the default view goes further and renders the robot itself: a Unitree G1 recording opens as the G1, a DROID episode as a Franka Panda with its Robotiq gripper, meshes streamed from the robot preset registry. Unknown structures keep the honest skeleton — nothing is guessed. To put a robot on a recording yourself, name its URDF in the 3D view:

<filename>.dreamrcyaml
- view: recon3d
  up: z
  tracks: [{ field: '/tf::*', as: transform3d }]
  models:
    - https://huggingface.co/datasets/live9080/dreamlake-robots/resolve/main/g1/g1_29dof.urdf

The viewer reads URDF directly — meshes resolve relative to the file (or through a packages: map for package:// references), each link drives from its /tf::<link> track by name, and links the recording never publishes ride their nearest tracked parent.

A bag that logs only joint angles and no per-link poses — very common, since the angles are the state and the poses are derivable — is 3D too: bind the topic with joints: beside a URDF and the viewer runs the forward kinematics itself, gripper linkages included. Full detail: ROS bags & robot models.

Sim recordings (MuJoCo & URDF)

A teleop recorder that runs MuJoCo can write a DreamLake sim-playback dataset directly — one folder per episode, plus the exact model the session ran snapshotted into model/. Each frames.parquet stores named channels — one column per joint, seven per free body, keyed to MJCF names by episode.json — so the data survives model recompilation and outlives any single scene version. The mujoco view loads the snapshot and plays the channels back kinematically: qpos in, forward kinematics out, never stepping physics — scrubbing the shared timeline is random access, frame-exact in either direction. Because channels are name-keyed, an env: override plays the same data on a scene-only env: hand channels with no match there are skipped and listed, while objects and fixtures replay on their own. ctrl.* columns carry the action stream for a lineChart. The recorder drops this .dreamrc at the root:

.dreamrcyaml
version: 1
name: sharpa-teleop
dataset:
  format: folder
  episodes: "episodes/*/"
views:
  - view: mujoco
    qpos: [frames]          # named channel columns
    meta: [episode]         # episode.json — channels map + model ref
    height: 420
    # env: ns/laundry       # optional: play on a scene-only env
  - view: timeline           # shared clock + playback bar
  - view: lineChart
    series: [{ field: [frames, 'ctrl.*'] }]

The same contract has a URDF flavor for motion that isn't an MJCF scene — a retargeted mocap clip, a policy rollout, anything that is "a robot description plus per-frame joint values". episode.json says sim: "urdf", model.path names the dataset's URDF (with model.files listing every mesh), and channels bind URDF joints by name — one column per revolute/continuous/prismatic joint — plus one freejoint channel naming the root link with seven columns (x, y, z, qw, qx, qy, qz — quaternion w-first, meters, Z-up world) for the floating base. The urdf view drives setJointValue and the base pose directly; joints the robot lacks are listed, not fatal, and the same env: override plays the data on a pushed URDF env:

.dreamrcyaml
version: 1
name: g1-dance
dataset:
  format: folder
  episodes: "episodes/*/"
views:
  - view: urdf
    joints: [frames]        # URDF joint columns, keyed by joint name
    meta: [episode]         # episode.json — sim "urdf", model + channels
    height: 480
  - view: timeline
  - view: lineChart
    series: [{ field: [frames, left_knee_joint] }]

Playback lives here, in sources; the envs you curate stay untouched — pushing and versioning scenes is the envs guide.

Let Claude do it

Install the datasets skill, point at a folder, and say "visualize this dataset in DreamLake":

StepDoes
connectinspects the data, preps the layout, gets it linked, verifies the source
.dreamrcwrites and debugs the .dreamrc — bindings, validation, iteration
bash
dl init                             # once: pick your agent and scope
dl skill install dreamlake-datasets

One skill does both — it gets the bytes connected, then makes them render. Without the CLI, download skills.tar.gz from the latest dreamlake-skills release and copy dreamlake-datasets/ and dreamlake-cli/ into your agent's skills directory.

Next steps

Dataset viz — start here →

The five-step authoring workflow, every view, and the .dreamrc gallery.

Sources (DreamDB) →

A separate kind — structured collections with schemas, pushed with the Python SDK.

Project reference access

A Source reference attaches the whole Source, using its existing reference ProjectSourceBinding relation and existing Bindr references. This release does not migrate Source mounts or create another association table. Unifying them with ordinary resource Nodes is a separate future optimization. Explicit user/team Project READ grants Source USE; WRITE and ADMIN grant Source EDIT for supported managed content. These grants do not confer Source ADMIN, settings/sharing management, Source deletion, or Connection/Vault credentials, and do not bypass Source lifecycle restrictions.

Public Project visibility alone gives visitors no access to an otherwise private Source. Independent Source sharing remains effective at its canonical address. Existing and later references follow current grants dynamically; detaching or revoking removes only that Project's contribution.

The Project-sharing backend and Web update are released. CLI Project commands require 0.62.0 or later; Python Project support requires SDK 0.29.0 or later. See the Project resource access contract.

The released whole-Project and owned-Notes transfer supports Project, ordinary Project files/images/videos, owned Memos, and owned Artifact and Env attachments (Artifact/Env members need the matching released clients). In the deployed service, a Source mount/reference blocks the entire transfer rather than silently excluding it. The Source's existing independent sharing and released Project-derived content access remain available.

Unreleased — intended design contract, not deployed behavior. A planned expansion of the same whole-Project transfer adds purely managed, eligible Sources as manifest members (source, alongside project, note, artifact, env and annotation). It is whole-Project only: no standalone Source move, conversion or migration procedure is added. Server-side blockers — readiness, lifecycle state, storage and active operations — decide each Source's eligibility; external or mixed-layer Sources and Sources carrying a Connection/credential remain unsupported and continue to block the entire operation. No file payload is copied, moved or relocated: the Source ID, its physical storage under the existing bucket/prefix and its in-Project binding IDs stay stable, and the moved Project keeps its original ProjectSourceBinding and BindrSourceReference rows. Bindings, references and catalog placements from outside the moved Project detach; the Memo-only preserveExternalFilings option does not retain them, and no generic Source-relationship migration is introduced. The Source becomes private at the destination; preserving a direct share is conditional on the server's eligibility policies and blockers, and signed capability URLs already issued keep their existing bounded expiry — there is no immediate-invalidation promise. No released backend or client version supports this yet; the whole-Project contract is authoritative for scope and current blockers.

USD scene previews

Selecting a .usd, .usda, .usdc or .usdz file opens the shared Libraries USD viewer in the file preview. Project files use the same renderer. Scene layers and textures load on demand through authorized reads; relative references must remain within the Project or Source. USDZ embedded assets remain inside the package. Source preview permissions still apply. USD previews have a 256 MiB file ceiling, enforced again while reading layers in the worker; texture and package expansion limits are also enforced.

For a collection, place a .dreamrc at its root using the USD collection configuration. Visualize provides a searchable scene catalog and one active 3D viewport, so browsing a large collection does not load all scenes. Orbit and zoom use the same controls, lighting and theme as Libraries. Previews show static geometry, not physics execution; inspect viewer warnings for unsupported content or missing dependencies.

Portable USD previews and collections

USD rendering is owned by @dreamlake/viz/dataset-viz. Libraries, Project file previews and Source file previews share this renderer. DreamLake supplies Project/Source authorization through UsdResourceProvider; independent hosts can use the same component with public HTTP resources or their own resolver. See the view parameters and live example and resource API.

Collection search belongs to DatasetBrowser, not to view: usd. Opt in explicitly in .dreamrc:

yaml
browser: { mode: collection, groupBy: directory, pageSize: 50 }

This presentation works with any dataset format and view. It searches only the enumerated names, paths and task metadata, and renders one selected episode. Task counts, filename details and selection state remain visible while filtering; filters do not silently change the current scene. The Source Visualize tab retains the native file-navigation column and its resize controls; changing views does not automatically hide that column. Collections open as a responsive thumbnail grid without loading a USD scene. Scrolling near the bottom automatically reveals another page until all matching items are shown; pageSize is the append size, not a total scene limit. A displayed/total count and an end-of-list message make progress explicit. Search and task filters apply to the complete indexed catalog, including items not yet displayed. Thumbnails stay lazy and offscreen images are released. Selecting an item narrows the grid alongside the preview while retaining multiple columns of thumbnail cards. Close preview restores the grid, search filters and scroll position, and releases the renderer. When the available preview panel is narrow, the visible two-column grid sits above the selected preview. Native mobile List/Detail navigation remains available through the existing application layout.

For tabletop robotics, framing: workbench offers a closer default view; Reset view and Show full scene remain available. Large scenes are loaded on demand in an abortable worker and render on demand. Unsupported materials and primitives remain documented preview limitations, not simulation support.

For collections with preview images, use the documented indexed USD catalog. The Source can hold the index and JPEG thumbnails in its managed layer while original USD files remain in external storage. The list reads only metadata and nearby thumbnails; selecting an entry starts the USD download. Indexes are snapshots and must be refreshed when the underlying scene collection changes.

Custom applications

For a visualization that needs its own UI and interaction, publish a static application under .dreamlake/visualization/ and select it in .dreamrc. Custom Source visualization covers publishing, the read-only Source bridge, permissions and SDK examples.