Visualize a source
A source is where DreamLake finds your data. It can have an external
layer — an S3 bucket or a Hugging Face repo connected to your namespace,
read in place and never written to by DreamLake — and/or a managed
layer of DreamLake-hosted files you upload with the CLI. Drop one
.dreamrc file at a dataset's root and the app renders every episode in
it: cameras, time series, timelines, 3D. Nothing is imported or converted
to render — connected storage is read where it lives.
Visibility and sharing
| Setting | Direct link | Public profile and source listings |
|---|---|---|
| Private | Existing members and explicitly admitted collaborators | Hidden from visitors |
| Unlisted | Anyone with the link can read | Hidden from visitors |
| Public | Anyone can read | Listed for visitors |
Changing visibility requires Source settings permission. It does not change signed-in share links, editors, or Project grants. Link readers cannot upload, edit settings, or delete data. Sharing covers the whole Source, including its external read layer; connection settings and credentials remain restricted. Disabled Sources cannot be read by public or unlisted links. Previously created Sources remain Private.
Read GET /namespaces/:namespace/sources/:id for visibility, capabilities, and
settingsRevision. Change visibility with
PATCH /namespaces/:namespace/sources/:id/metadata:
Use the current returned revision, not the example value. A stale revision is
rejected. If the response is lost, read the Source again before retrying.
Creation and external import accept visibility; omission means private.
The namespace list accepts visibility=private|unlisted|public. For visitors,
filtering, pagination and counts include only active Public Sources. Signed-in
nonmembers request visibility=public for that catalog; their default namespace
listing retains its existing access-denied response so older clients can resolve
shared Sources directly by name. Anonymous catalog reads need no query flag.
The built-in resource API tools discover these schemas through
dreamlake_resource_api_discover (family sources). The repository's source-only
local MCP adapter exposes source_get
and source_visibility_set; pass the settings revision from source_get.
Use a checkout containing those tools and a backend supporting this contract.
There is no hosted MCP endpoint or published adapter package.
Connect your data
Linking takes two steps: a Connection holds the credential,
and a Source reads through it. Create the Connection in the web app
(Connections, /<namespace>/connections), with dl connection create, from
the Python or TypeScript SDK, or by asking the assistant, which hands secret
entry to the web form. Then import the Source in Sources
(/source/<namespace>) or with dl source import --connection <id>.
Credentials stay server-side, encrypted — never in files or URLs.
| Storage | Connect with | Get bytes in today |
|---|---|---|
| S3 bucket | Connection: access key or role | aws s3 cp --recursive ./my-dataset s3://bucket/prefix |
| Hugging Face repo | Connection: OAuth, token or public | hf upload you/dataset ./my-dataset --repo-type dataset |
| Any source (managed files) | dl source create (namespace owner) | dl source upload ./my-dataset -r --source <name> --to <prefix> |
Uploads land in the source's managed layer, never in your bucket or repo. An uploaded file at a path that also exists in the connected storage takes precedence in the source view, and uploading to an existing managed path replaces it silently — use a fresh prefix. Limits, permissions and overwrite rules: the CLI sources guide.
Then verify from a shell (CLI 0.45.3+) that the source holds what you think:
Public GitHub repository support is an implementation preview, with commit-pinned reads and explicit pilot limits.
Shape the dataset
Most robot-learning exports work as-is — a published dataset renders unmodified, with one file added:
| Your dataset root holds | format | Anything to do first? |
|---|---|---|
meta/info.json (LeRobot v2.x / v3.0) | lerobot | no |
*.zarr.zip or a .zarr/ directory (UMI) | umi | no |
*.mcap files — Foxglove schemas or plain ROS 1 / ROS 2 bags | mcap | no |
model/ + episodes/*/frames.parquet + episode.json | folder | no — a DreamLake sim-playback dataset renders as-is |
| anything else — videos, images, CSV, parquet | folder | one folder per episode |
Layout details and shape rules: prepare your data.
Make it render
The .dreamrc names which fields feed which views — that's the whole
job. Open the source in the app and the dataset renders:
The full authoring workflow — reading the field inventory, every view and its options, a live gallery of finished files — starts here.
Single MCAP files
Selecting one .mcap file in the browser renders it as a one-episode
dataset. Which config drives the render is a fixed chain — the first hit
wins:
<filename>.dreamrcbeside the file —test.mcaplooks fortest.mcap.dreamrc. A config written for this file: its views render with the selected file as the one episode.- The folder's
.dreamrc, when itsdataset.formatismcap— the unified config for a folder of recordings, applied to just this file. - The source root's
.dreamrc, same condition. - No config at all — the file still renders: a default view generated
from its Foxglove or ROS well-known schemas.
Camera topics become a frame stack, point clouds a 3D pane, tf a moving
robot or skeleton (below), numeric channels charts. The header marks it
as generated, and copy .dreamrc hands you the exact YAML — save it
as
<filename>.dreamrc(step 1) and the layout is yours to edit.
The more standard the recording, the more the no-config default shows; channels outside the Foxglove / JSON / ROS schema set fall back to the metadata table with the reason stated.
Show an annotation as an Output Source
A Source can present a published annotation instead of describing files. Give
its root .dreamrc exactly one key:
Visualize then opens that annotation in the annotation viewer: its episodes, video and labelled timeline, plus semantic search when the annotation stores search vectors.
- Private reads: reads go through the annotation's own private access under the reader's session. A reader who may open the Source but not the annotation sees the annotation's refusal, not its data.
- Strict format: the file must contain only the
annotationkey, with adreamlake://<namespace>/<name>value. Any other.dreamrcis read as a dataset config, exactly as above.
ROS bags and robots
Plain ROS bags need no conversion — ROS 2 (rosbag2's own MCAP
output) and ROS 1 alike. Images, point clouds and poses land in the
same views as their Foxglove twins, and the /tf tree renders in 3D with
Foxglove semantics: transforms compose down the declared parent chain, so
a tf-only recording plays as a moving skeleton — joints, bones, no
model required.
When the names a recording publishes — tf frame ids, or the joint names on a
/joint_states topic — carry a known robot's own naming, the default view
goes further and renders the robot itself: a Unitree G1 recording
opens as the G1, a DROID episode as a Franka Panda with its Robotiq gripper,
meshes streamed from the
robot preset registry.
Unknown structures keep the honest skeleton — nothing is guessed. To put a
robot on a recording yourself, name its URDF in the 3D view:
The viewer reads URDF directly — meshes resolve relative to the file (or
through a packages: map for package:// references), each link drives from
its /tf::<link> track by name, and links the recording never publishes ride
their nearest tracked parent.
A bag that logs only joint angles and no per-link poses — very common,
since the angles are the state and the poses are derivable — is 3D too: bind
the topic with joints: beside a URDF and the viewer runs the forward
kinematics itself, gripper linkages included. Full detail:
ROS bags & robot models.
Sim recordings (MuJoCo & URDF)
A teleop recorder that runs MuJoCo can write a DreamLake sim-playback
dataset directly — one folder per episode, plus the exact model the
session ran snapshotted into model/. Each frames.parquet stores named
channels — one column per joint, seven per free body, keyed to MJCF names
by episode.json — so the data survives model recompilation and outlives
any single scene version. The mujoco view loads the snapshot and plays the
channels back kinematically: qpos in, forward kinematics out, never
stepping physics — scrubbing the shared timeline is random access,
frame-exact in either direction. Because channels are name-keyed, an env:
override plays the same data on a scene-only env: hand channels
with no match there are skipped and listed, while objects and fixtures
replay on their own. ctrl.* columns carry the action stream for a
lineChart. The recorder drops this .dreamrc at the root:
The same contract has a URDF flavor for motion that isn't an MJCF scene —
a retargeted mocap clip, a policy rollout, anything that is "a robot
description plus per-frame joint values". episode.json says sim: "urdf",
model.path names the dataset's URDF (with model.files listing every
mesh), and channels bind URDF joints by name — one column per
revolute/continuous/prismatic joint — plus one freejoint channel naming
the root link with seven columns (x, y, z, qw, qx, qy, qz — quaternion
w-first, meters, Z-up world) for the floating base. The urdf view
drives setJointValue and the base pose directly; joints the robot lacks
are listed, not fatal, and the same env: override plays the data on a
pushed URDF env:
Playback lives here, in sources; the envs you curate stay untouched — pushing and versioning scenes is the envs guide.
Let Claude do it
Install the datasets skill, point at a folder, and say "visualize this dataset in DreamLake":
| Step | Does |
|---|---|
| connect | inspects the data, preps the layout, gets it linked, verifies the source |
.dreamrc | writes and debugs the .dreamrc — bindings, validation, iteration |
One skill does both — it gets the bytes connected, then makes them render.
Without the CLI, download skills.tar.gz from the latest
dreamlake-skills release
and copy dreamlake-datasets/ and dreamlake-cli/ into your agent's skills
directory.
Next steps
The five-step authoring workflow, every view, and the .dreamrc gallery.
A separate kind — structured collections with schemas, pushed with the Python SDK.
Project reference access
A Source reference attaches the whole Source, using its existing reference
ProjectSourceBinding relation and existing Bindr references. This release does
not migrate Source mounts or create another association table. Unifying them
with ordinary resource Nodes is a separate future optimization. Explicit user/team
Project READ grants Source USE; WRITE and ADMIN grant Source EDIT for supported managed content. These grants do not confer
Source ADMIN, settings/sharing management, Source deletion, or Connection/Vault
credentials, and do not bypass Source lifecycle restrictions.
Public Project visibility alone gives visitors no access to an otherwise private Source. Independent Source sharing remains effective at its canonical address. Existing and later references follow current grants dynamically; detaching or revoking removes only that Project's contribution.
The Project-sharing backend and Web update are released. CLI Project commands require 0.62.0 or later; Python Project support requires SDK 0.29.0 or later. See the Project resource access contract.
The released whole-Project and owned-Notes transfer supports Project, ordinary Project files/images/videos, owned Memos, and owned Artifact and Env attachments (Artifact/Env members need the matching released clients). In the deployed service, a Source mount/reference blocks the entire transfer rather than silently excluding it. The Source's existing independent sharing and released Project-derived content access remain available.
Unreleased — intended design contract, not deployed behavior. A planned expansion of the same whole-Project transfer adds purely managed, eligible Sources as manifest members (
source, alongsideproject,note,artifact,envandannotation). It is whole-Project only: no standalone Source move, conversion or migration procedure is added. Server-side blockers — readiness, lifecycle state, storage and active operations — decide each Source's eligibility; external or mixed-layer Sources and Sources carrying a Connection/credential remain unsupported and continue to block the entire operation. No file payload is copied, moved or relocated: the Source ID, its physical storage under the existing bucket/prefix and its in-Project binding IDs stay stable, and the moved Project keeps its originalProjectSourceBindingandBindrSourceReferencerows. Bindings, references and catalog placements from outside the moved Project detach; the Memo-onlypreserveExternalFilingsoption does not retain them, and no generic Source-relationship migration is introduced. The Source becomes private at the destination; preserving a direct share is conditional on the server's eligibility policies and blockers, and signed capability URLs already issued keep their existing bounded expiry — there is no immediate-invalidation promise. No released backend or client version supports this yet; the whole-Project contract is authoritative for scope and current blockers.
USD scene previews
Selecting a .usd, .usda, .usdc or .usdz file opens the shared Libraries
USD viewer in the file preview. Project files use the same renderer. Scene
layers and textures load on demand through authorized reads; relative
references must remain within the Project or Source. USDZ embedded assets
remain inside the package. Source preview permissions still apply. USD previews
have a 256 MiB file ceiling, enforced again while reading layers in the worker; texture and
package expansion limits are also enforced.
For a collection, place a .dreamrc at its root using the
USD collection configuration.
Visualize provides a searchable scene catalog and one active 3D viewport,
so browsing a large collection does not load all scenes. Orbit and zoom use
the same controls, lighting and theme as Libraries. Previews show static
geometry, not physics execution; inspect viewer warnings for unsupported
content or missing dependencies.
Portable USD previews and collections
USD rendering is owned by @dreamlake/viz/dataset-viz. Libraries, Project file
previews and Source file previews share this renderer. DreamLake supplies
Project/Source authorization through UsdResourceProvider; independent hosts
can use the same component with public HTTP resources or their own resolver.
See the view parameters and live example
and resource API.
Collection search belongs to DatasetBrowser, not to view: usd. Opt in
explicitly in .dreamrc:
This presentation works with any dataset format and view. It searches only the
enumerated names, paths and task metadata, and renders one selected episode.
Task counts, filename details and selection state remain visible while filtering;
filters do not silently change the current scene. The Source Visualize tab retains the native file-navigation column and its
resize controls; changing views does not automatically hide that column.
Collections open as a responsive thumbnail grid without loading a USD scene.
Scrolling near the bottom automatically reveals another page until all matching
items are shown; pageSize is the append size, not a total scene limit.
A displayed/total count and an end-of-list message make progress explicit.
Search and task filters apply to the complete indexed catalog, including items
not yet displayed. Thumbnails stay lazy and offscreen images are released.
Selecting an item narrows the grid alongside the preview while retaining multiple
columns of thumbnail cards.
Close preview restores the grid, search filters and scroll position, and releases
the renderer. When the available preview panel is narrow, the visible two-column
grid sits above the selected preview. Native mobile List/Detail navigation remains
available through the existing application layout.
For tabletop robotics, framing: workbench offers a closer default view; Reset
view and Show full scene remain available. Large scenes are loaded on demand in
an abortable worker and render on demand. Unsupported materials and primitives
remain documented preview limitations, not simulation support.
For collections with preview images, use the documented indexed USD catalog. The Source can hold the index and JPEG thumbnails in its managed layer while original USD files remain in external storage. The list reads only metadata and nearby thumbnails; selecting an entry starts the USD download. Indexes are snapshots and must be refreshed when the underlying scene collection changes.
Custom applications
For a visualization that needs its own UI and interaction, publish a static
application under .dreamlake/visualization/ and select it in .dreamrc.
Custom Source visualization covers publishing,
the read-only Source bridge, permissions and SDK examples.