Native Experiment charts and rows
DreamLake computes chart series and run summary rows from native project
Tracks. These endpoints read DreamLake data only; historical Dash runs continue
to use Dash. Both require a signed-in user with read access to the project. As
with ml-dash, there is no limit on the number of runs, Tracks, stored points or
returned cells: every requested Track is read chunk by chunk, keeping only the
requested numeric columns. A reduced series
keeps its own budget. A missing or unreadable Track chunk returns
503 NATIVE_METRIC_UNAVAILABLE instead of a partial result.
Chart series
POST /native-experiments/ns/:namespace/projects/:project/chart-series
Select runs with experiments (run IDs) or glob (a run-name pattern using
* and ?). With neither, every active run in the project is used. A glob is
matched against every active run whose name starts with the glob's text before
its first wildcard. metric_names limits which Tracks are read. A column is named
either bare (loss) or with its Track name (train.loss). index is the
stored point index, and ISO timestamp strings become epoch milliseconds.
Without agg, data has one entry per run with experiment_id,
experiment_name, and arrays for x_key and each y_keys entry, sorted by x.
A missing value is null. With agg, data pools every selected run and has
the raw arrays plus one value per <y_key>_<function> (for example
loss_mean). The supported functions are MEAN, MEDIAN, STD (population),
MIN, MAX, SUM, PERCENTILE_25, and PERCENTILE_75.
filters.index_range keeps points with start <= index < end, and
filters.columns limits the returned columns.
With "response_format": "ARROW", the response is an Arrow IPC file
(application/vnd.apache.arrow.file) holding the same values as float64
columns. The X-Data-Points-Processed, X-Compute-Time-Ms, and
X-Data-Fetch-Time-Ms headers carry the compute metadata. Without agg, the table has one row per point,
with experiment_id and experiment_name columns. Those two names cannot be
used as chart keys in this format. With agg, each aggregate value repeats on
every row of the pooled raw columns. If the pooled raw columns have different
lengths, the request returns 422 NATIVE_CHART_NOT_TABULAR; use JSON instead, or
use filters.columns to return only the aggregates. Errors are JSON in both formats.
Reduced series for long runs
Add "max_points": N (2 to 10,000, JSON only, not with agg) to get a reduced
series per run instead of every point. Each Track's stored index range,
narrowed by filters.index_range, is split into equal bins: max_points
divided by the run's Track count. Each bin returns the mean x_key and the mean
of each y_keys column in data, and that column's minimum and maximum in
low and high. The minimum and maximum are exact: every stored point in the
range is read. When a bin holds at most one point, the values are the stored
values. Each run also reports reduced (true if any bin averaged more than one
point) and points (the number of bins returned, never more
than max_points). A run with more selected Tracks than max_points returns
413 NATIVE_COMPUTE_TOO_LARGE; use metric_names to read only the charted Tracks.
A reduced request may read up to 2,000,000 stored points, counted from the
chunks that overlap index_range. To look closer, send a narrower
index_range with the same max_points; only chunks in that range are read.
Bins average x, so x must not decrease as the stored index grows; otherwise the
request returns 422 NATIVE_CHART_NOT_MONOTONIC.
Run summary row
POST /native-experiments/ns/:namespace/projects/:project/runs/:runId/row
accepts {"max_plot_points": 1000, "metric_names": ["train"]}. Both fields are
optional. max_plot_points can be 1 to 5,000; a value of 1 returns summary
values only.
The response has row.single_values, row.plots, and row.metadata. For each
numeric track.column, a single value is chosen from its name: the minimum for
losses and errors, the maximum for accuracy and scores, the latest for learning
rates and counters, and the mean for time, resources, and gradients. Values for
unrecognized names are classified from the data. step is the stored point
index of the chosen value. Each plot has x, y, high, and low arrays and
summary statistics. A series longer than twice max_plot_points is divided
into equal bins, with the mean and the minimum and maximum of each bin.
Per-run dashboard configuration
A run's .dashrc is a YAML document (JSON is also valid) stored as a file on
that run. A project-root or parent-folder .dashrc does not configure its
child runs. The dashboard uses metrics.fields for header values and charts
for its chart list; header is not a supported configuration block.
Without that file, the dashboard currently chooses up to six charts and two
summary values. Its naming heuristic prioritizes train/eval loss and return
fields, so a run may have recorded rollout and system data that the overview
does not show. Stored metric groups, charted fields and header summaries are
different counts. A missing chart alone is not evidence of a failed write.
An explicit chart list replaces that automatic selection; it is not truncated to six entries:
Use keys the run actually recorded. For grouped metrics, qualify the field as
group.column, and log an x value in each group. xKey: default lets the
server choose an axis; an explicit key makes an RL environment-step axis
unambiguous. bins is a chart data-reduction budget, not the number of charts.
Image and video entries filter files by media kind and match glob against
the project-relative file path, including the run path. **/* includes the
selected run's media regardless of its folder. Each grid requests
nCols * nRows files, capped at 100; size the grid for the files you want to
show. A grid limit is not a request to display every future upload.
The RL logging example includes a complete 28-field dashboard and image/video previews. Refresh the run page after uploading or replacing the configuration outside that page. File readback confirms the saved configuration; rendered-page acceptance is a separate check.