Machine metrics
Implementation in progress; not yet released. The existing Compute Brief shows missing readings and an explicitly labeled example. Installing a current released CLI does not install this collector or connect the page to real metrics. Local Nymph collection, control-plane storage, backend access and UI rollout must be verified separately before treating readings as live.
Nymph collects resource readings on the machine, including machines outside AWS.
CloudWatch is not required. Linux readings cover CPU utilization, memory,
filesystem capacity for the worker directory, network rates and block-device
I/O. NVIDIA GPUs additionally report utilization, VRAM, temperature and power
when nvidia-smi supports those fields. Missing drivers and unsupported fields
produce missing readings. macOS currently provides filesystem and supported
NVIDIA readings only. This feature does not collect or index process arguments,
program output, tool output or terminal contents.
Local collection
Since Nymph 0.1.34: Nymph collects and reports readings only when the daemon
configuration sets [metrics] enabled = true. Configurations that omit the
key stop collecting after upgrading. Nymph does not rewrite configuration
files, and an explicit enabled = true keeps collection on. See
host testing for the related job output setting.
Direct Source snapshots (implementation in progress)
An owner can enable a managed metrics Source for an enrollment with
POST /namespaces/:slug/hosts/:hostId/enrollments/:enrollmentId/metrics/source
(an empty JSON object and an Idempotency-Key). The matching GET reports
unconfigured, pending, or ready; retry the POST after a pending result.
Configuration uses the existing Source permissions and storage. A scoped broker
credential stays on the control plane; Nymph receives only short-lived upload
URLs for its enrollment's fixed snapshot path.
Nymph uploads the complete bounded history directly to object storage. SHA256, byte length, content type and snapshot identity are signed. Conditional writes prevent overlapping uploads from replacing a newer snapshot. A failed upload leaves the previous snapshot readable. The backend checks object metadata to confirm completion without fetching the readings or writing datapoints to Mongo.
POST .../metrics/source/read-capability returns a short-lived GET capability;
the dashboard fetches the snapshot directly from storage. A missing object means
the first upload is still pending. Revocation blocks new capabilities; issued
capabilities remain valid until expiry (uploads two minutes, reads five minutes).
Changing a binding uses a new object prefix so old upload URLs cannot change the
new binding. This does not add a metrics ingestion service or a new storage tier.
For an existing owned enrollment, using the same authenticated API origin as the dashboard:
This requires the compatible Nymph, Lakeshore control plane and DreamLake
backend releases. The operator pins LAKESHORE_METRICS_BROKER_ORIGIN on the
existing control plane to the DreamLake API's HTTPS origin. The managed Source
uses its existing bucket configuration; no new host, bucket or paid service is
provisioned. Configure the bucket's existing CORS policy to allow the dashboard
to GET snapshots; never make the bucket public. The scoped broker credential
is created and encrypted by the existing control-plane secret store during
setup and never returned to the browser or daemon.
The source implementation samples every 10 seconds in a separate background task. CPU and I/O rates need two readings. Counter resets and gaps longer than 30 seconds invalidate rates until a new baseline is available. GPU probes have a two-second deadline and bounded output. Collection and persistence failures do not interrupt workload execution.
With a binary built from the matching source:
--once reads the current host without starting/registering a worker or writing
history. The other commands read the running daemon's loopback API. Do not expose
that unauthenticated endpoint on the network.
Bounded history
| Resolution | Maximum retention |
|---|---|
| 10 seconds | 1 hour |
| 1 minute | 24 hours |
| 15 minutes | 7 days |
| 1 hour | 30 days |
Buckets retain averages, peaks, valid-value counts and total sample counts. Missing buckets remain gaps; missing values are distinct from zero. Coverage is the valid count relative to expected 10-second samples. Current buckets are partial. A snapshot is replaced atomically once per minute and on orderly shutdown; each snapshot is limited to 2 MiB, keeping active and temporary files together under 5 MiB. Fine history is evicted first when space is tight. Malformed or oversized saved history is ignored on restart. A crash can lose the most recent minute. Retention windows are maxima, not guaranteed coverage.
Signed worker heartbeats carry recent idempotent bucket updates. Replayed updates replace the same bucket without double counting. A long disconnection can leave gaps in remote fine history; local history remains authoritative.
Read through DreamLake
The new read-only endpoint is scoped to the authenticated owner, namespace, host and enrollment. It verifies the worker's exact enrollment binding and an active matching daemon key before reading the control-plane metrics. Queue-only mounts do not grant this access. It never executes a command to obtain readings.
Allowed resolutions are 10, 60, 900 and 3600 seconds; the default is 60.
The response contains latest, tiers and stale. latest is null when no
readings exist. All timestamps are Unix seconds. A reading older than 90 seconds
is stale. HTTP 401/403 indicates missing access, 404 a missing owned binding,
409 an unverified worker identity, and 502/503 unavailable upstream readings.
Do not replace an error with zero utilization.
latest.values uses unit-bearing keys such as cpu_percent,
memory_used_bytes, memory_total_bytes, filesystem_used_bytes, and
filesystem_total_bytes. Network series use
net/<interface>/rx_bytes_per_second and tx_bytes_per_second; block-device
series use disk/<device>/read_bytes_per_second and write_bytes_per_second.
GPU series use gpu/<GPU-uuid>/utilization_percent, memory_used_bytes,
memory_total_bytes, temperature_celsius, and power_watts.
Numeric extensions (source contract; rollout pending)
New numeric metric names can be collected and retained without a backend
allowlist release. Add extensions=1 to the metrics URL to return extensions
alongside known resource readings. Existing callers retain their known-resource
response shape. sensors=1 independently includes the existing Apple CPU
temperature, CPU power and Neural Engine power keys.
Extension names are at most 96 ASCII characters, start with a letter, digit or
underscore, and use letters, digits, _, ., :, @, / or -. Diagnostic,
output, credential and prototype-reserved path segments are excluded. Values
must be finite numbers; extensions may represent negative measurements. Extension
aggregates require finite mean/peak, peak at least mean and integer count from
1 through the bucket's sample count. Invalid extensions are ignored. Known
resource readings retain their existing strict validation, including fields
hidden from a legacy response. Payloads remain capped at two MiB and input/stored
value maps at 256 keys; arbitrary metadata is never returned.
Partial bucket updates preserve existing series that were omitted from the update. Retransmitted series replace their own aggregates without double counting. Updates with no usable numeric latest values cannot erase existing readings. When a bucket reaches its key cap, existing series take precedence over adding more names. Clients must handle unfamiliar metric names and units before opting in; extensions do not change scheduling, billing or ownership decisions.
Each tier is an array of { timestamp, samples, values } buckets. Each value is
{ mean, peak, count }. Derive memory/filesystem/VRAM percentages from matching
used and total readings. Device-mapper and backing disks are separate series;
adding them double counts I/O. GPU UUIDs identify devices independently of
index order.
These readings are observational. They are not ownership receipts, billing measurements, scheduler capacity claims or sufficient proof that a machine is idle. Cost display and idle shutdown remain separate work.