<!-- source: https://docs.dreamlake.ai/dev/product-analytics @ 0.1.192 b2f973b1 -->
# Product analytics

API and CLI usage events are stored in an owned PostgreSQL database. Clients
contact only their configured DreamLake API. PostHog imports the selected
`analytics.events` table on a schedule for SQL and product analysis. There is
no direct client or server event forwarding to PostHog.

Collection requires an explicitly configured database, stable pseudonymization
secret and persistent journal mount. Installing a CLI does not configure a server.

## Event catalog (schema version 1)

| Event | Question | Dimensions |
| --- | --- | --- |
| `api_request_completed` | Which API surfaces are used or fail? | registered route template, method, HTTP status, duration |
| `cli_command_completed` | Which commands are used or fail? | registered command path, CLI version, OS, exit code, duration |

Both carry event UUID, UTC timestamp, source, deployment environment and HMAC
pseudonymous actor ID. No email, user name, IP, raw URL, query, body, arguments,
file content, terminal bytes or error text is recorded. API events require an
authenticated principal. CLI events are client-reported, not proof of execution.
Exclude health checks, preflights, telemetry ingestion and unauthenticated calls.
HTTP 200 is transport success, not GraphQL business success. API requests and
CLI invocations are separate units; never sum them as user actions.

## Quiet operator setup

Set `DREAMLAKE_ANALYTICS_DATABASE_SECRET_ARN` to the dedicated writer credential
in AWS Secrets Manager, `DREAMLAKE_ANALYTICS_HMAC_KEY` to a stable secret of at
least 32 characters, and `DREAMLAKE_ANALYTICS_ENV` to the deployment name.
Credentials never enter client builds or Terraform state. The task role reads
only the writer secret. Missing or invalid configuration disables collection.

The bootstrap creates a separate `dreamlake_analytics` database in the existing
private RDS instance. The writer can call the ingestion function but cannot read
or directly modify event rows. PostHog uses a separate read-only account and a
restricted SSH tunnel, with a pinned host key and TLS to PostgreSQL. Connect only
`analytics.events`; schema-version and infrastructure-state tables are excluded.

Use incremental sync on integer `id`, primary key `id`. IDs are allocated under
a transaction lock with sequence cache 1, so committed IDs form a safe cursor
across replicas. Original `timestamp` is occurrence time, `received_at` is server
receipt time. A delayed offline event receives a new ID regardless of its old
timestamp. Duplicate actor/UUID pairs preserve the first accepted row. The
production PostHog source syncs every 15 minutes; uploads to DreamLake attempt
immediately after each eligible event. These are separate schedules.

`DREAMLAKE_TELEMETRY=0` or `DO_NOT_TRACK=1` disables server collection globally.
Clients opt out per request using `X-DreamLake-Telemetry: off` or `DNT: 1`.
The CLI also provides persistent `dl telemetry disable --quiet` and the
same environment switches. Opt-out prevents new collection, not retroactive
removal. HMAC secret rotation starts new actor histories.

Each event is committed to a local journal before an asynchronous upload is
attempted. The collector returns 204 only after persistence (503 on storage
failure). CLI UUIDs and the original UTC timestamp, at millisecond precision,
are preserved through replay; client clocks may be inaccurate. Server-generated
API timestamps record response completion. Delivery uses the same UUID on retry,
not an exactly-once guarantee.

Set `DREAMLAKE_ANALYTICS_JOURNAL_DIR` to an absolute path on a **persistent
mounted filesystem**. The deployment mounts encrypted EFS at
`/var/lib/dreamlake-analytics` through an IAM-authorized access point, and sets
the journal to `/var/lib/dreamlake-analytics/events`. Replicas share immutable
record files, tolerate duplicate delivery and reconcile peer acknowledgements.
Replacement tasks reuse the mount. An ECS writable container layer is insufficient.
Missing the journal setting disables collection; partial mount configuration
fails deployment before activation. Verify filesystem permissions and restart
recovery before enabling production.

The server retains both pending and delivered records for at most seven days,
100 MiB or 100,000 records, evicting oldest records when any limit is reached.
Metadata is rebuilt from disk after restart and refreshed once per minute;
normal writes use an incremental index rather than scanning retained history.
Shared-replica byte/count limits are soft between refreshes and converge during
rotation; sustained outage or retention pressure can evict pending records.
Rotation runs on writes and scheduled flushes. The journal is private to the
service account (directories 0700, files 0600). Upload attempts start after event
persistence and retry every second in batches up to 100, with a three-second
connection deadline, two-second lock wait, five-second SQL statement limit and
eight-second query deadline. A record is acknowledged locally only after a
synchronous PostgreSQL commit; uncertain commits replay safely by actor/UUID. Up to 1,000 writes may wait in memory; rejection is counted,
and CLI ingestion returns 503 so the client retains its copy. Graceful shutdown
attempts one final batch; pending disk records survive. Abrupt termination before
persistence can still lose a newly generated API event. Aggregate delivery,
rotated-pending, expired-pending and corruption counters contain no payloads.

The CLI keeps a separate journal under `~/.dreamlake/telemetry-events`, bounded
by seven days, 10 MiB and 10,000 records. Successful uploads remain in local
history until rotation. The detached uploader attempts delivery after every
command completion and retries pending records at the next eligible invocation.
It exits within an eight-second network budget and never holds the command open
for network access. Disk commits take local I/O time; disk failure does not alter
command results. Explicit CLI opt-out purges local history and stops uploads.
Destination/account hashes prevent replay under another user's credentials.

Connectivity from China must be measured against the configured DreamLake API.
Clients never need a direct PostHog connection. Local buffering covers temporary
unreachability within retention limits; it cannot make an unreachable endpoint
reachable. No China connectivity acceptance has been performed.

## SQL analysis and owned storage

PostgreSQL is the queryable source of truth. Retain its event history and RDS
backups independently of PostHog. Local journals rotate; database history has no
automatic deletion policy in this rollout. Monitor database storage and define
an explicit retention policy before capacity becomes a constraint.

PostHog imports this table into its warehouse for SQL, views and insights. It is
a warehouse table, not automatically a native `events` table; configure the
appropriate warehouse-backed insight when building product reports. Original
UUID, timestamp and properties remain available for validation and independent
SQL analysis. No event SDK or PostHog-to-S3 export is required for this flow.
The previously provisioned S3 export resources are retained but unused.

## Product review

Start with weekly active actors, usage by command/route, CLI nonzero exit rate,
API 5xx rate, and p50/p95 duration. Filter to production and separate CLI from
API. Exclude service automation when defining human adoption; actor identity
alone does not prove a human ran a request. Opt-out, authenticated-only collection
and best-effort delivery make these observed trends, not a census.

Assign an owner to each metric; review weekly and record decisions linked to
one insight and one issue. Add events when a product question needs them, and
retire unused metrics. Define a domain success event before claiming activation
or a business funnel from generic transport events.

The CLI setup guide gains opt-out instructions. Upload, Memos, artifact,
workflow, SDK-only and unsupported-capability task boundaries are unchanged.
Public skill synchronization follows release; unreleased docs must not imply
that installed clients or deployed servers already have this behavior.

## Optional CLI failure diagnostics

Failed `cli_command_completed` events may include `last_http_status` (400–599)
and `last_http_error_code` (bounded machine code). The collector validates and
projects these fields and discards them for successful commands. The last HTTP
failure may have been handled; it is context, not proof of the final cause.
Response bodies, messages, request IDs and resource identifiers are excluded.
Deploy this collector revision before the corresponding CLI release. Older
clients remain valid. These fields stay in the owned usage pipeline and are not
automatically PostHog Error Tracking exceptions.

The built-in agent separately logs typed tool/turn errors with HTTP status,
receipt/turn correlation and partial/blocked/uncertain outcomes. Ownership
transfer refusals include a stable reason and request ID in the API response.
A service-account project grant currently yields
`NOTE_METADATA_PRINCIPAL_UNSUPPORTED` with reason
`service_account_project_grant`; this includes the built-in agent's access.
The diagnostic change does not broaden transfer admission.

Memo move and metadata-transfer commands also record `operation_error_code`
and `operation_outcome` (`failed`, `blocked`, `partial`, or `uncertain`) when
classified by that command. A blocked HTTP 200 preview therefore records its
blocker without inventing an HTTP error status. These optional fields are
validated again by the collector and omitted for successful exits.
