DreamLake

Product analytics

API and CLI usage events are stored in an owned PostgreSQL database. Clients contact only their configured DreamLake API. PostHog imports the selected analytics.events table on a schedule for SQL and product analysis. There is no direct client or server event forwarding to PostHog.

Collection requires an explicitly configured database, stable pseudonymization secret and persistent journal mount. Installing a CLI does not configure a server.

Event catalog (schema version 1)

EventQuestionDimensions
api_request_completedWhich API surfaces are used or fail?registered route template, method, HTTP status, duration
cli_command_completedWhich commands are used or fail?registered command path, CLI version, OS, exit code, duration

Both carry event UUID, UTC timestamp, source, deployment environment and HMAC pseudonymous actor ID. No email, user name, IP, raw URL, query, body, arguments, file content, terminal bytes or error text is recorded. API events require an authenticated principal. CLI events are client-reported, not proof of execution. Exclude health checks, preflights, telemetry ingestion and unauthenticated calls. HTTP 200 is transport success, not GraphQL business success. API requests and CLI invocations are separate units; never sum them as user actions.

Quiet operator setup

Set DREAMLAKE_ANALYTICS_DATABASE_SECRET_ARN to the dedicated writer credential in AWS Secrets Manager, DREAMLAKE_ANALYTICS_HMAC_KEY to a stable secret of at least 32 characters, and DREAMLAKE_ANALYTICS_ENV to the deployment name. Credentials never enter client builds or Terraform state. The task role reads only the writer secret. Missing or invalid configuration disables collection.

The bootstrap creates a separate dreamlake_analytics database in the existing private RDS instance. The writer can call the ingestion function but cannot read or directly modify event rows. PostHog uses a separate read-only account and a restricted SSH tunnel, with a pinned host key and TLS to PostgreSQL. Connect only analytics.events; schema-version and infrastructure-state tables are excluded.

Use incremental sync on integer id, primary key id. IDs are allocated under a transaction lock with sequence cache 1, so committed IDs form a safe cursor across replicas. Original timestamp is occurrence time, received_at is server receipt time. A delayed offline event receives a new ID regardless of its old timestamp. Duplicate actor/UUID pairs preserve the first accepted row. The production PostHog source syncs every 15 minutes; uploads to DreamLake attempt immediately after each eligible event. These are separate schedules.

DREAMLAKE_TELEMETRY=0 or DO_NOT_TRACK=1 disables server collection globally. Clients opt out per request using X-DreamLake-Telemetry: off or DNT: 1. The CLI also provides persistent dl telemetry disable --quiet and the same environment switches. Opt-out prevents new collection, not retroactive removal. HMAC secret rotation starts new actor histories.

Each event is committed to a local journal before an asynchronous upload is attempted. The collector returns 204 only after persistence (503 on storage failure). CLI UUIDs and the original UTC timestamp, at millisecond precision, are preserved through replay; client clocks may be inaccurate. Server-generated API timestamps record response completion. Delivery uses the same UUID on retry, not an exactly-once guarantee.

Set DREAMLAKE_ANALYTICS_JOURNAL_DIR to an absolute path on a persistent mounted filesystem. The deployment mounts encrypted EFS at /var/lib/dreamlake-analytics through an IAM-authorized access point, and sets the journal to /var/lib/dreamlake-analytics/events. Replicas share immutable record files, tolerate duplicate delivery and reconcile peer acknowledgements. Replacement tasks reuse the mount. An ECS writable container layer is insufficient. Missing the journal setting disables collection; partial mount configuration fails deployment before activation. Verify filesystem permissions and restart recovery before enabling production.

The server retains both pending and delivered records for at most seven days, 100 MiB or 100,000 records, evicting oldest records when any limit is reached. Metadata is rebuilt from disk after restart and refreshed once per minute; normal writes use an incremental index rather than scanning retained history. Shared-replica byte/count limits are soft between refreshes and converge during rotation; sustained outage or retention pressure can evict pending records. Rotation runs on writes and scheduled flushes. The journal is private to the service account (directories 0700, files 0600). Upload attempts start after event persistence and retry every second in batches up to 100, with a three-second connection deadline, two-second lock wait, five-second SQL statement limit and eight-second query deadline. A record is acknowledged locally only after a synchronous PostgreSQL commit; uncertain commits replay safely by actor/UUID. Up to 1,000 writes may wait in memory; rejection is counted, and CLI ingestion returns 503 so the client retains its copy. Graceful shutdown attempts one final batch; pending disk records survive. Abrupt termination before persistence can still lose a newly generated API event. Aggregate delivery, rotated-pending, expired-pending and corruption counters contain no payloads.

The CLI keeps a separate journal under ~/.dreamlake/telemetry-events, bounded by seven days, 10 MiB and 10,000 records. Successful uploads remain in local history until rotation. The detached uploader attempts delivery after every command completion and retries pending records at the next eligible invocation. It exits within an eight-second network budget and never holds the command open for network access. Disk commits take local I/O time; disk failure does not alter command results. Explicit CLI opt-out purges local history and stops uploads. Destination/account hashes prevent replay under another user's credentials.

Connectivity from China must be measured against the configured DreamLake API. Clients never need a direct PostHog connection. Local buffering covers temporary unreachability within retention limits; it cannot make an unreachable endpoint reachable. No China connectivity acceptance has been performed.

SQL analysis and owned storage

PostgreSQL is the queryable source of truth. Retain its event history and RDS backups independently of PostHog. Local journals rotate; database history has no automatic deletion policy in this rollout. Monitor database storage and define an explicit retention policy before capacity becomes a constraint.

PostHog imports this table into its warehouse for SQL, views and insights. It is a warehouse table, not automatically a native events table; configure the appropriate warehouse-backed insight when building product reports. Original UUID, timestamp and properties remain available for validation and independent SQL analysis. No event SDK or PostHog-to-S3 export is required for this flow. The previously provisioned S3 export resources are retained but unused.

Product review

Start with weekly active actors, usage by command/route, CLI nonzero exit rate, API 5xx rate, and p50/p95 duration. Filter to production and separate CLI from API. Exclude service automation when defining human adoption; actor identity alone does not prove a human ran a request. Opt-out, authenticated-only collection and best-effort delivery make these observed trends, not a census.

Assign an owner to each metric; review weekly and record decisions linked to one insight and one issue. Add events when a product question needs them, and retire unused metrics. Define a domain success event before claiming activation or a business funnel from generic transport events.

The CLI setup guide gains opt-out instructions. Upload, Memos, artifact, workflow, SDK-only and unsupported-capability task boundaries are unchanged. Public skill synchronization follows release; unreleased docs must not imply that installed clients or deployed servers already have this behavior.

Optional CLI failure diagnostics

Failed cli_command_completed events may include last_http_status (400–599) and last_http_error_code (bounded machine code). The collector validates and projects these fields and discards them for successful commands. The last HTTP failure may have been handled; it is context, not proof of the final cause. Response bodies, messages, request IDs and resource identifiers are excluded. Deploy this collector revision before the corresponding CLI release. Older clients remain valid. These fields stay in the owned usage pipeline and are not automatically PostHog Error Tracking exceptions.

The built-in agent separately logs typed tool/turn errors with HTTP status, receipt/turn correlation and partial/blocked/uncertain outcomes. Ownership transfer refusals include a stable reason and request ID in the API response. A service-account project grant currently yields NOTE_METADATA_PRINCIPAL_UNSUPPORTED with reason service_account_project_grant; this includes the built-in agent's access. The diagnostic change does not broaden transfer admission.

Memo move and metadata-transfer commands also record operation_error_code and operation_outcome (failed, blocked, partial, or uncertain) when classified by that command. A blocked HTTP 200 preview therefore records its blocker without inventing an HTTP error status. These optional fields are validated again by the collector and omitted for successful exits.