Add stream request observability

Adds request observability for streams, seeded demo traces and logs, documentation, tests, and review feedback fixes.

Author: sorenbs

.changeset/busy-onions-obey.md

@@ -0,0 +1,5 @@
+---
+"@prisma/studio-core": minor
+---
+
+Add stream request observability

AGENTS.md

@@ -39,6 +39,8 @@ These instructions apply to the `@prisma/studio-core` package.
 - `pnpm lint`
 - `pnpm test`
 - `pnpm test:data`
+- `pnpm test:data:mysql` when `STUDIO_MYSQL_TEST_URL` or local Vitess/MySQL is available
+- `STUDIO_INCLUDE_HEAVY_LOCAL_TESTS=1 pnpm test` only when intentionally exercising heavyweight local suites excluded from the default aggregate
 - `pnpm test:checkpoint`
 - `pnpm build`
 - `pnpm check:exports`

Architecture/navigation-url-state.md

@@ -11,6 +11,7 @@ This architecture governs:
 - active Studio view (`table`, `schema`, `console`, `sql`, `stream`, `queries`)
 - active schema/table/stream
 - active stream follow mode
+- active stream request-observability sheet lookup
 - active stream aggregation-panel visibility
 - active stream aggregation range while the aggregation panel is open
 - pagination URL state
@@ -42,6 +43,7 @@ Only keys declared in [`ui/hooks/nuqs.ts`](../ui/hooks/nuqs.ts) are allowed:
 - `table`
 - `stream`
 - `streamFollow`
+- `streamObserve`
 - `aggregations`
 - `streamAggregationRange`
 - `filter`
@@ -61,6 +63,7 @@ Notes:
 - `pageIndex` remains URL-backed for table navigation.
 - `pageSize` remains a supported hash key for compatibility, but table rendering now takes its authoritative rows-per-page preference from `studioUiCollection.tablePageSize` in [`Architecture/ui-state.md`](ui-state.md).
 - `streamFollow` stores the active stream follow mode (`paused`, `live`, or `tail`).
+- `streamObserve` stores the active request-observability lookup for supported Streams profiles. Values serialize as `req:<requestId>`, `trace:<traceId>`, or `span:<spanId>`.
 - `aggregations` is an open-only flag for the active stream aggregation strip; when present it MUST be serialized as a bare key with no explicit value.
 - `streamAggregationRange` stores the active stream aggregation range, but MUST only be serialized while `aggregations` is present.
 
@@ -81,6 +84,7 @@ Adding a new URL key requires updating `StateKey` in `nuqs.ts` first.
 - `queries`: no standalone default; only meaningful when the current adapter provides query insights
 - `stream`: no default; only meaningful when `view=stream`
 - `streamFollow`: no global default in `useNavigation`; the active stream view MUST resolve an absent value to `tail` and materialize that into the hash
+- `streamObserve`: no global default in `useNavigation`; the active stream view MUST treat an absent or malformed value as a closed request-observability sheet
 - `aggregations`: no global default in `useNavigation`; the active stream view MUST treat an absent flag as closed and MUST NOT materialize that closed state into the hash
 - `streamAggregationRange`: no standalone default; the active stream view MUST clear it whenever `aggregations` is absent, and MUST materialize its default range only after the aggregation panel is opened
 

Architecture/non-standard-ui.md

@@ -145,6 +145,25 @@ It deliberately excludes:
   - The storage breakdowns also need collapsible ledger-style accounting boxes whose headers surface the section totals when folded shut, plus faint shared-cap annotations that sit beside right-aligned byte values and one shared cap marker spanning both Routing and Exact cache rows, which is not a stock ShadCN pattern.
   - No stock ShadCN pattern covers that descriptor-driven observability layout, especially when the UI must distinguish logical bytes from physical storage signals, separate search coverage from historical run indexes, hide unconfigured routing rows, and keep the remaining cost caveats explicit instead of inventing unavailable totals.
 
+### Stream Request Observability Sheet
+
+- Canonical components:
+  - [`ui/studio/views/stream/StreamObserveSheet.tsx`](../ui/studio/views/stream/StreamObserveSheet.tsx)
+  - [`ui/studio/views/stream/StreamObserveTimelineSection.tsx`](../ui/studio/views/stream/StreamObserveTimelineSection.tsx)
+  - [`ui/studio/views/stream/StreamObserveTraceSection.tsx`](../ui/studio/views/stream/StreamObserveTraceSection.tsx)
+  - [`ui/studio/views/stream/StreamObserveEventSection.tsx`](../ui/studio/views/stream/StreamObserveEventSection.tsx)
+  - [`ui/hooks/use-stream-observe-request.ts`](../ui/hooks/use-stream-observe-request.ts)
+- Closest standard ShadCN alternatives:
+  - `Sheet`
+  - `ToggleGroup`
+  - `Table`
+  - `Badge`
+  - `Skeleton`
+- Why it stays non-standard:
+  - The request detail surface needs to combine a merged event/span timeline, a trace waterfall with proportional span bars, expandable raw span details, service-call edges, root-cause event fields, source stream labels, and coverage warnings in one compact sheet.
+  - ShadCN provides the surrounding primitives, but no stock component models that request-correlation workflow or the proportional waterfall rows.
+  - The section selector still uses `ToggleGroup`, the shell uses `Sheet`, and the status chips use `Badge`; only the request-specific timeline and waterfall composition remain custom.
+
 ### Queries Live Table And Detail Sheet
 
 - Canonical component:

Architecture/request-observability.md

@@ -0,0 +1,125 @@
+# Request Observability Architecture
+
+This document is normative for Studio's request observability surface over Prisma Streams `evlog` and `otel-traces` streams.
+
+The feature is a stream-detail drilldown, not a standalone Studio view. It lets users expand an observability event or span row, open a request detail sheet, and inspect the correlated event, trace timeline, trace waterfall, errors, service calls, and partial-result warnings returned by Prisma Streams.
+
+## Scope
+
+This architecture governs:
+
+- detection of observability-capable stream profiles
+- URL-backed request lookup state
+- loading request correlation data from Prisma Streams
+- rendering the request detail sheet from a single correlation response
+- demo seeding for local request-observability validation
+
+## Canonical Components
+
+- [`ui/hooks/use-stream-observe-request.ts`](../ui/hooks/use-stream-observe-request.ts)
+- [`ui/studio/views/stream/StreamObserveSheet.tsx`](../ui/studio/views/stream/StreamObserveSheet.tsx)
+- [`ui/studio/views/stream/StreamObserveTimelineSection.tsx`](../ui/studio/views/stream/StreamObserveTimelineSection.tsx)
+- [`ui/studio/views/stream/StreamObserveTraceSection.tsx`](../ui/studio/views/stream/StreamObserveTraceSection.tsx)
+- [`ui/studio/views/stream/StreamObserveEventSection.tsx`](../ui/studio/views/stream/StreamObserveEventSection.tsx)
+- [`ui/studio/views/stream/StreamView.tsx`](../ui/studio/views/stream/StreamView.tsx)
+- [`demo/ppg-dev/seed-streams.ts`](../demo/ppg-dev/seed-streams.ts)
+- [`demo/ppg-dev/seed-streams-scale.ts`](../demo/ppg-dev/seed-streams-scale.ts)
+
+## Non-Negotiable Rules
+
+- Request observability MUST only appear for streams whose resolved profile is `evlog` or `otel-traces`.
+- Stream profile detection and request-pair descriptors MUST come from Streams metadata normalized by `useStreams` and `useStreamDetails`; feature code MUST NOT infer observability support from stream names.
+- The active lookup MUST be URL-backed through `streamObserve` and `useNavigation`; components MUST NOT write or parse `window.location.hash` directly.
+- `streamObserve` values MUST serialize as `req:<requestId>`, `trace:<traceId>`, or `span:<spanId>`.
+- Expanded event rows MAY expose the request-detail action only when the decoded event body has a usable request ID, trace ID, or span ID for the active profile.
+- Correlation loading MUST go through `useStreamObserveRequest`; view components MUST NOT call `/v1/observe/request` directly.
+- The request sheet MUST treat the Streams response as authoritative and surface `coverage.warnings` when present.
+- Missing counterpart streams MUST be explained in the sheet instead of rendering an empty trace or event section as complete.
+- The UI MUST use ShadCN primitives for the sheet, badges, buttons, skeletons, and section selector. The waterfall and timeline are custom request-observability composites and are documented in [`non-standard-ui.md`](non-standard-ui.md).
+
+## API Contract
+
+Studio expects the configured Streams base URL to expose:
+
+- `POST {streamsUrl}/v1/observe/request`
+
+The hook sends:
+
+```json
+{
+  "streams": {
+    "events": "app-events",
+    "traces": "app-traces"
+  },
+  "lookup": {
+    "requestId": "req_123"
+  },
+  "include": {
+    "events": true,
+    "trace": true,
+    "timeline": true
+  },
+  "limits": {
+    "events": 50,
+    "spans": 2000
+  }
+}
+```
+
+`lookup` contains exactly one of `requestId`, `traceId`, or `spanId`. If one counterpart stream is unavailable, Studio omits that stream and sets the matching include flag to `false`.
+
+The response is normalized into:
+
+- `lookup`: resolved request, trace, and span IDs
+- `summary`: title, method/path, service, environment, duration, status, level, and error summary fields
+- `evlog`: the primary event plus match count
+- `trace`: deduplicated spans, tree, critical path, errors, service map, and partial-state metadata
+- `timeline`: merged event/span timeline items
+- `coverage`: searched sides and warnings
+
+The sheet renders three sections from that one response:
+
+- `Timeline`: merged event, span-start, span-event, and exception items
+- `Trace`: waterfall rows, span details, errors, and service calls
+- `Event`: primary evlog event, root-cause fields, and raw JSON
+
+## Pairing Model
+
+When the active stream is `evlog`, Studio uses that stream as the event stream. Its trace counterpart MUST come from `details.observability.request.tracesStream`.
+When the active stream is `otel-traces`, Studio uses that stream as the trace stream. Its event counterpart MUST come from `details.observability.request.eventsStream`.
+
+If the descriptor is absent, Studio may still open the sheet for the active stream side and MUST explain the missing event or trace side. Studio MUST NOT choose the first stream with the opposite profile.
+
+## Demo Contract
+
+`pnpm demo:ppg` seeds two local profiled streams:
+
+- `app-events` with the `evlog` profile
+- `app-traces` with the `otel-traces` profile
+
+The seed data MUST include successful requests, failed requests with root-cause fields, slow requests, event-only requests, trace-only requests, and at least one deeper multi-service trace that exercises nested service calls, repeated network spans, and downstream worker/service spans. The demo also starts a ticker that appends fresh correlated requests so `Tail` mode and request-detail refresh can be exercised locally.
+
+The demo MUST create both streams with `Content-Type: application/json` before profile installation, and the installed profiles MUST declare their request-observability counterparts.
+
+`pnpm demo:ppg:seed-scale -- --streams-url <url>` appends deterministic scale data to the same two streams. It MUST use the shared seed builder so local performance checks exercise the same profile shape as `pnpm demo:ppg`.
+
+## Forbidden Patterns
+
+- matching observability streams by hard-coded stream names in the UI
+- storing request-detail data in component-local state outside React Query
+- adding request-observability methods to the database adapter
+- hiding coverage warnings or partial trace state
+- inventing request IDs from arbitrary payload text
+- adding a standalone request-observability route before the stream row workflow needs one
+
+## Testing Requirements
+
+Request observability changes MUST include tests for:
+
+- lookup param serialization and parsing
+- event-row lookup extraction for both `evlog` and `otel-traces`
+- descriptor-based counterpart stream resolution without first-profile fallback
+- `useStreamObserveRequest` request body, disabled state, failure state, and response normalization
+- sheet loading, warning, timeline, trace, event, missing-stream, and close behavior
+- stream-row affordance visibility and URL-backed sheet opening
+- demo seed shape, profiled stream creation, and scale-seed batch generation

Architecture/stream-event-view.md

@@ -22,6 +22,7 @@ This architecture governs:
 - URL-backed stream follow mode selection
 - URL-backed stream search term state
 - URL-backed stream routing-key selection state
+- URL-backed request-observability lookup state
 - URL-backed aggregation-panel visibility and aggregation range selection
 - batched reveal of newly arrived events
 - transient highlighting of newly revealed event rows
@@ -34,6 +35,7 @@ This architecture governs:
 - [`ui/hooks/use-stream-events.ts`](../ui/hooks/use-stream-events.ts)
 - [`ui/hooks/use-stream-details.ts`](../ui/hooks/use-stream-details.ts)
 - [`ui/hooks/use-stream-aggregations.ts`](../ui/hooks/use-stream-aggregations.ts)
+- [`ui/hooks/use-stream-observe-request.ts`](../ui/hooks/use-stream-observe-request.ts) (`useStreamObserveRequest`)
 - [`ui/hooks/use-ui-state.ts`](../ui/hooks/use-ui-state.ts)
 - [`ui/hooks/use-navigation.tsx`](../ui/hooks/use-navigation.tsx)
 - [`ui/studio/views/stream/StreamView.tsx`](../ui/studio/views/stream/StreamView.tsx)
@@ -158,6 +160,9 @@ The stream view MUST treat that latest metadata count separately from `visibleEv
 - while the suggestion panel is open, background stream refreshes MUST NOT rewrite the suggestion content underneath the user's keyboard navigation; only explicit input changes may do that
 - keyboard navigation inside the suggestion panel MUST keep exactly one suggestion visually selected at a time and MUST scroll the active row into view as the highlight moves
 - when `useStreamDetails` exposes one or more aggregation rollups, the header MUST render a sibling icon-only aggregation toggle button with an accessible label instead of a numbered text pill
+- when the active stream profile is `evlog` or `otel-traces`, an expanded event row MAY render a request-detail action if the decoded event body has a request ID, trace ID, or span ID usable by that profile
+- clicking the request-detail action MUST write the serialized lookup into `streamObserve` through `useNavigation`; it MUST NOT keep the request sheet open state only in component-local state
+- when `streamObserve` contains a valid lookup for a supported profile, the stream page MUST render the request-observability sheet and resolve counterpart streams from `useStreamDetails().details.observability`
 - the aggregation toggle open/closed state MUST be URL-backed through `useNavigation`
 - that header count SHOULD fall back to the rollup-definition count from `useStreamDetails`, but once aggregate window data has loaded it MUST prefer the resolved aggregation-series count so metrics-style rollups report their real card count
 - the list remains bounded by `visibleEventCount` until the user reveals newer events
@@ -263,6 +268,7 @@ Stream navigation chrome MUST be URL-backed through `useNavigation` with keys su
 
 - `streamFollow`
 - `streamRoutingKey`
+- `streamObserve`
 - `aggregations`
 - `streamAggregationRange`
 - `search`
@@ -287,6 +293,7 @@ The infinite-scroll `pageCount` and `visibleEventCount` are view-local transient
 - introducing stream-event URL pagination params
 - allowing more than one expanded row at a time
 - fetching aggregation rollups or aggregate windows directly inside `StreamView` without going through the dedicated hooks
+- fetching request-observability correlation directly inside `StreamView` without going through `useStreamObserveRequest`
 - deriving fake indexed fields from arbitrary payload properties
 
 ## Testing Requirements
@@ -306,6 +313,7 @@ Changes to this architecture MUST include tests for:
 - expanded-row match highlighting for stream search
 - aggregation-rollup request normalization in `useStreamAggregations`
 - stream-view aggregation toggle plus range switching, including range cleanup when the panel closes
+- stream-view request-observability affordance and URL-backed sheet state
 - infinite-scroll page growth behavior for both older history and newly revealed events
 - stream-view transient highlighting for newly revealed rows, including automatic clearance
 - stream navigation into `view=stream`