feat(otel): cxtx OTEL bootstrap, usage parser, LLM cost spans

[divySDM]

Summary

Adds OpenTelemetry observability to the cxtx wrapper for per-application LLM cost attribution. Server-side OTEL is intentionally deferred to a follow-up PR — this one ships the cxtx piece, where the cost-attribution KPI lives.

What's in

• New cxdb-otel crate (workspace top-level) — env-gated OTEL bootstrap. Fully inert when OTEL_EXPORTER_OTLP_ENDPOINT is unset (no subscriber installed, no exporter spun up); W3C TraceContext + OTLP/gRPC traces + metrics when set. Helpers for tiny_http extraction and reqwest header injection. Lazy gen_ai.* metric instrument creation.
• cxtx/src/otel/ — call context, finish-reason mapping (14 provider → canonical rows), derived buckets, finalize_llm_call (single emit site for chat <model> spans + gen_ai.client.token.usage histogram + gen_ai.calls counter, shared by Anthropic and OpenAI provider finalize paths).
• cxtx/src/provider/usage.rs — typed UsageOutcome parser covering the 16-cell Anthropic/OpenAI matrix (SSE + JSON, ChatCompletions + Responses, with/without include_usage) plus aborted-stream classification. Real token counts now stamp TurnMetrics instead of zeros.
• Wire-schema additions (additive, backward-compatible):
  • TurnMetrics.usage_status: Option<String> — msgpack tag 8, tags non-happy-path turns (not_reported, error:<class>).
  • ContextMetadata.tenant: Option<String> — msgpack tag 5, used for app.tenant OTEL attribution. Empty-string is treated as absent (no sentinels).
• HTTP client wraps every outbound call in inject_reqwest so W3C traceparent/tracestate flow downstream. Async delivery worker captures parent context at enqueue and threads it through retries via an explicit-context variant (ContextGuard is not Send).
• Replay-dedup invariance: session.rs normalization strips telemetry fields so OTEL attribution does not perturb HistoryItem equality. Pinned by regression test.
• 17 fixtures under cxtx/tests/fixtures/usage/ (16-cell matrix + aborted stream) with a redaction lint test.
• 6 new test suites: otel_emit, otel_noop, trace_continuity, usage_integration, usage_matrix, fixtures_lint.

What's NOT in (deferred)

• Server-side OTEL (request spans, route metrics, store/registry instrumentation, trace continuity on receiving end, XFF trust contract, app.tenant server-side stamping). The server's HTTP layer has substantial drift requiring careful merging — separate PR. Note: ContextMetadata.tenant (msgpack tag 5) is wire-only in this PR — the server's extract_context_metadata does not yet read tag 5 and the cached ContextMetadata struct in server/src/store.rs has no tenant field, so list/projection APIs do not surface the tenant. Will land alongside server-side OTEL in the follow-up.
• Validation pipeline (golden + wire OTEL test gates against an OTLP collector). Separate effort.
• Binary-protocol TRACE frame (capability bit + wire-format trailer for client → server context propagation over the binary protocol). No wire-protocol bump in this PR.

Configuration

OTEL stays fully inert by default. Operators opt in by setting OTEL_EXPORTER_OTLP_ENDPOINT (and standard companion env vars: OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, etc.). cxtx adds one app-specific env: CXTX_TENANT for app.tenant attribution.

Cardinality control: gen_ai.client.token.usage histogram drops app.session_id / app.user / app.wrapper_version from metric attribute sets (kept on spans).

Test plan

• cargo check --workspace --all-targets clean
• cargo test --workspace — 271 tests across 27 suites pass, including all 6 new cxtx OTEL suites + the new stream_aborted suite
• OTEL no-op regression: otel_noop confirms unset endpoint installs no subscriber
• Replay dedup invariance: assert_dedup_ignores_telemetry confirms CallContext and tenant do not affect HistoryItem equality
• Fixtures lint: fixtures_lint confirms no provider response bodies contain redactable tokens
• Live OTLP-collector smoke test (do once a tenant/collector is reachable)

🤖 Generated with Claude Code