Skip to content

feat(observability): export uploader telemetry via BYOO OTLP #1046

Description

@kristinapathak

Why

The request-trace-uploader needs its own operational telemetry through the workload-scoped BYOO Collector. A local Prometheus scrape endpoint is not the uploader telemetry contract.

Scope

  • Replace the uploader-local Prometheus registry and /metrics endpoint with direct, signal-specific OTLP export for metrics.
  • Export uploader structured logs directly through the signal-specific OTLP logs endpoint and protocol.
  • Configure tracing from the signal-specific OTLP traces endpoint and protocol. Use W3C context propagation and bounded server spans for health endpoints, then upload-operation spans as those operations land.
  • Use only OTEL_EXPORTER_OTLP_LOGS_ENDPOINT and OTEL_EXPORTER_OTLP_LOGS_PROTOCOL for logs, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL for metrics, and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL for traces. Support grpc, http, and http/protobuf.
  • Do not use OTEL_EXPORTER_OTLP_ENDPOINT as a fallback. It can describe Collector outbound configuration rather than the workload-to-Collector path.
  • Retain health-only /livez and /readyz endpoints. Rename the listener setting from METRICS_ADDR to HEALTH_ADDR.
  • Export only finite metric attributes. Do not emit request payloads, credentials, paths, filenames, upload IDs, request IDs, NCA IDs, or error text as metric attributes, trace attributes, or log fields.
  • Flush providers with a bounded timeout on SIGTERM. Export failures must not change readiness.

Tests

  • Direct OTLP delivery tests for logs, traces, and metrics for every supported protocol.
  • Unit tests for endpoint and protocol selection, W3C propagation, structured-log redaction, and bounded trace attributes.
  • Metric-reader tests for uploader lifecycle and backlog metric names with finite attributes.
  • NVCF integration coverage that proves the uploader receives signal-specific logs, metrics, and traces endpoint plus protocol for its workload-scoped BYOO Collector.

Non-goals

  • A PodMonitor or Prometheus scrape path for uploader metrics.
  • Sending request payload records through OTLP logs.

Parent: #1004

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions