feat(request-trace-uploader): measure delivery latency

[kristinapathak]

Why

Request-record delivery latency needs two success-only views: end-to-end time from the earliest request-record timestamp when that timestamp is valid, and backlog time from confirmed segment close or seal to terminal S3 success.

Scope

• Export these OTLP histograms directly to the workload-scoped BYOO Collector:
  • nvcf_request_trace_uploader_earliest_record_to_s3_success_latency_seconds: terminal S3 success time minus the earliest valid embedded request-record timestamp in the closed segment.
  • nvcf_request_trace_uploader_segment_seal_to_s3_success_latency_seconds: terminal S3 success time minus the durable timestamp at which closed-segment discovery confirms the segment is sealed and eligible for upload.
• Record samples only after terminal S3 success. Failed, pending, quarantined, and source-delete-only outcomes do not produce a success-latency sample.
• Limit attributes to capture_type=trace|audit. Do not use request IDs, NCA IDs, filenames, paths, S3 identifiers, timestamps, or error text as attributes.
• Select the earliest valid embedded request-record timestamp. If no valid timestamp exists, or every candidate is malformed, omit the earliest-record histogram sample. Do not substitute filesystem timestamps, discovery time, or first-read time. The seal-to-success histogram remains available.
• Persist the selected earliest-record timestamp and confirmed seal timestamp in durable lifecycle state before terminal completion so a restart cannot reset or shorten either latency.
• Export through signal-specific OTLP metrics configuration. Do not expose a Prometheus endpoint or scrape path.

Tests

• Valid single-record and multi-record segments use the earliest valid request-record timestamp.
• Trace and audit segments produce only the bounded capture_type attribute.
• Missing and malformed embedded timestamps omit only the earliest-record sample and retain the seal-to-success sample.
• Pending, failed, quarantined, and delete-failure paths do not record success-latency samples.
• Restart recovery preserves both timestamps and produces identical success-latency values.
• OTLP integration coverage verifies metric delivery to BYOO without a Prometheus scrape endpoint.

Relationships

• feat(observability): export uploader telemetry via BYOO OTLP #1046 provides the uploader direct BYOO OTLP telemetry.
• feat(request-trace-uploader): add initial S3 segment upload #1047 provides the initial closed-segment S3 success path.
• feat(request-trace-uploader): persist S3 upload lifecycle state #1050 provides durable lifecycle state for restart-safe timestamps.

Parent: #1004