Skip to content

feat(request-trace-uploader): add initial S3 segment upload #1047

Description

@kristinapathak

Why

The uploader scaffold only discovers closed Dynamo request-trace segments. The next increment should prove the narrow happy path before adding durable recovery and configurable policy complexity.

Scope

  • Read the next closed trace or audit .jsonl.gz segment from the configured shared volume.
  • Transform the segment into the S3 request-record envelope required by the destination contract.
  • Submit the segment to S3 and delete the source file only after a successful upload response.
  • On a failed read, transform, upload, or delete, retain the source file and return the error through structured logs, OTLP traces, and OTLP metrics.
  • Process one closed segment at a time. Do not add background queues, retries, persistent state, status polling, quarantine, or automatic deployment injection in this increment.

Tests

  • Closed-versus-active segment selection.
  • Successful read, transform, upload, and source deletion.
  • Read, transform, upload, and delete failure paths that retain the source and produce bounded telemetry.
  • Concatenated gzip input and destination-envelope tests.

Follow-on work:

  • Durable lifecycle state and failure quarantine.
  • Configurable S3 policies.
  • NCA audit payload drop.
  • Full observability and documentation.

Parent: #1004

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions