Skip to content

Add async raw S3 packet writes#124

Merged
cliffburdick merged 3 commits into
mainfrom
cburdick/async-raw-s3-writes
Jun 11, 2026
Merged

Add async raw S3 packet writes#124
cliffburdick merged 3 commits into
mainfrom
cburdick/async-raw-s3-writes

Conversation

@cliffburdick

Copy link
Copy Markdown
Collaborator

No description provided.

@cliffburdick cliffburdick force-pushed the cburdick/async-raw-s3-writes branch from ac6e515 to f90b6fa Compare June 4, 2026 22:43
@cliffburdick cliffburdick marked this pull request as ready for review June 9, 2026 21:01
@greptile-apps

greptile-apps Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces DAQIRI_ENABLE_S3, an opt-in CMake feature that adds asynchronous raw packet uploads to Amazon S3 or any S3-compatible object store, backed by the AWS SDK for C++.

  • New src/common_s3.cpp: a self-contained async S3 write path — per-packet PutObject pipeline with a mutex/condition-variable completion tracker, pinned-host staging buffers (cudaMallocHost), and a reference-counted AWS SDK lifecycle manager. Correct stub implementations are compiled when S3 is disabled.
  • Public API additions in include/daqiri/common.h: S3WriterConfig, S3WriteStatus, opaque S3Writer/S3WriteHandle types, and six free functions (daqiri_s3_writer_create, daqiri_write_raw_to_s3_objects_async, poll/wait/destroy variants).
  • Python bindings in python/daqiri_common_pybind.cpp: RAII PyS3Writer/PyS3WriteHandle wrappers with GIL-scoped release on all blocking calls, bound as daqiri.S3Writer and daqiri.S3WriteHandle.

Confidence Score: 5/5

Safe to merge. The S3 write path is additive and off by default; existing packet I/O paths are unaffected.

All three commits carry a DCO sign-off and use the required #124 - ... title format. The async completion logic (mutex, condition variable, inflight counter) is correct — daqiri_s3_write_wait holds the lock until done is set, preventing the handle from being deleted while callbacks are still live. Staging buffers use cudaMallocHost/cudaFreeHost correctly, release_memory() intentionally preserves size so that SetContentLength and the bytes_uploaded counter downstream remain accurate. The AWS SDK refcount is properly guarded. Python bindings release the GIL on every blocking call and the RAII destructors prevent double-free. Doc updates cover the new option. No burst-free violations, no backend-type branching in common code.

No files require special attention.

Important Files Changed

Filename Overview
src/common_s3.cpp New S3 write implementation — async PutObject pipeline with correct mutex/CV lifecycle, staging-buffer management, and AWS SDK refcount. Stub fns provided for disabled builds.
include/daqiri/common.h Adds S3WriterConfig, S3WriteStatus, S3Writer/S3WriteHandle opaque types, and six new API functions. Follows existing BurstParams-free-contract conventions.
python/daqiri_common_pybind.cpp Adds PyS3Writer/PyS3WriteHandle RAII wrappers with GIL-release on blocking calls; write_raw_objects correctly handles wait+destroy sequence without double-free.
src/CMakeLists.txt Adds DAQIRI_ENABLE_S3 CMake option; common_s3.cpp unconditionally compiled with stub fns when disabled; socket→rdma invariant untouched.
Dockerfile Builds AWS SDK for C++ S3-only from source; pinned version via AWS_SDK_CPP_VERSION ARG with sane defaults; SDK installed unconditionally in base image.
cmake/daqiriConfig.cmake.in Correctly propagates DAQIRI_ENABLE_S3 and calls find_dependency(AWSSDK COMPONENTS s3) for downstream CMake consumers when S3 is enabled.

Reviews (4): Last reviewed commit: "#124 - Guard S3 helpers in disabled buil..." | Re-trigger Greptile

Comment thread docs/api-reference/cpp.md
Comment thread src/common_s3.cpp
cliffburdick added a commit that referenced this pull request Jun 10, 2026
Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
@cliffburdick cliffburdick force-pushed the cburdick/async-raw-s3-writes branch from 857467d to 1f73c20 Compare June 10, 2026 23:07
Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
@cliffburdick cliffburdick merged commit b2484f8 into main Jun 11, 2026
5 checks passed
@cliffburdick cliffburdick deleted the cburdick/async-raw-s3-writes branch June 11, 2026 15:24
chloecrozier pushed a commit that referenced this pull request Jun 11, 2026
* #124 - Add async raw S3 packet writes

Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant