Skip to content

perf(s3): use spawn-based fetcher for S3 block source#125

Merged
sprites0 merged 1 commit intonode-builderfrom
fix/s3-path-style
Mar 16, 2026
Merged

perf(s3): use spawn-based fetcher for S3 block source#125
sprites0 merged 1 commit intonode-builderfrom
fix/s3-path-style

Conversation

@sprites0
Copy link
Collaborator

@sprites0 sprites0 commented Mar 16, 2026

Summary

  • Override collect_blocks on S3BlockSource with tokio::spawn batches instead of the default buffered() stream
  • Avoids single-task polling bottleneck, yielding ~10x speedup on fast networks (4s -> 3.1s in slow network, -> 0.3s in fast network per 10K blocks in testnet)
  • Keeps buffered() as the default trait impl (better for filesystem and other sources)
  • Both approaches result in the same number of file descriptors — hyper's connection pool opens sockets based on concurrent requests regardless of whether they're driven by buffered() or tokio::spawn

Test plan

  • Verify S3 block fetching performance on fast network
  • Verify local/hl_node block sources are unaffected (still use default buffered())

🤖 Generated with Claude Code

@sprites0 sprites0 force-pushed the fix/block-store-refactor branch from 383de5f to 6aac87b Compare March 16, 2026 02:38
@sprites0 sprites0 changed the base branch from fix/block-store-refactor to node-builder March 16, 2026 06:20
@sprites0 sprites0 force-pushed the fix/s3-path-style branch from 6f5f9d1 to f443875 Compare March 16, 2026 06:21
@sprites0 sprites0 changed the title feat(s3): path-style URLs and spawn-based concurrent fetching refactor(sources): replace buffered stream with spawn-based concurrent fetching Mar 16, 2026
@sprites0 sprites0 force-pushed the fix/s3-path-style branch from f443875 to c4b9d10 Compare March 16, 2026 13:10
@sprites0 sprites0 changed the title refactor(sources): replace buffered stream with spawn-based concurrent fetching perf(s3): use spawn-based concurrent fetching for S3 block source Mar 16, 2026
@sprites0 sprites0 force-pushed the fix/s3-path-style branch from c4b9d10 to c7944fd Compare March 16, 2026 13:14
@sprites0 sprites0 changed the title perf(s3): use spawn-based concurrent fetching for S3 block source perf(s3): spawn-based block fetching Mar 16, 2026
@sprites0 sprites0 changed the title perf(s3): spawn-based block fetching perf(s3): use spawn-based fetcher for S3 block source Mar 16, 2026
@sprites0 sprites0 force-pushed the fix/s3-path-style branch from c7944fd to 7ef573f Compare March 16, 2026 13:17
Override collect_blocks on S3BlockSource with tokio::spawn batches
instead of the default buffered() stream. This avoids the single-task
polling bottleneck and yields ~10x speedup on fast networks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sprites0 sprites0 force-pushed the fix/s3-path-style branch from 7ef573f to 92eb71a Compare March 16, 2026 13:18
@sprites0 sprites0 merged commit 5447bac into node-builder Mar 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant