Skip to content

Lazy remote POD5 reading over range requests (#83, phases 0–1) - #153

Open
jayhesselberth wants to merge 1 commit into
mainfrom
feat/issue-83-remote-reading
Open

Lazy remote POD5 reading over range requests (#83, phases 0–1)#153
jayhesselberth wants to merge 1 commit into
mainfrom
feat/issue-83-remote-reading

Conversation

@jayhesselberth

Copy link
Copy Markdown
Member

Implements #83 through phase 1. No POD5 format change — this is entirely
reader-side I/O plumbing; bytes on disk are untouched, and the write path is
not involved at all.

What this enables

cargo install escapepod-cli --features remote

escpod inspect summary s3://my-bucket/run1.pod5
escpod view https://example.org/data/run1.pod5

summary, view, and inspect accept s3://, gs://, az://, and
http(s):// URLs wherever they accept a path. Opening transfers only the file
tail and footer; the command then fetches just the reads table. Inspecting a
multi-GB object costs a few MB of range GETs instead of a full download.

It is behind a flag

remote is a non-default Cargo feature, declared at each layer and
forwarded down:

  • escapepod-pod5: remote = ["dep:object_store", "dep:tokio", "dep:url"]
  • escapepod-signal: remote = ["escapepod-pod5/remote"]
  • escapepod-cli: remote = ["signal", "escapepod-signal/remote"]

A default build compiles no object_store, no tokio, no reqwest — and
still recognises a URL, so it refuses one with an explanation rather than
reporting a missing path:

$ escpod inspect summary s3://bucket/x.pod5      # built without --features remote
Error: Remote input s3://bucket/x.pod5 needs the `remote` feature; this binary
was built without it. Rebuild with `cargo install escapepod-cli --features
remote`, or download the file first.

Phase 0 — ByteSource (no behaviour change)

Reader no longer holds an Mmap; it holds an Arc<dyn ByteSource> and asks
for byte ranges, which come back as refcounted bytes::Bytes.

The local path stays zero-copy: MmapSource::read_range slices a Bytes view
over the mapping, so a range is a refcount bump, not a copy. There is a test
asserting the returned pointer is inside the mapping.

I deviated from the issue's plan in one way, for the better. The issue proposed
migrating &[u8]Bytes crate-wide and accepted a breaking API change
(SignalExtractor<'a> losing its lifetime). Instead the Reader caches each
embedded table's Bytes in a OnceLock and keeps handing out &[u8] borrowed
from &self. That preserves the ownership story a remote source needs while
leaving every existing signature unchangedmerge.rs, filter.rs,
signal_extractor.rs, read_iter.rs, and the CLI compiled untouched. It also
means each table is fetched at most once, which is what a source with real
per-request cost wants anyway.

Footer parsing was split so it can run from the tail alone:

  • footer_body_range(trailer, file_len) — given the last 32 bytes and the
    object size, returns the absolute range holding the footer.
  • parse_footer_region(region) — parses that range.

Reader::from_source reads one 64 KiB tail, which in practice contains both,
so a remote open is 2 round trips (head + tail).

Phase 1 — RemoteSource

object_store behind a single process-wide tokio runtime bridged with
block_on. Process-wide is load-bearing: rayon workers call into the reader
concurrently, so a per-call runtime would be both ruinous and prone to nesting
panics. read_ranges is overridden to use get_ranges, which coalesces
adjacent ranges.

Two things worth calling out:

  • Cleartext HTTP. object_store refuses http:// by default. When the
    user typed http:// themselves that refusal is noise, so allow_http is set
    for that scheme only; https:// and the cloud schemes keep the strict
    default, and a cleartext S3 endpoint still requires AWS_ALLOW_HTTP=true.
  • Error messages. object_storereqwesthyper each Display only
    their own layer, so the real cause hides two links down — the failure that
    led to the allow_http fix surfaced as a bare builder error. Remote errors
    now render the full source chain.
  • Azure needs an account name. A test asserting that every scheme
    is_remote_url advertises is one parse_url_opts can actually route caught
    that az://container/path carries no account, so store construction needs
    AZURE_STORAGE_ACCOUNT_NAME — documented, and the test now supplies the
    minimum config per backend rather than assuming all seven build bare.

Verification

  • cargo nextest run --workspace: 396 passed. cargo test --doc --workspace:
    19 passed. cargo nextest run -p escapepod-pod5 --features remote green.

  • cargo clippy --workspace --all-targets and the same with
    --features escapepod-cli/remote: zero warnings. cargo fmt --check clean.

  • End-to-end HTTP parity, reproducible via scripts/test_remote_http.sh
    (added). It serves a POD5 over a Range-capable HTTP server — python -m http.server can't be used, it ignores Range — and diffs local vs remote
    output while counting bytes actually transferred:

    file: data/drna/yeast_trna_reads.pod5 (1771984 bytes)
    ok    escpod inspect summary: identical output; 98100 bytes in 4 requests (5.5% of file)
    ok    escpod inspect reads:   identical output; 98092 bytes in 4 requests (5.5% of file)
    ok    escpod view:            identical output; 98092 bytes in 4 requests (5.5% of file)
    

    The four ranges are the leading signature (8 B), the 64 KiB footer probe, the
    run-info table, and the reads table. The signal table is never fetched
    which is the whole claim.

Also fixed along the way

  • POD5 footer parsing now rejects a corrupt footer length instead of computing
    an out-of-range slice offset. A negative length, or one placing the footer
    magic before the file's leading signature, is a clean InvalidFooter error.
    These bytes come straight from an untrusted file.
  • Two escapepod-pod5 tests pointed at ../data/… instead of ../../data/…
    and had been silently skipping their assertions for as long as they've
    existed. They now actually run (visible as Parsed 2 batches from 1698.31 KB signal table in test output).

Not in scope (issue phases 2–3)

Signal is still fetched a whole table at a time, so demux, resquiggle,
repack, merge, and filter would pull essentially the entire object over
the network. Documented as such, with the advice to download first. Per-batch
lazy signal is phase 2. Remote access is read-only; there is no remote write
path.

… 0-1)

Read POD5 objects straight from S3/GCS/Azure/HTTPS instead of a local path,
fetching only the ranges a command actually needs. No POD5 format change --
this is reader-side I/O plumbing; the write path is untouched and remote
access is read-only.

    escpod inspect summary s3://my-bucket/run1.pod5
    escpod view https://example.org/data/run1.pod5

Gated behind a non-default `remote` Cargo feature at every layer
(escapepod-pod5 -> escapepod-signal -> escapepod-cli). A default build pulls
in no object_store/tokio/reqwest at all, and still recognises a URL so it can
refuse one with an explanation rather than reporting a missing path.

Phase 0 -- ByteSource. Reader holds an Arc<dyn ByteSource> instead of an Mmap
and asks for byte ranges, returned as refcounted Bytes. The local path stays
zero-copy: MmapSource slices a Bytes view over the mapping, so a range is a
refcount bump rather than a copy.

Rather than the crate-wide &[u8] -> Bytes migration the issue proposed (which
would have broken SignalExtractor<'a>), Reader caches each embedded table's
Bytes in a OnceLock and keeps handing out &[u8] borrowed from &self. That
gives a remote source the ownership it needs while leaving every existing
signature unchanged -- merge.rs, filter.rs, signal_extractor.rs, read_iter.rs
and the CLI all compiled untouched -- and it fetches each table at most once.

Footer parsing splits so it can run from the tail alone: footer_body_range()
turns the fixed 32-byte trailer plus the object size into the absolute range
holding the footer, and parse_footer_region() parses it. Reader::from_source
reads one 64 KiB tail, which in practice contains both, so a remote open is
two round trips.

Phase 1 -- RemoteSource over object_store, bridged into the synchronous Reader
by a single process-wide tokio runtime. Process-wide is load-bearing: rayon
workers call in concurrently, and a per-call runtime would be ruinous and
prone to nesting panics. read_ranges() is overridden to use get_ranges, which
coalesces adjacent ranges.

object_store refuses cleartext HTTP by default; when the user typed http://
themselves that refusal is noise, so allow_http is set for that scheme only.
https:// and the cloud schemes keep the strict default, and a cleartext S3
endpoint still requires AWS_ALLOW_HTTP=true. Remote errors now render their
full source chain -- object_store/reqwest/hyper each Display only their own
layer, so the real cause hid behind a bare "builder error".

Verification: scripts/test_remote_http.sh serves a POD5 over a Range-capable
HTTP server (python -m http.server ignores Range) and diffs local vs remote
output while counting bytes transferred. All three commands produce identical
output pulling 98 KB of a 1.77 MB file in 4 requests -- signature, footer
probe, run-info table, reads table. The signal table is never fetched.

Signal is still fetched a whole table at a time, so demux/resquiggle/repack/
merge remain a poor fit for a remote object; documented as such. Per-batch
lazy signal is issue phase 2.

Also fixed:
- Footer parsing rejects corrupt footer lengths instead of computing an
  out-of-range slice offset. These bytes come from an untrusted file.
- Two escapepod-pod5 tests pointed at ../data/ instead of ../../data/ and had
  been silently skipping their assertions; they now run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant