Expose keep-last-N (and event count) under --warpdemux-compat so window width can be tuned

[jayhesselberth]

--warpdemux-compat hardcodes "110 t-test events, keep last 25". There is no way to vary the retained window without leaving compat mode entirely — and leaving it destroys the signal, so the effect of window width is currently unmeasurable.

Why this matters

In rnabioco/escapepod-models the 25-feature fingerprint is the accuracy ceiling for barcode classification: training volume is saturated (9.6x data buys +0.5pp) and soft labels are refuted, leaving "richer features / wider window" as the one untested lever. Our discriminability with compat mode is 0.374 against WarpDemuX's own reference of 0.43, so there may be headroom in the representation.

The problem, measured

I tried to test a wider window by leaving compat mode (--num-segments 50 --window-width 12). It collapses to chance:

arm	features	balanced recall@100
--warpdemux-compat (25 means)	25	0.9824
--warpdemux-compat --emit-dwell	50	0.9829
--num-segments 50 --window-width 12	50	0.2616
same + --emit-dwell	100	0.2598

0.26 on 4 classes is chance. The cause is that the non-compat path fingerprints a fixed sample range (--segment-start 1000 --segment-end 2000 defaults) rather than the boundary-anchored adapter that compat mode uses, so it is not a wider window — it is a different, signal-free region. That matches escapepod-models' earlier discriminability ladder (escpod default ≈ 0.03–0.078 vs compat 0.374).

So the two modes are not on a continuum, and "does a wider window help?" cannot currently be answered.

Ask

Expose the compat-mode constants as options, defaulting to today's values so nothing changes unless asked:

• --wdx-keep-last <N> (default 25) — how many trailing event means are retained
• --wdx-num-events <N> (default 110) — t-test event count

Equivalently, let --segment-start/--segment-end be specified relative to adapter_end so the non-compat path can address the same boundary-anchored region.

Either would make the window a measurable axis rather than a hardcoded constant.

Aside: --emit-dwell works, and is a clean null

For the record, since it may save someone else the experiment: --warpdemux-compat --emit-dwell composes correctly (51 columns, fp_0..24 + dwell_0..24). On 33,438 training reads / 11,145 paired test reads it made no difference — +0.0006 balanced recall, McNemar p=0.34. Dwell time is orthogonal to segment means in principle, but for these RNA004 tRNA barcodes it carries no additional class signal.

[jayhesselberth]

Correction to the --emit-dwell aside — that null is mis-specified, and it strengthens the ask

I wrote above that dwell time "carries no additional class signal." That overstates it. The arm was mis-specified, and the reason makes this issue more important rather than less.

--emit-dwell pairs dwell_i with fp_i at the same event index, which assumes both channels describe the same nucleotides. They don't. The motor protein's active site and the pore constriction are physically separated along the strand: current level is set by the bases in the constriction, dwell by motor stepping kinetics at the motor site. At any instant the two channels report on different nucleotides — in other contexts the dwell variation appears ~10 nt closer to the 3' end than the matching current signature.

RNA004 translocates 3'-end first, so that offset is ~10 events earlier in the signal. Compat mode keeps the last 25 events before adapter_end, so if a barcode's current signature occupies events [-25, 0], its dwell counterpart sits near [-35, -10] — largely outside the retained window. The 50-feature vector was pairing each position's current level with a dwell value from ~10 nt away, so a null there is expected regardless of whether dwell is informative.

Why this matters for the requested knob

A tunable --wdx-keep-last isn't only about "does a wider window help". It is currently the only way to test dwell correctly: retaining ~35 events would bring the offset dwell region inside the window, letting a model pair fp[-25:] against dwell[-35:-10].

A more targeted alternative, if you'd rather not widen the default: allow the dwell channel its own offset, e.g. --dwell-offset <N events> (default 0), emitting dwell_{i+N} alongside fp_i. That directly encodes the motor-to-constriction separation and keeps the feature width at 2N.

Either works. Without one of them the dwell hypothesis can't be evaluated from escpod at all — the workaround is two fingerprint passes with shifted boundaries and a manual column join, which is fragile and won't match what the runtime computes.

So please read the earlier table as "index-aligned dwell is null" rather than "dwell is uninformative".

[jayhesselberth]

Deprioritising this — a per-event discriminability sweep (scripts/dwell_sweep.py in escapepod-models) has since shown the knob would mostly buy noise.

Segmenting the full adapter into 110 t-test events and measuring between/within-class variance per event position:

window	share of total mean-discriminability
last 25 events (what compat keeps)	87.3%
last 15 events	83.0%
the 85 events compat discards	12.7%, ~0.01 each

The signal is sharply concentrated: segment means peak at 0.9240 at event 100 of 110 and fall to near-zero below event ~90. So a wider retained window has at most 12.7% of thinly-spread signal to gain, and the current choice of 25 is well placed — compat's own peak sits at fp_13 of 25, i.e. event ~98, matching the full-adapter peak at 100.

The dwell-offset motivation in my earlier comment is also weaker than I thought. Dwell's best single event scores 0.036 against the means' 0.924 — 3.9% — and a targeted arm using only the top dwell events was null on a held-out run (+0.0001, McNemar p=1.0, 44 vs 45 discordant). Worth noting that arm's summary table initially looked like a +0.0019 win; it turned out to be differential dropout, since 54 short-adapter reads have NaN dwell and silently left the dwell arms, and those reads score 0.2963 in the baseline.

Still a reasonable API cleanup if the constants get exposed for other reasons, but not worth doing for accuracy. Refs #158.

[jayhesselberth]

Re-raising this — my deprioritisation above was correct for WDX and is likely wrong for what comes next.

The 87.3%-in-the-last-25-events result that justified deprioritising was measured on WDX barcodes, which are short. The incoming nbc pool is 27 nt (with a 45-nt variant), emitting far more events than the fixed last-25 window can hold, and escapepod-models' dev-notes/barcode-pilot-screening.md states the window "must be widened" for exactly this reason.

So a tunable --wdx-keep-last moves from "mostly buys noise" to plausibly required, and the number should be chosen from a per-event discriminability sweep on real nbc data rather than inherited from WDX.

No action needed until that data exists, but please don't close this on the strength of my earlier comment. Tracked from rnabioco/escapepod-models#25.

[jayhesselberth]

Softening my re-raise from earlier today. The nbc barcodes are flanked by constant sequence, which makes a better window strategy available than a fixed trailing count: locate the flanks per read and name the barcode window directly in the boundaries CSV, which escpod demux fingerprint already honours (this is how the EDX work retargeted the 3' adapter — no Rust change needed).

If that works, a tunable --wdx-keep-last is not required for nbc after all; it becomes the fallback if per-read flank localisation proves unreliable. So: still worth keeping open, but please read my earlier "likely required" as "possible fallback". Tracked in rnabioco/escapepod-models#25.