Prefetched pretoken-cache encode pipeline + single-pass allocation fixes

[marcelroed]

Summary

Replaces the pretoken-cache HashMap with a purpose-built encode pipeline and fixes the allocation behavior that dominated first-pass encoding. Includes the front-cache commit (1c10270) from optimize-encode-front-cache in its history; the front cache itself is superseded by the new table.

• ShortPretokenCache (src/bpe/pretoken_cache.rs): open-addressing table with self-contained 32-byte entries (one cache line per probe vs. hashbrown's two), 2 MiB-aligned MADV_HUGEPAGE backing, and an explicit prefetch API.
• Chunked encode loop: memoized_encode pulls 256 pretokens per chunk through the new PretokenSpans::fill_spans_keyed interface — keys, hashes, and cache-line prefetches are derived in the pretokenizer's fused walker loop, so by probe time the lines are in L1 instead of costing a serial DRAM stall. Each Fast* scheme gets a fused out-of-line chunk fill; ~90 % of encodings (1–2 tokens) are stored inline in the entry's u64 value, eliminating the dependent token_arena load.
• Single-pass allocation fixes (src/batch.rs): chunk output Vecs are reserved once from a bytes/4 estimate instead of doubling from empty (which re-copied ~the final size per chunk with realloc churn across 255 threads), token slices are appended with extend, and the multi-GB flat gather buffer is MADV_HUGEPAGEd before first touch.
• Benchmark methodology: encode_doc now measures one cold round per process (restart for independent samples; ENCODE_ROUNDS opts into repetition). In-process repetition inherits worker pretoken caches, allocator arenas, and faulted pages, overstating steady throughput by ~45 % vs. a true first pass over fresh data — which is the workload that matters.

Benchmarks

Whole-document encode (cargo bench --bench encode_doc), GPT-2 vocab, 10 GB OWT as one document, 255 threads, AMD EPYC 7742 (Zen 2, AVX2). Single pass per process, 5 interleaved samples per variant:

variant	samples (MB/s)	mean
encode-perf baseline (0e27c71)	4565 / 4382 / 4489 / 4599 / 4575	4522
front cache (1c10270)	4486 / 4367 / 4872 / 4474 / 4996	4639
this PR (fb0be88)	6251 / 5995 / 6057 / 6164 / 6058	6105 (+35 %)

All variants produce identical token counts (2,268,672,027). The single-pass gain comes from the allocation fixes; the two cache designs measure within noise of each other on a first pass (with the allocation fix applied to both: 6034 vs. 5995 MB/s), because a first pass is not hit-path-bound. On warm re-encodes of the same input (retained worker pools, earlier 5-round protocol at 10 GB) the pipeline reaches 8–10 GB/s vs. 6.3–6.5 GB/s for the front cache, so the cache redesign is kept for workloads that re-encode.

Also measured and rejected: a vocab-seeded shared cache (prebuilt vocab-entry → token table shared across workers) was 7 % slower single-pass — first-pass misses live in the Zipf tail, and the extra probe per lookup costs more than the saved merges.

Testing

• memoized_encode_matches_reference_owt (ignored, run manually): token-for-token differential of the cached pipeline vs. the uncached BPE reference over 50 MB of OWT — all 11.4 M tokens match.
• New span_source_tests: every scheme's fused fill_spans_keyed reproduces its iterator's spans/keys/hashes exactly, including chunk-boundary and end-of-input handling.
• cargo test --release: 46 passed; the 6 failures are pre-existing missing-data-file failures on this host (~/data/tokenizers/*.tiktoken), unrelated to this change.
• Empty pretokens through the public API pack to key 0 and route to the long map (pack_pretoken_key gained the n == 0 guard, which also makes its 16-byte read provably in-bounds).