perf: bundle all ongoing optimization work - #2272
Closed
SBrandeis wants to merge 13 commits into
Closed
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
A set-associative cache keyed on the word bytes, plus a scratch pile reused across encode calls, so BPE, WordPiece and Unigram all stop re-tokenizing words they have already seen. Squashed from feat/bpe-cache-min, which also carries #2261 (perf(scratch): reuse Scratch buffers across encode() calls).
A vectorized literal scanner in atomsplit, wired into the Replace normalizer and the Split pre-tokenizer so literal patterns stop going through the regex engine. Squashed from feat/literal-simd.
… the vocabulary (#2266) SentencePiece models hand the model one whole chunk. Where the vocabulary proves no merge can cross a word boundary, cut there and tokenize the words independently. Squashed from perf/metaspace-proven-cuts.
…r pass Prepend + Replace (or a lone Replace) collapse into a single MetaspaceNormalizer at pipeline build time, so the text is walked once instead of twice. Squashed from perf/fuse-metaspace (no PR).
The fuse (perf/fuse-metaspace) and the cuts (perf/metaspace-proven-cuts) were written on branches that did not know about each other. Both detections read the declared config, so they compose, but nothing asserted that: an encode oracle failure would only show if one of them also changed ids. This pins the pipeline shape for the real llama-2 and gemma-4 configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports the char-fold idea from Arthur's tables engine (#2241): at build time, prove from the vocabulary which multi-byte characters always assemble into one token (lowest-rank replay, no boundary neighbour able to pre-empt a step), and seed the merge loop with that token instead of 2-3 byte symbols. Differences from #2241: external ids throughout (no internal renumbering), seeding feeds the existing exact merge loop instead of the WIP multipass engine, and the WordCache above it is untouched. The pair table / grid parts stay behind until that engine is green. llama-3 folds 298 characters (185 CJK); model stage on the jpn fixture, cache off: 37.0 -> 29.3 ns/B (-21%). Ids pinned equal with the fold blanked (llama-3 A/B over CJK/Greek/Cyrillic/emoji corpora) and by the 9/9 encode oracle; thief and non-BMP cases covered in unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MetaspaceNormalizer swapped spaces by restarting a memchr search at every space and guessed the rewrite's size. It now counts the spaces first (count_matches), so the rewrite is sized exactly and a text without spaces is handed back without a scan of its own, and streams the swap through for_each_match. Normalize stage 4-7% faster on llama-2/gemma-4, e2e +3-5% (4 interleaved process rounds), ids unchanged. The drop_whitespace path sizes its rewrite from the word spans instead of the same guess. split_literal in pipeline.rs is now the one streaming literal split: Split's plain-string arm and CharDelimiterSplit both call it. CharDelimiterSplit loses its per-call span buffer (one heap Vec per chunk) and atomsplit's memchr-restart recipe struct, which the SIMD Literal scanner superseded, is deleted. Gates: 352 tk-encode + 20 atomsplit tests green, pipeline_oracle 9/9 against released 0.23.1, fmt and clippy clean (the 8 parallelism.rs dead-code warnings predate this change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pack_word built its u128 key by zeroing a 16-byte buffer and copying the word in with a variable length, which LLVM lowers to a memset and a memcpy libcall. One pair per looked-up word put a third of gpt2's whole encode inside libsystem_platform (samply: 27.5% memset + 6.4% memmove, both under WordCache::lookup). Two fixed-width reads, one from each end of the word, produce the same value with no calls: the overlapped middle bytes are identical, so or-ing the halves is harmless, and every byte above the length stays zero for key equality. pack_word_matches_the_buffer_form pins the new form to the old one for every packable length and byte pattern. Measured (interleaved process rounds, ids identical, oracle 9/9): gpt2 +38/+30% (eng/agentic), llama-3 +39/+33%, glm +33/+31%, gpt-oss +23/+19%, mistral +5/+21%, llama-2 +16/+5%, gemma-4 +15/+12%. An 8-way co-runner test holds per-process throughput within 3% of solo, so the encode loop stays core-bound, not memory-bound. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SBrandeis
force-pushed
the
perf/all-optims
branch
from
July 31, 2026 23:51
4481600 to
8092462
Compare
The staged path collects every span of a chunk into a buffer, then walks the buffer calling the model through the per-span (model, scratch) dispatch match: a span-buffer round trip and a dispatch per word. Every FSM-routed pre-tokenizer over a BPE model (the gpt2, cl100k-family, o200k, tekken and deepseek shapes: all eight byte-level benchmark models but the two SentencePiece ones) now runs fused instead: encode drives the scan_* form of the FSM, which hands each span to the model the moment it is cut. No span buffer, dispatch settled once per chunk, and the model reads each word while its bytes are still hot from the scan. Every fsm_* keeps its no-push slice form as a thin wrapper over the new scan_* (one emit closure writing spans in place), so the existing byte-exactness tests keep pinning both forms at once. PipelineBPE::tokenize_span is the per-word body factored out of the Model trait impl, so the fused and staged paths share one body and the ignore_merges branch keeps living in exactly one place. Routing asks the pre-tokenizer itself: Split::native_fsm and Sequence::native_fsm apply the same recognition their pre_tokenize applies (deepseek's three-Split composition, the identity-children collapse), so the fused check cannot disagree with the split it bypasses. Measured with a flat encode loop, interleaved process rounds (eng_Latn / agentic_swe): gpt2 +17%, llama-3 +15/+16%, glm-5.2 +14/+15%, gpt-oss +14/+15%, mistral-small-4 +12/+17%, deepseek-v4 +11/+9%. llama-2 and gemma-4 do not route (ProvenCuts / literal splits) and are unchanged. Ids identical on every fixture; oracle 9/9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-run scalar walk of every regex-shaped pre-tokenizer FSM (byte-level, cl100k at digit caps 1/3/unbounded, o200k, tekken, deepseek) with 64-byte batch scanners behind one shared walker: per- class bitmasks built from the classify tag stream, token starts computed by shifted-mask algebra in u64 registers (transcribed per scheme from gigatoken's scanners, MIT), and everything the algebra cannot prove locally deferred to the scheme's scalar advance, extracted from each scan so both paths share one body. Feeding the masks from tags instead of raw bytes keeps non-ASCII batches on the fast path. Bad zones: char-counted digit groups over multi-byte chars, mark chars (run- contextual class), upper-after-caseless case splits, contraction and whitespace batch-edge straddles, multi-byte whitespace, and any deepseek batch containing a CJK-range char (tested before the tag masks are built, so CJK text degenerates to the scalar scan plus one test). The walker interleaves proven starts with bad zones: a span must never be emitted across one. The per-target surface is one 64-byte block classifier each for NEON, SSE2 and wasm32 simd128; other targets fall back to the scalar scans. FSM stage on eng_Latn/agentic_swe: byte_level 3.5x/2.5x, cl100k 2.8x/1.9x, o200k 3.2x/1.9x, tekken 3.3x/2.1x, deepseek 3.1x/1.7x; cmn_Hani 1.2-1.9x with deepseek at parity by construction. Encode e2e, process-alternated rounds with identical token streams: gpt2 249 -> 321 MB/s eng (+29%), +11% swe; llama-3 +28%/+16% (cmn -1.6%, noise); glm-5.2 +27%/+13%; gpt-oss +49%/+26%; mistral +46%/+25%; deepseek +52%/+23% (fancy-regex builds both sides). Byte-exactness pinned per scheme by padding-sweep differentials (every 64-byte-edge offset, also run on x86_64) and byte-exact spans over three full fixtures; pipeline_oracle 9/9 vs the released crate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ting them A normalizer whose rewrite changes the text one character at a time does not have to write it: positions map one to one, so downstream stages can read the rewritten form off the raw text. `PendingRewrite` names that class of rewrite (every `from` reads as `to`, one may be prepended), and `Normalizer::pending_rewrite` lets every normalizer say whether it is one: the fused MetaspaceNormalizer (the SentencePiece space swap) and a single-character literal Replace are, everything else keeps writing. `NormalizedText::from_chain` runs the chain and hands a trailing pending rewrite back unwritten; `write` produces the text for consumers that need the bytes, and MetaspaceNormalizer's own swap now goes through the same writer. The consumer is the zero-copy encode path, built when a chain ends in the space swap over proven cuts into a char-atom BPE with no `normalized` added tokens (llama-2, gemma-4). `ZeroCopyMetaspace` finds the same cuts on the raw text (one memchr2 pass over `from` and `to`), and the model reads each raw span through a `CharSwap`, seeding the delimiter's id for every raw `from`. The word cache keys on the raw bytes; a rewritten form never holds a `from`, so overlapping keys always agree on their ids. Only the one word a prepend touches is still rewritten for real. The scan compares single bytes, not runtime-length slices: a slice compare of run-time width compiles to a memcmp call per candidate, measured at 15-20% of the whole win. The veto is taken at its prefilter's word: a cut whose preceding byte could open a veto piece is skipped instead of checked against rewritten bytes this path never builds. Skipping a cut never changes the ids, and the byte fires for under 1% of spaces on the worst corpus (gemma-4's one piece, code text). Same-binary interleaved A/B (examples/normalize_claims.rs, 512 kB per corpus, warm cache): llama-2 +10-64%, gemma-4 +29-75% encode throughput across eng/cmn/code at 10 kB and per-line inputs. Ids match the written path over every corpus, and the released-crate oracle stays green 9/9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…akes The stage breakdown drove `encode_generic::<STAGE>`, which gated the fused scan at `STAGE_MODEL` and the zero-copy route at `STAGE_POSTPROCESS`. Below those the ladder fell back to the staged array FSM, so on every native-FSM model the `pre_tokenize` bar timed a splitter no shipping encode uses, and the `model` bar was a subtraction across two different pipelines (fused split+model minus unfused split), clamped at zero so the pathology never showed. The same held for the metaspace models: their whole zero-copy win landed in `post` or vanished into the clamp. `pretok_vs_regex` inherited the bad number and undersold the split against onig/fancy/pcre2/logos by roughly 2x. Both routes are now STAGE-aware, so every rung runs the route a full encode runs. A fused pipeline's split rung drives the same masked FSM scan the model rung does and emits each span into `pre_tokens`; a zero-copy pipeline's split rung is its proven-cut pass, with neither the rewrite write nor the added-token scan over rewritten text that the route never performs. That last one also fixes the frame rung, which was charging zero-copy models for a normalized added-token scan they never run. Measured on gpt2 (10 kB warm chunks, 5 reps, M3 Max), ns/byte: pre_tokenize eng 1.73 -> 0.57 swe 1.28 -> 0.61 cmn 1.23 -> 1.20 vs onig eng ~20x -> 63.8x Production throughput is unchanged: threading `pre_tokens` into the fused emit adds a closure capture that folds away in the `STAGE_MODEL` specialization. Alternating two binaries at process level over 5 rounds x 7 reps, 4 models x 2 fixtures, the spread is -1.6% to +2.5% with no systematic direction, inside this bench's noise floor. Also reported so the chart cannot be read as more complete than it is: - `route` per model, since the three decompose differently. - `clamped`, the residual the five stages do not account for. A rung at or below the noise floor goes negative and clamps, and a `clamped` far from zero means that fixture's breakdown should not be trusted. - `alloc`, the per-chunk output/span buffer allocation `encode` pays and the reused-buffer rungs do not. It measures 0.07 to 0.11 ns/byte on gpt2, so `Vec` growth is not the gap between the ladder's total and the throughput phase. Timing the real `encode` to close that gap was tried and dropped: phase 2 shares one pipeline across fixtures, its scratch pool saturates, and the delta came out at +0.9 to +7.2 ns/byte measuring cache eviction rather than call overhead. The remainder is left stated and unattributed instead. The split rung pays one span write per pre-token that a fused encode does not, worth 0.03 to 0.07 ns/byte (scan with a span push 0.638 vs with an xor 0.610 on gpt2/eng), and the model rung carries the checked `&str` slice, worth 0.15 to 0.27. Both are documented on `stage_secs`. Rendering the chart then turned up three readability defects the JSON hid. A row's stages sum past 100% wherever a rung clamped, and the bars were drawn against a fixed `PLOT_W`, so those rows ran off the right edge: ell_Grek at 108% and added_normalized_sparse at 109% were cut off, which reads as a broken chart rather than as noise. The plot is now scaled by the widest row, so an overflowing bar crosses the 100% gridline instead of leaving the canvas, and the subtitle carries the mean parts-sum. The header claimed "each bar = 100%", never quite true for the same reason, and now says 100% is that fixture's total. The new route note pushed the subtitle past the SVG width and clipped its tail, so stages under 2% mean go unlisted (on gpt2 that was "post 1% · normalize 0%"). Three tests pin the ladder, since `pipeline_oracle` only ever drives `STAGE_POSTPROCESS` and so proves ids without proving which path made them. The fused one asserts `encode_fused` returns true at `STAGE_SPLIT` rather than only comparing spans: the staged FSM produces identical spans, so span equality alone would not have caught this bug. Verified red by making the fused scan decline below `STAGE_MODEL`. 370 lib tests, `pipeline_oracle` 9/9 against released 0.23.1, clippy and fmt clean. `zero_copy_is_built_for_the_real_sentencepiece_configs` was already red at ac50768 and is untouched here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HorT6V8fzJXUmdJwJPXfS5
Collaborator
|
Superseded by #2279, which is the same intent — one tree with the ongoing optimization work — rebuilt on the current cache stack ( Geomean vs gigatoken there: gpt2 1.14×, llama-3 1.24×, i.e. ahead overall; 3.3× the base branch on English. The pieces are also split into reviewable PRs underneath it: #2276, #2277, #2278. Closing in favour of that — happy to reopen if there is anything in here it does not cover. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PipelineTokenizer benchmark
10 / 10 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) · ~10 kB inputs · add_special_tokens on · single thread + 1/2/4/8/max-thread sweep972adc0f4 · 2026-08-01 12:50 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 16 coresvs base branch (
d183afea5) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.Decode
Round-trip: v0.23.1
encode_fastproduces the id streams (same fixtures,add_special_tokens=true); both implementations decode those SAME ids withskip_special_tokens=false. MB/s counts decoded text bytes.bert-base-uncased — normalizer-heavy WordPiece · ×5.22 vs v0.23.1 · ×1.04 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 12+0 (peak 12) · Pipeline 8+2 (peak 17)
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×28.90 vs v0.23.1 · ×6.94 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 68) · Pipeline 84+0 (peak 84)
Pre-tokenize: our split vs regex engines. ns/byte, lower better.
splitis the splitter this model's route actually runs, straight off the stage ladder.scalar-clsswaps only the classify pass for its scalar version. It is not the split without SIMD, since a regex-shaped fsm is a boundary-mask scanner and carries SIMD of its own.×vs= engine ÷ our split (shipped / scalar-classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×15.22 vs v0.23.1 · ×8.54 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 304+0 (peak 371) · Pipeline 275+0 (peak 370)
gpt2 — gpt2 ByteLevel regex · ×33.61 vs v0.23.1 · ×4.16 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 28+0 (peak 28)
Pre-tokenize: our split vs regex engines. ns/byte, lower better.
splitis the splitter this model's route actually runs, straight off the stage ladder.scalar-clsswaps only the classify pass for its scalar version. It is not the split without SIMD, since a regex-shaped fsm is a boundary-mask scanner and carries SIMD of its own.×vs= engine ÷ our split (shipped / scalar-classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×26.20 vs v0.23.1 · ×5.20 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 241+0 (peak 315) · Pipeline 234+0 (peak 316)
Pre-tokenize: our split vs regex engines. ns/byte, lower better.
splitis the splitter this model's route actually runs, straight off the stage ladder.scalar-clsswaps only the classify pass for its scalar version. It is not the split without SIMD, since a regex-shaped fsm is a boundary-mask scanner and carries SIMD of its own.×vs= engine ÷ our split (shipped / scalar-classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×26.60 vs v0.23.1 · ×4.13 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 169+0 (peak 231) · Pipeline 170+0 (peak 232)
Pre-tokenize: our split vs regex engines. ns/byte, lower better.
splitis the splitter this model's route actually runs, straight off the stage ladder.scalar-clsswaps only the classify pass for its scalar version. It is not the split without SIMD, since a regex-shaped fsm is a boundary-mask scanner and carries SIMD of its own.×vs= engine ÷ our split (shipped / scalar-classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).llama-2 — model-bounded BPE, no pre-tokenizer · ×14.68 vs v0.23.1 · ×3.90 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 19+0 (peak 23) · Pipeline 23+0 (peak 23)
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×27.60 vs v0.23.1 · ×4.31 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 73+0 (peak 95) · Pipeline 93+0 (peak 95)
Pre-tokenize: our split vs regex engines. ns/byte, lower better.
splitis the splitter this model's route actually runs, straight off the stage ladder.scalar-clsswaps only the classify pass for its scalar version. It is not the split without SIMD, since a regex-shaped fsm is a boundary-mask scanner and carries SIMD of its own.×vs= engine ÷ our split (shipped / scalar-classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×23.22 vs v0.23.1 · ×4.58 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 152+0 (peak 194) · Pipeline 109+0 (peak 195)
Pre-tokenize: our split vs regex engines. ns/byte, lower better.
splitis the splitter this model's route actually runs, straight off the stage ladder.scalar-clsswaps only the classify pass for its scalar version. It is not the split without SIMD, since a regex-shaped fsm is a boundary-mask scanner and carries SIMD of its own.×vs= engine ÷ our split (shipped / scalar-classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).t5-base — Unigram + Metaspace · ×5.82 vs v0.23.1 · ×2.24 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 34+2 (peak 36) · Pipeline 61+2 (peak 66)