perf(scratch): reuse Scratch buffers across encode() calls - #2261
Open
SBrandeis wants to merge 2 commits into
Open
perf(scratch): reuse Scratch buffers across encode() calls#2261SBrandeis wants to merge 2 commits into
SBrandeis wants to merge 2 commits into
Conversation
`PipelineTokenizer::encode` built a fresh scratch every call, so every encode started by asking the allocator for buffers the last one had just freed. They are now borrowed from a pool held by the tokenizer and handed back on drop. A tokenizer encodes through `&self`, so the pool has to cope with several threads being in it at once: it keeps a pile of scratches behind a mutex, taken and returned one at a time. The lock is only held while moving a scratch in or out, never while encoding. One thread encoding in a loop gets the same scratch every time; N threads encoding at once leave N behind for later calls to use. Nothing needs clearing between calls: every model empties the buffers it uses where it uses them (`merge_all` drains the merge queue, `merge_word` clears the word, WordPiece clears its candidate), which is already what makes one scratch safe across the many words of a single encode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
SBrandeis
changed the title
SBrandeis
marked this pull request as ready for review
July 29, 2026 14:14
SBrandeis
added a commit
that referenced
this pull request
Jul 31, 2026
A set-associative cache keyed on the word bytes, plus a scratch pile reused across encode calls, so BPE, WordPiece and Unigram all stop re-tokenizing words they have already seen. Squashed from feat/bpe-cache-min, which also carries #2261 (perf(scratch): reuse Scratch buffers across encode() calls).
ArthurZucker
approved these changes
Aug 3, 2026
ArthurZucker
left a comment
ArthurZucker
left a comment
Collaborator
There was a problem hiding this comment.
lgtm and clean ty!
| for _ in 0..1000 { | ||
| pipeline.encode("hello", false).unwrap(); | ||
| } | ||
| assert_eq!(pipeline.scratch_pool.len(), after_burst); |
Collaborator
There was a problem hiding this comment.
we should check they are not empty? / properly modified?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PipelineTokenizer benchmark
9 / 10 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) · ~10 kB inputs · add_special_tokens on · single thread + 1/2/4/8/max-thread sweepcb8ff8b28 · 2026-07-29 17:16 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 8 coresvs base branch (
28b2b9633) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.Decode
Round-trip: v0.23.1
encode_fastproduces the id streams (same fixtures,add_special_tokens=true); both implementations decode those SAME ids withskip_special_tokens=false. MB/s counts decoded text bytes.bert-base-uncased — normalizer-heavy WordPiece · ×4.96 vs v0.23.1 · ×1.02 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 12+0 (peak 11) · Pipeline 8+0 (peak 17)
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×4.03 vs v0.23.1 · ×0.98 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 68) · Pipeline 83+0 (peak 82)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×1.87 vs v0.23.1 · ×1.09 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 304+0 (peak 371) · Pipeline 275+0 (peak 371)
gpt2 — gpt2 ByteLevel regex · ×7.65 vs v0.23.1 · ×1.00 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 28+0 (peak 28)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×4.83 vs v0.23.1 · ×1.02 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 241+0 (peak 315) · Pipeline 234+0 (peak 316)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×6.23 vs v0.23.1 · ×1.01 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 169+0 (peak 231) · Pipeline 170+0 (peak 232)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).llama-2 — model-bounded BPE, no pre-tokenizer · ×3.82 vs v0.23.1 · ×1.05 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 18+0 (peak 23) · Pipeline 24+0 (peak 24)
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×6.87 vs v0.23.1 · ×1.00 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 73+0 (peak 95) · Pipeline 93+0 (peak 95)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×5.40 vs v0.23.1 · ×1.01 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 152+0 (peak 194) · Pipeline 109+0 (peak 195)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).Not yet supported:
t5-base