bench(pipeline): add sebpop/upstream as a second reference series - #2221
Draft
SBrandeis wants to merge 3 commits into
Draft
bench(pipeline): add sebpop/upstream as a second reference series#2221SBrandeis wants to merge 3 commits into
SBrandeis wants to merge 3 commits into
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Bench the tokenizers crate from sebpop's performance branch (github.com/sebpop/tokenizers#upstream) alongside the released 0.23.1 in the pipeline benchmark: single-thread throughput per fixture, the multi-thread sweep, per-implementation memory footprint, minimal-binary size, and a report-only id-diff flag. The charts keep a single vs-release axis — sebpop renders as a second (green) bar next to the pipeline's. The git dep uses sebpop's default features minus mimalloc: its override feature would swap the global allocator for the whole bench process and skew every series. Cargo.lock pins the resolved commit; `cargo update tokenizers@0.22.3-dev.0` moves it to the branch tip. Older cached bench JSONs (base-branch baselines) without the sebpop keys still render unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plain `encode` tracks offsets and silently bypasses sebpop's fused byte-level fast path (gated on OffsetType::None behind encode_fast / encode_batch_fast — the API his blog numbers measure). Switch both references to `encode_fast`, the same offsets-free regime the PipelineTokenizer plays in, so all three series answer the same question. New `bench-mimalloc` feature installs mimalloc as the bench binary's global allocator — the allocator sebpop's branch ships by default — uniformly for all three series, since one process can't give each series its own allocator. CI enables it for the bench shards only (not binsize: that chart measures the library, not the allocator). Local gpt2 check (M3 Max, 18 fixtures): ids match across all three implementations, fused path included; sebpop jumps from ~x1.5 to ~x7.3 geomean vs v0.23.1, landing within ~15% of the pipeline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sebpop/upstream@13699f62 guards transform_range for its no-offsets fast
path but not NormalizedString::replace, so encode_fast panics ("offset
data accessed on fast path") on any model with a Replace normalizer —
llama-2's Prepend+Replace killed bench shard 2. Probe under catch_unwind
and demote the series to None, the same fallback used for configs that
fail to load; charts render the model without a sebpop bar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bench the tokenizers crate from sebpop's performance branch (github.com/sebpop/tokenizers#upstream) alongside the released 0.23.1 in the pipeline benchmark: single-thread throughput per fixture, the multi-thread sweep, per-implementation memory footprint, minimal-binary size, and a report-only id-diff flag. The charts keep a single vs-release axis — sebpop renders as a second (green) bar next to the pipeline's.
The git dep uses sebpop's default features minus mimalloc: its override feature would swap the global allocator for the whole bench process and skew every series. Cargo.lock pins the resolved commit;
cargo update tokenizers@0.22.3-dev.0moves it to the branch tip.Older cached bench JSONs (base-branch baselines) without the sebpop keys still render unchanged.
PipelineTokenizer benchmark
7 / 8 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) andsebpop/upstream· ~10 kB inputs · refs via encode_fast · mimalloc (all series) · single thread + 1/2/4/8/max-thread sweepae0478f5d · 2026-07-20 16:38 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 32 coresvs base branch (
bb98d4e1f) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.bert-base-uncased — normalizer-heavy WordPiece · ×4.21 vs v0.23.1 · ×2.77 vs sebpop/upstream · ×1.01 vs base
Memory (RSS MB, load+encode): v0.23.1 8+13 (peak 21) · sebpop/upstream 8+11 (peak 19) · Pipeline 17+2 (peak 19)
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×3.76 vs v0.23.1 · ×2.05 vs sebpop/upstream · ×1.01 vs base
Memory (RSS MB, load+encode): v0.23.1 76+0 (peak 76) · sebpop/upstream 78+2 (peak 80) · Pipeline 114+0 (peak 114)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt2 — gpt2 ByteLevel regex · ×6.40 vs v0.23.1 · ×0.91 vs sebpop/upstream · ×1.00 vs base
Memory (RSS MB, load+encode): v0.23.1 20+13 (peak 34) · sebpop/upstream 27+1 (peak 28) · Pipeline 39+2 (peak 41)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×5.04 vs v0.23.1 · ×1.94 vs sebpop/upstream · ×1.00 vs base
Memory (RSS MB, load+encode): v0.23.1 4+15 (peak 20) · sebpop/upstream 11+13 (peak 23) · Pipeline 20+0 (peak 20)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×8.20 vs v0.23.1 · ×2.48 vs sebpop/upstream · ×1.00 vs base
Memory (RSS MB, load+encode): v0.23.1 2+26 (peak 28) · sebpop/upstream 11+25 (peak 36) · Pipeline 10+0 (peak 10)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).llama-2 — model-bounded BPE, no pre-tokenizer · ×2.78 vs v0.23.1 · ×0.98 vs base
Memory (RSS MB, load+encode): v0.23.1 36+11 (peak 46) · sebpop/upstream — · Pipeline 41+0 (peak 41)
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×5.58 vs v0.23.1 · ×2.41 vs sebpop/upstream · ×1.05 vs base
Memory (RSS MB, load+encode): v0.23.1 174+0 (peak 175) · sebpop/upstream 178+0 (peak 179) · Pipeline 200+0 (peak 200)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).Not yet supported:
t5-base