Skip to content

implement ultra-basic, erroneous decode - #2232

Draft
SBrandeis wants to merge 17 commits into
feat/train_encode_splitfrom
feat/pipeline-decode-impl
Draft

implement ultra-basic, erroneous decode#2232
SBrandeis wants to merge 17 commits into
feat/train_encode_splitfrom
feat/pipeline-decode-impl

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Jul 22, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

PipelineTokenizer benchmark

7 / 8 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · ~10 kB inputs · add_special_tokens on · single thread + 1/2/4/8/max-thread sweep

6810fa505 · 2026-07-27 10:26 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 48 cores

Per-model encode throughput vs latest release

vs base branch (8abd11215) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.

Per-model encode throughput vs base branch

Per-model memory footprint

Minimal encode binary size

Decode

Round-trip: v0.23.1 encode_fast produces the id streams (same fixtures, add_special_tokens=true); both implementations decode those SAME ids with skip_special_tokens=false. MB/s counts decoded text bytes.

Per-model decode throughput vs latest release

Per-model decode memory footprint

bert-base-uncased — normalizer-heavy WordPiece · ×4.87 vs v0.23.1 · ×1.04 vs base · decode ×14.56 bert-base-uncased speedup bert-base-uncased stage decomposition bert-base-uncased thread scaling bert-base-uncased decode speedup bert-base-uncased decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 12+0 (peak 12) · Pipeline 17+0 (peak 17)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 7.9 27.5 ×3.49 ×1.02 3% (1.2) 75% (27.1) 13% (4.8) 8% (3.0) 0% (0.0) match
arb_Arab lang 4.2 24.9 ×5.87 ×1.03 3% (1.2) 68% (27.5) 8% (3.3) 21% (8.3) 0% (0.0) match
ben_Beng lang 6.1 34.8 ×5.74 ×1.03 4% (1.2) 66% (19.1) 10% (2.9) 20% (5.6) 0% (0.0) match
cmn_Hani lang 3.9 18.9 ×4.84 ×1.05 2% (1.2) 70% (37.5) 11% (5.8) 17% (9.0) 0% (0.0) match
ell_Grek lang 3.8 23.7 ×6.18 ×1.03 3% (1.2) 68% (28.7) 8% (3.4) 21% (9.0) 0% (0.0) match
eng_Latn lang 4.6 18.4 ×4.02 ×1.08 5% (2.5) 69% (38.7) 8% (4.4) 19% (10.7) 0% (0.0) match
heb_Hebr lang 4.2 19.9 ×4.70 ×1.03 2% (1.2) 75% (37.5) 7% (3.7) 16% (8.0) 0% (0.0) match
hin_Deva lang 6.6 27.4 ×4.17 ×1.03 3% (1.2) 74% (27.0) 9% (3.2) 14% (5.2) 0% (0.0) match
jpn_Jpan lang 4.4 27.4 ×6.29 ×1.02 3% (1.2) 64% (23.2) 13% (4.6) 20% (7.4) 0% (0.0) match
kat_Geor lang 6.2 28.0 ×4.49 ×1.02 3% (1.2) 74% (26.5) 9% (3.1) 14% (4.8) 0% (0.0) match
kor_Hang lang 2.5 17.7 ×7.18 ×1.01 2% (1.2) 62% (35.1) 13% (7.3) 23% (12.8) 0% (0.0) match
rus_Cyrl lang 3.7 23.7 ×6.39 ×1.03 3% (1.2) 65% (27.4) 7% (3.2) 25% (10.5) 0% (0.1) match
tam_Taml lang 7.1 38.3 ×5.40 ×1.03 4% (1.2) 71% (18.6) 10% (2.6) 15% (3.8) 0% (0.0) match
tha_Thai lang 8.5 32.9 ×3.85 ×1.02 4% (1.2) 82% (24.9) 8% (2.5) 5% (1.6) 0% (0.0) match
added_normalized_dense modalities 6.8 19.7 ×2.91 ×1.08 3% (1.3) 78% (40.8) 13% (6.7) 6% (3.3) 0% (0.0) match
added_normalized_sparse modalities 5.3 18.4 ×3.45 ×1.07 3% (1.8) 72% (40.0) 13% (7.0) 12% (6.8) 0% (0.0) match
added_special_dense modalities 5.4 38.4 ×7.17 ×1.00 22% (5.4) 36% (8.9) 35% (8.6) 7% (1.9) 0% (0.0) match
added_special_sparse modalities 4.0 21.2 ×5.30 ×1.02 8% (3.7) 59% (28.1) 18% (8.6) 14% (6.7) 0% (0.2) match
agentic-traces modalities 3.9 18.2 ×4.69 ×1.07 4% (2.3) 68% (38.4) 9% (4.9) 19% (11.0) 0% (0.0) match
agentic_swe modalities 4.2 19.5 ×4.65 ×1.08 4% (1.9) 73% (38.6) 7% (3.7) 17% (9.0) 0% (0.0) match
code_mixed modalities 4.0 19.0 ×4.76 ×1.07 4% (2.1) 71% (38.5) 8% (4.2) 18% (9.6) 0% (0.0) match
math_latex modalities 4.1 18.2 ×4.42 ×1.07 4% (2.5) 68% (38.6) 8% (4.7) 19% (10.8) 0% (0.0) match
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×4.08 vs v0.23.1 · ×1.00 vs base · decode ×3.14 deepseek-v4 speedup deepseek-v4 stage decomposition deepseek-v4 thread scaling deepseek-v4 decode speedup deepseek-v4 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 67) · Pipeline 82+0 (peak 82)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.4 33.2 ×7.62 ×0.98 2% (0.7) 0% (0.0) 17% (4.8) 81% (23.5) 0% (0.0) match
arb_Arab lang 4.4 16.4 ×3.76 ×1.02 1% (0.6) 0% (0.0) 6% (3.4) 93% (55.2) 0% (0.0) match
ben_Beng lang 6.3 17.9 ×2.83 ×1.01 1% (0.6) 0% (0.0) 6% (3.1) 93% (51.4) 0% (0.0) match
cmn_Hani lang 3.7 16.5 ×4.42 ×1.00 1% (0.8) 0% (0.0) 5% (3.1) 94% (53.9) 0% (0.0) match
ell_Grek lang 4.7 17.8 ×3.77 ×0.99 1% (0.6) 0% (0.0) 6% (3.4) 93% (50.9) 0% (0.0) match
eng_Latn lang 3.1 12.9 ×4.11 ×1.00 3% (2.0) 0% (0.0) 7% (5.0) 90% (68.0) 1% (0.7) match
heb_Hebr lang 4.1 13.0 ×3.21 ×1.00 1% (0.6) 0% (0.0) 5% (3.5) 94% (71.5) 0% (0.2) match
hin_Deva lang 5.9 20.9 ×3.52 ×0.99 1% (0.6) 0% (0.0) 7% (3.3) 91% (42.9) 0% (0.1) match
jpn_Jpan lang 4.3 17.7 ×4.08 ×1.01 1% (0.7) 0% (0.0) 5% (3.0) 93% (50.6) 0% (0.0) match
kat_Geor lang 6.0 17.9 ×2.95 ×1.03 1% (0.6) 0% (0.0) 5% (2.9) 94% (51.7) 0% (0.0) match
kor_Hang lang 3.8 19.6 ×5.20 ×0.99 1% (0.6) 0% (0.0) 7% (3.7) 92% (45.6) 0% (0.0) match
rus_Cyrl lang 4.4 14.2 ×3.21 ×0.98 1% (0.6) 0% (0.0) 5% (3.3) 94% (64.9) 1% (0.6) match
tam_Taml lang 6.3 17.6 ×2.81 ×1.01 1% (0.6) 0% (0.0) 5% (2.7) 94% (52.6) 0% (0.1) match
tha_Thai lang 7.1 14.3 ×2.02 ×1.00 1% (0.6) 0% (0.0) 3% (2.2) 96% (65.7) 0% (0.0) match
added_normalized_dense modalities 6.0 23.3 ×3.85 ×1.00 2% (0.7) 0% (0.0) 6% (2.7) 92% (38.8) 0% (0.0) match
added_normalized_sparse modalities 5.4 19.4 ×3.62 ×0.99 3% (1.3) 0% (0.0) 7% (3.8) 90% (45.5) 0% (0.0) match
added_special_dense modalities 3.6 36.3 ×10.09 ×0.99 25% (6.5) 2% (0.6) 25% (6.6) 48% (12.8) 0% (0.0) match
added_special_sparse modalities 4.0 20.5 ×5.13 ×0.99 8% (3.9) 0% (0.2) 14% (6.9) 76% (36.7) 1% (0.5) match
agentic-traces modalities 3.0 14.2 ×4.81 ×1.00 3% (1.8) 0% (0.0) 8% (5.4) 90% (61.2) 0% (0.0) match
agentic_swe modalities 3.0 14.9 ×4.96 ×1.00 2% (1.3) 0% (0.0) 6% (3.8) 92% (59.8) 0% (0.0) match
code_mixed modalities 3.5 15.4 ×4.42 ×0.99 2% (1.6) 0% (0.0) 7% (4.6) 93% (58.4) 0% (0.0) match
math_latex modalities 2.8 13.8 ×4.88 ×0.99 3% (1.9) 0% (0.0) 8% (5.3) 90% (62.3) 0% (0.1) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.79 3.43 4.79 5.43 41.4 19.9 9.1 — 8.6× / 7.6× 4.2× / 3.7× 1.9× / 1.7× —
arb_Arab 1.15 2.97 3.37 5.19 47.7 22.6 10.2 — 14.2× / 9.2× 6.7× / 4.4× 3.0× / 2.0× —
ben_Beng 1.62 2.89 3.15 4.42 35.5 16.0 7.5 — 11.3× / 8.0× 5.1× / 3.6× 2.4× / 1.7× —
cmn_Hani 1.12 2.24 3.14 4.26 58.7 33.7 14.2 — 18.7× / 13.8× 10.7× / 7.9× 4.5× / 3.3× —
ell_Grek 0.59 2.99 3.39 5.79 48.2 20.8 9.6 — 14.2× / 8.3× 6.1× / 3.6× 2.8× / 1.7× —
eng_Latn 0.10 1.46 5.02 6.39 63.6 38.8 16.1 — 12.7× / 10.0× 7.7× / 6.1× 3.2× / 2.5× —
heb_Hebr 1.15 3.00 3.50 5.36 49.9 23.8 10.6 — 14.2× / 9.3× 6.8× / 4.4× 3.0× / 2.0× —
hin_Deva 1.51 3.09 3.31 4.90 37.3 18.5 8.5 — 11.3× / 7.6× 5.6× / 3.8× 2.6× / 1.7× —
jpn_Jpan 1.70 3.36 2.96 4.62 51.3 27.0 11.7 — 17.3× / 11.1× 9.1× / 5.8× 4.0× / 2.5× —
kat_Geor 1.54 2.57 2.93 3.97 31.5 15.0 7.2 — 10.7× / 7.9× 5.1× / 3.8× 2.4× / 1.8× —
kor_Hang 1.23 2.70 3.68 5.15 48.6 26.7 11.9 — 13.2× / 9.4× 7.3× / 5.2× 3.2× / 2.3× —
rus_Cyrl 1.16 2.88 3.26 4.99 45.7 20.8 9.5 — 14.0× / 9.2× 6.4× / 4.2× 2.9× / 1.9× —
tam_Taml 0.93 2.98 2.69 4.74 31.3 13.5 6.6 — 11.7× / 6.6× 5.0× / 2.8× 2.4× / 1.4× —
tha_Thai 1.51 2.57 2.18 3.24 27.0 10.1 5.2 — 12.4× / 8.3× 4.6× / 3.1× 2.4× / 1.6× —
added_normalized_dense 0.06 1.47 2.74 4.16 41.4 20.0 9.0 — 15.1× / 10.0× 7.3× / 4.8× 3.3× / 2.2× —
added_normalized_sparse 0.06 1.47 3.78 5.20 47.7 25.8 11.3 — 12.6× / 9.2× 6.8× / 5.0× 3.0× / 2.2× —
added_special_dense 0.06 1.47 6.65 8.06 167.2 97.1 38.0 — 25.2× / 20.7× 14.6× / 12.0× 5.7× / 4.7× —
added_special_sparse 0.06 1.48 6.95 8.37 99.1 56.7 23.8 — 14.3× / 11.9× 8.2× / 6.8× 3.4× / 2.8× —
agentic-traces 0.73 1.48 5.37 6.12 78.5 53.0 19.9 — 14.6× / 12.8× 9.9× / 8.6× 3.7× / 3.2× —
agentic_swe 0.68 1.44 3.82 4.58 92.9 65.3 23.2 — 24.3× / 20.3× 17.1× / 14.2× 6.1× / 5.1× —
code_mixed 0.07 1.44 4.56 5.93 72.0 53.4 18.3 — 15.8× / 12.2× 11.7× / 9.0× 4.0× / 3.1× —
math_latex 0.71 1.48 5.28 6.05 76.1 47.4 19.2 — 14.4× / 12.6× 9.0× / 7.8× 3.6× / 3.2× —
gpt2 — gpt2 ByteLevel regex · ×7.55 vs v0.23.1 · ×0.99 vs base · decode ×3.40 gpt2 speedup gpt2 stage decomposition gpt2 thread scaling gpt2 decode speedup gpt2 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 28+0 (peak 28)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.4 50.0 ×11.28 ×0.98 3% (0.7) 0% (0.0) 19% (3.6) 78% (14.8) 0% (0.0) match
arb_Arab lang 4.2 25.6 ×6.04 ×0.99 2% (0.6) 0% (0.0) 6% (2.3) 93% (35.5) 0% (0.0) match
ben_Beng lang 3.2 43.9 ×13.54 ×0.94 3% (0.6) 0% (0.0) 13% (3.0) 84% (19.0) 0% (0.0) match
cmn_Hani lang 4.0 28.3 ×7.09 ×0.96 2% (0.6) 0% (0.0) 7% (2.3) 92% (31.4) 0% (0.0) match
ell_Grek lang 4.6 28.8 ×6.28 ×1.02 2% (0.6) 0% (0.0) 6% (2.1) 92% (31.4) 0% (0.0) match
eng_Latn lang 3.8 13.9 ×3.68 ×0.99 3% (2.0) 0% (0.0) 4% (2.9) 93% (65.8) 0% (0.0) match
heb_Hebr lang 4.1 28.7 ×6.97 ×1.01 2% (0.6) 0% (0.0) 7% (2.3) 92% (31.3) 0% (0.0) match
hin_Deva lang 3.3 40.6 ×12.42 ×1.00 2% (0.6) 0% (0.0) 13% (3.1) 85% (20.6) 0% (0.0) match
jpn_Jpan lang 4.6 21.0 ×4.57 ×0.95 1% (0.6) 0% (0.0) 5% (2.1) 94% (43.9) 0% (0.0) match
kat_Geor lang 4.9 70.0 ×14.41 ×1.02 4% (0.6) 0% (0.0) 14% (2.1) 82% (11.8) 0% (0.0) match
kor_Hang lang 3.5 41.8 ×12.01 ×0.96 3% (0.6) 0% (0.0) 10% (2.4) 86% (20.1) 1% (0.2) match
rus_Cyrl lang 4.2 27.4 ×6.51 ×0.95 2% (0.6) 0% (0.0) 6% (2.1) 93% (33.2) 0% (0.0) match
tam_Taml lang 2.7 72.6 ×27.34 ×1.06 4% (0.6) 0% (0.0) 20% (2.6) 76% (10.2) 0% (0.0) match
tha_Thai lang 3.6 38.4 ×10.55 ×1.03 2% (0.6) 0% (0.0) 10% (2.5) 88% (22.0) 0% (0.0) match
added_normalized_dense modalities 6.0 25.3 ×4.20 ×1.00 2% (0.7) 0% (0.0) 3% (1.1) 95% (36.7) 0% (0.0) match
added_normalized_sparse modalities 5.2 21.4 ×4.14 ×1.00 3% (1.3) 0% (0.0) 4% (1.9) 93% (42.8) 0% (0.0) match
added_special_dense modalities 4.1 47.4 ×11.47 ×0.99 25% (5.0) 0% (0.0) 20% (4.0) 55% (11.0) 0% (0.1) match
added_special_sparse modalities 4.3 23.6 ×5.47 ×0.98 8% (3.1) 0% (0.1) 10% (4.1) 82% (33.6) 1% (0.2) match
agentic-traces modalities 3.2 16.2 ×5.00 ×1.00 3% (1.8) 0% (0.0) 6% (3.5) 92% (55.6) 0% (0.0) match
agentic_swe modalities 3.4 24.6 ×7.17 ×0.98 3% (1.3) 0% (0.0) 6% (2.4) 91% (36.0) 0% (0.0) match
code_mixed modalities 3.5 20.5 ×5.82 ×0.98 3% (1.6) 0% (0.0) 6% (2.9) 90% (43.4) 1% (0.2) match
math_latex modalities 3.2 15.2 ×4.69 ×0.99 3% (1.9) 0% (0.0) 5% (3.2) 92% (60.0) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.78 3.43 3.64 4.28 26.0 21.3 5.7 4.8 7.1× / 6.1× 5.8× / 5.0× 1.6× / 1.3× 1.3× / 1.1×
arb_Arab 1.15 2.96 2.25 4.06 30.2 25.9 6.7 5.1 13.4× / 7.4× 11.5× / 6.4× 3.0× / 1.7× 2.3× / 1.3×
ben_Beng 1.85 2.93 2.98 4.06 62.0 57.1 13.6 4.0 20.8× / 15.3× 19.1× / 14.1× 4.5× / 3.3× 1.3× / 1.0×
cmn_Hani 1.12 2.23 2.32 3.43 24.9 21.4 5.9 2.4 10.8× / 7.3× 9.2× / 6.2× 2.5× / 1.7× 1.0× / 0.7×
ell_Grek 1.15 2.95 2.14 3.95 26.8 21.9 5.9 4.8 12.5× / 6.8× 10.2× / 5.6× 2.7× / 1.5× 2.2× / 1.2×
eng_Latn 0.10 1.46 2.88 4.25 41.0 42.5 11.7 3.8 14.2× / 9.7× 14.8× / 10.0× 4.1× / 2.7× 1.3× / 0.9×
heb_Hebr 1.14 3.03 2.25 4.14 29.8 26.2 6.8 3.0 13.2× / 7.2× 11.6× / 6.3× 3.0× / 1.6× 1.3× / 0.7×
hin_Deva 1.51 3.14 3.06 4.69 57.8 54.5 13.3 4.2 18.9× / 12.3× 17.8× / 11.6× 4.4× / 2.8× 1.4× / 0.9×
jpn_Jpan 1.70 3.35 2.10 3.76 22.3 17.8 4.9 3.9 10.6× / 5.9× 8.4× / 4.7× 2.3× / 1.3× 1.8× / 1.0×
kat_Geor 1.54 2.51 2.06 3.03 16.5 14.3 4.1 1.9 8.0× / 5.5× 7.0× / 4.7× 2.0× / 1.4× 0.9× / 0.6×
kor_Hang 1.23 2.73 2.42 3.92 30.3 28.7 7.4 3.7 12.5× / 7.7× 11.9× / 7.3× 3.1× / 1.9× 1.5× / 0.9×
rus_Cyrl 1.16 2.93 2.08 3.84 25.6 21.0 5.7 2.5 12.3× / 6.7× 10.1× / 5.5× 2.8× / 1.5× 1.2× / 0.7×
tam_Taml 0.93 2.92 2.64 4.64 65.3 58.6 14.1 3.8 24.7× / 14.1× 22.2× / 12.6× 5.3× / 3.0× 1.4× / 0.8×
tha_Thai 1.52 2.57 2.48 3.54 38.3 31.7 8.7 3.2 15.4× / 10.8× 12.8× / 9.0× 3.5× / 2.5× 1.3× / 0.9×
added_normalized_dense 0.06 1.48 1.10 2.52 22.2 22.8 6.3 1.9 20.1× / 8.8× 20.7× / 9.0× 5.7× / 2.5× 1.8× / 0.8×
added_normalized_sparse 0.06 1.48 1.91 3.33 29.3 31.4 8.3 2.7 15.3× / 8.8× 16.4× / 9.4× 4.3× / 2.5× 1.4× / 0.8×
added_special_dense 0.06 1.47 4.00 5.42 89.0 98.4 19.7 2.8 22.2× / 16.4× 24.6× / 18.2× 4.9× / 3.6× 0.7× / 0.5×
added_special_sparse 0.06 1.77 4.15 5.86 57.7 61.2 14.1 3.4 13.9× / 9.9× 14.8× / 10.5× 3.4× / 2.4× 0.8× / 0.6×
agentic-traces 0.74 1.48 3.46 4.20 54.9 61.3 15.1 4.6 15.9× / 13.1× 17.7× / 14.6× 4.4× / 3.6× 1.3× / 1.1×
agentic_swe 0.66 1.45 2.42 3.21 56.3 69.5 14.4 3.5 23.2× / 17.5× 28.7× / 21.6× 5.9× / 4.5× 1.4× / 1.1×
code_mixed 0.10 1.44 2.91 4.25 55.4 68.0 14.8 4.1 19.0× / 13.0× 23.4× / 16.0× 5.1× / 3.5× 1.4× / 1.0×
math_latex 0.71 1.47 3.22 3.99 48.9 51.1 13.8 4.2 15.2× / 12.3× 15.8× / 12.8× 4.3× / 3.4× 1.3× / 1.0×
gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×7.59 vs v0.23.1 · ×1.00 vs base · decode ×3.39 gpt-oss speedup gpt-oss stage decomposition gpt-oss thread scaling gpt-oss decode speedup gpt-oss decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 5+12 (peak 17) · Pipeline 7+0 (peak 7)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 3.1 62.1 ×20.19 ×1.06 4% (0.7) 0% (0.0) 32% (5.0) 64% (9.8) 0% (0.0) match
arb_Arab lang 3.7 24.7 ×6.66 ×0.98 2% (0.6) 0% (0.0) 10% (3.8) 89% (35.3) 0% (0.0) match
ben_Beng lang 5.1 27.2 ×5.37 ×0.99 2% (0.6) 0% (0.0) 11% (3.9) 87% (31.4) 0% (0.0) match
cmn_Hani lang 4.1 38.5 ×9.44 ×1.00 2% (0.6) 0% (0.0) 13% (3.3) 84% (21.3) 0% (0.0) match
ell_Grek lang 4.0 31.5 ×7.92 ×1.01 2% (0.6) 0% (0.0) 11% (3.3) 87% (27.0) 0% (0.0) match
eng_Latn lang 3.1 21.3 ×6.86 ×0.99 4% (2.0) 0% (0.0) 10% (4.6) 86% (39.9) 0% (0.0) match
heb_Hebr lang 3.7 26.7 ×7.18 ×0.95 2% (0.6) 0% (0.0) 10% (3.9) 89% (32.8) 0% (0.0) match
hin_Deva lang 5.0 23.7 ×4.78 ×1.01 1% (0.6) 0% (0.0) 10% (4.0) 89% (36.7) 0% (0.0) match
jpn_Jpan lang 4.7 36.8 ×7.82 ×0.98 2% (0.6) 0% (0.0) 12% (3.1) 86% (22.6) 0% (0.0) match
kat_Geor lang 5.5 26.2 ×4.72 ×1.05 2% (0.6) 0% (0.0) 8% (2.9) 91% (34.1) 0% (0.0) match
kor_Hang lang 3.5 47.9 ×13.77 ×1.02 3% (0.6) 0% (0.0) 19% (3.9) 78% (15.8) 0% (0.0) match
rus_Cyrl lang 4.4 22.4 ×5.03 ×0.99 1% (0.6) 0% (0.0) 7% (3.2) 94% (41.0) 0% (0.0) match
tam_Taml lang 5.5 28.8 ×5.20 ×1.00 2% (0.6) 0% (0.0) 10% (3.4) 88% (29.9) 0% (0.0) match
tha_Thai lang 6.3 26.7 ×4.24 ×0.99 2% (0.6) 0% (0.0) 9% (3.4) 89% (32.1) 0% (0.0) match
added_normalized_dense modalities 4.7 45.2 ×9.67 ×0.99 3% (0.7) 0% (0.0) 11% (2.3) 86% (18.9) 0% (0.0) match
added_normalized_sparse modalities 4.2 36.1 ×8.52 ×0.98 5% (1.3) 0% (0.0) 12% (3.2) 84% (22.9) 0% (0.0) match
added_special_dense modalities 3.7 55.5 ×14.86 ×1.02 30% (5.0) 0% (0.1) 28% (4.8) 42% (7.1) 0% (0.0) match
added_special_sparse modalities 3.8 34.0 ×9.06 ×0.98 11% (3.2) 0% (0.0) 20% (5.7) 68% (19.3) 0% (0.0) match
agentic-traces modalities 3.1 22.5 ×7.21 ×0.98 4% (1.8) 0% (0.0) 11% (5.0) 85% (36.7) 0% (0.0) match
agentic_swe modalities 3.4 21.7 ×6.32 ×0.99 3% (1.3) 0% (0.0) 8% (3.8) 89% (40.1) 0% (0.0) match
code_mixed modalities 3.4 29.6 ×8.81 ×1.00 5% (1.6) 0% (0.0) 13% (4.4) 85% (28.3) 0% (0.0) match
math_latex modalities 3.1 22.4 ×7.29 ×0.99 4% (1.9) 0% (0.0) 11% (4.8) 85% (36.7) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.78 3.43 4.95 5.60 36.3 14.2 6.9 4.7 7.3× / 6.5× 2.9× / 2.5× 1.4× / 1.2× 1.0× / 0.8×
arb_Arab 1.15 3.00 3.77 5.63 41.8 16.2 7.5 5.1 11.1× / 7.4× 4.3× / 2.9× 2.0× / 1.3× 1.4× / 0.9×
ben_Beng 1.84 2.98 3.90 5.05 28.7 11.0 5.5 2.7 7.4× / 5.7× 2.8× / 2.2× 1.4× / 1.1× 0.7× / 0.5×
cmn_Hani 1.11 2.25 3.30 4.44 25.6 10.8 5.5 2.5 7.7× / 5.8× 3.3× / 2.4× 1.7× / 1.2× 0.8× / 0.6×
ell_Grek 1.15 2.95 3.28 5.08 37.1 15.0 6.9 5.1 11.3× / 7.3× 4.6× / 3.0× 2.1× / 1.4× 1.6× / 1.0×
eng_Latn 0.10 1.45 4.61 5.96 58.3 29.2 13.4 4.1 12.7× / 9.8× 6.3× / 4.9× 2.9× / 2.2× 0.9× / 0.7×
heb_Hebr 1.14 3.02 3.87 5.75 42.0 17.1 8.3 2.8 10.8× / 7.3× 4.4× / 3.0× 2.1× / 1.4× 0.7× / 0.5×
hin_Deva 1.54 3.06 4.02 5.55 32.1 13.3 6.3 3.1 8.0× / 5.8× 3.3× / 2.4× 1.6× / 1.1× 0.8× / 0.6×
jpn_Jpan 1.70 3.36 3.06 4.72 23.4 9.2 4.8 3.8 7.7× / 5.0× 3.0× / 1.9× 1.6× / 1.0× 1.2× / 0.8×
kat_Geor 1.54 2.55 2.92 3.93 22.9 10.3 4.6 2.1 7.8× / 5.8× 3.5× / 2.6× 1.6× / 1.2× 0.7× / 0.5×
kor_Hang 1.23 2.74 3.87 5.38 42.0 18.6 8.7 3.5 10.9× / 7.8× 4.8× / 3.5× 2.3× / 1.6× 0.9× / 0.7×
rus_Cyrl 1.16 2.90 3.18 4.92 35.5 14.8 6.7 4.8 11.1× / 7.2× 4.7× / 3.0× 2.1× / 1.4× 1.5× / 1.0×
tam_Taml 0.93 2.94 3.44 5.45 22.7 8.7 4.2 2.9 6.6× / 4.2× 2.5× / 1.6× 1.2× / 0.8× 0.8× / 0.5×
tha_Thai 1.53 2.58 3.42 4.48 14.8 5.7 2.8 2.2 4.3× / 3.3× 1.7× / 1.3× 0.8× / 0.6× 0.7× / 0.5×
added_normalized_dense 0.06 1.47 2.33 3.74 42.3 17.1 11.3 2.2 18.2× / 11.3× 7.3× / 4.6× 4.9× / 3.0× 0.9× / 0.6×
added_normalized_sparse 0.06 1.47 3.15 4.57 47.3 20.6 11.5 2.9 15.0× / 10.3× 6.5× / 4.5× 3.6× / 2.5× 0.9× / 0.6×
added_special_dense 0.06 1.47 4.82 6.24 116.9 67.6 23.8 3.1 24.3× / 18.7× 14.0× / 10.8× 4.9× / 3.8× 0.7× / 0.5×
added_special_sparse 0.06 1.48 5.74 7.16 77.7 41.7 16.8 3.6 13.6× / 10.9× 7.3× / 5.8× 2.9× / 2.3× 0.6× / 0.5×
agentic-traces 0.72 1.48 4.99 5.76 69.3 40.7 16.5 4.8 13.9× / 12.0× 8.2× / 7.1× 3.3× / 2.9× 1.0× / 0.8×
agentic_swe 0.66 1.46 3.75 4.56 70.4 48.7 17.1 3.6 18.8× / 15.4× 13.0× / 10.7× 4.6× / 3.7× 1.0× / 0.8×
code_mixed 0.07 1.44 4.36 5.73 69.6 47.3 16.8 4.3 16.0× / 12.2× 10.9× / 8.3× 3.8× / 2.9× 1.0× / 0.7×
math_latex 0.74 1.49 4.84 5.59 67.3 35.2 15.4 4.4 13.9× / 12.0× 7.3× / 6.3× 3.2× / 2.8× 0.9× / 0.8×
glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×12.88 vs v0.23.1 · ×1.00 vs base · decode ×3.54 glm-5.2 speedup glm-5.2 stage decomposition glm-5.2 thread scaling glm-5.2 decode speedup glm-5.2 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 4+12 (peak 17) · Pipeline 6+0 (peak 6)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.6 57.7 ×12.52 ×0.94 7% (1.2) 0% (0.0) 22% (3.7) 70% (11.8) 0% (0.1) match
arb_Arab lang 4.1 72.6 ×17.61 ×1.27 9% (1.2) 0% (0.0) 17% (2.3) 74% (10.0) 0% (0.0) match
ben_Beng lang 3.4 71.0 ×21.04 ×1.02 8% (1.2) 0% (0.0) 23% (3.1) 68% (9.4) 0% (0.0) match
cmn_Hani lang 4.7 69.0 ×14.64 ×1.03 8% (1.2) 0% (0.0) 17% (2.4) 74% (10.4) 0% (0.0) match
ell_Grek lang 4.0 71.3 ×17.75 ×0.99 9% (1.2) 0% (0.0) 16% (2.2) 75% (10.2) 0% (0.0) match
eng_Latn lang 3.5 22.7 ×6.48 ×0.99 6% (2.5) 0% (0.0) 7% (3.0) 87% (37.9) 0% (0.0) match
heb_Hebr lang 3.7 69.7 ×18.78 ×0.94 8% (1.2) 0% (0.0) 17% (2.3) 75% (10.5) 0% (0.0) match
hin_Deva lang 3.1 64.7 ×20.98 ×0.95 8% (1.2) 0% (0.0) 21% (3.2) 71% (10.6) 0% (0.0) match
jpn_Jpan lang 4.7 74.2 ×15.85 ×1.05 9% (1.2) 0% (0.0) 17% (2.2) 74% (9.6) 0% (0.0) match
kat_Geor lang 4.4 78.7 ×17.71 ×0.93 9% (1.2) 0% (0.0) 17% (2.1) 73% (8.9) 0% (0.0) match
kor_Hang lang 3.5 70.5 ×20.43 ×1.04 9% (1.2) 0% (0.0) 18% (2.5) 73% (10.0) 0% (0.0) match
rus_Cyrl lang 4.0 40.9 ×10.29 ×1.02 5% (1.2) 0% (0.0) 9% (2.2) 86% (20.8) 1% (0.1) match
tam_Taml lang 3.2 75.0 ×23.75 ×1.04 9% (1.2) 0% (0.0) 21% (2.7) 70% (9.1) 0% (0.0) match
tha_Thai lang 3.9 72.0 ×18.43 ×1.00 9% (1.2) 0% (0.0) 19% (2.6) 72% (9.8) 0% (0.0) match
added_normalized_dense modalities 4.6 45.8 ×9.92 ×0.99 6% (1.3) 0% (0.0) 5% (1.1) 89% (18.4) 0% (0.0) match
added_normalized_sparse modalities 4.1 39.0 ×9.43 ×0.97 8% (1.8) 0% (0.0) 9% (2.1) 85% (20.5) 0% (0.0) match
added_special_dense modalities 3.4 40.5 ×11.99 ×1.00 53% (12.7) 1% (0.2) 20% (4.8) 25% (5.9) 1% (0.3) match
added_special_sparse modalities 3.5 33.8 ×9.78 ×0.99 25% (7.1) 0% (0.0) 16% (4.5) 59% (16.7) 0% (0.0) match
agentic-traces modalities 3.2 23.8 ×7.39 ×0.99 6% (2.5) 0% (0.0) 9% (3.7) 85% (35.2) 0% (0.0) match
agentic_swe modalities 3.6 22.2 ×6.16 ×0.99 4% (1.9) 0% (0.0) 7% (2.9) 89% (39.6) 0% (0.0) match
code_mixed modalities 3.6 31.0 ×8.71 ×0.99 7% (2.2) 0% (0.0) 10% (3.3) 83% (26.3) 0% (0.0) match
math_latex modalities 3.2 24.1 ×7.54 ×0.99 6% (2.5) 0% (0.0) 8% (3.4) 86% (34.9) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.78 3.41 3.70 4.32 26.2 15.9 5.8 4.8 7.1× / 6.1× 4.3× / 3.7× 1.6× / 1.4× 1.3× / 1.1×
arb_Arab 1.15 2.96 2.32 4.13 29.6 19.5 6.7 5.2 12.8× / 7.2× 8.4× / 4.7× 2.9× / 1.6× 2.2× / 1.2×
ben_Beng 1.61 2.91 3.14 4.44 43.5 27.7 10.1 3.7 13.8× / 9.8× 8.8× / 6.2× 3.2× / 2.3× 1.2× / 0.8×
cmn_Hani 1.11 2.22 2.39 3.50 18.7 11.6 4.7 2.4 7.8× / 5.3× 4.9× / 3.3× 2.0× / 1.3× 1.0× / 0.7×
ell_Grek 0.59 2.97 2.21 4.59 27.1 17.0 6.1 4.7 12.3× / 5.9× 7.7× / 3.7× 2.7× / 1.3× 2.1× / 1.0×
eng_Latn 0.11 1.46 3.01 4.37 41.5 32.7 12.0 3.7 13.8× / 9.5× 10.8× / 7.5× 4.0× / 2.8× 1.2× / 0.8×
heb_Hebr 1.14 3.06 2.32 4.24 29.6 19.8 6.9 3.0 12.7× / 7.0× 8.5× / 4.7× 3.0× / 1.6× 1.3× / 0.7×
hin_Deva 1.52 3.12 3.22 4.82 46.1 30.2 11.0 4.0 14.3× / 9.6× 9.4× / 6.3× 3.4× / 2.3× 1.3× / 0.8×
jpn_Jpan 1.69 3.33 2.20 3.84 17.5 9.9 4.0 3.7 8.0× / 4.6× 4.5× / 2.6× 1.8× / 1.1× 1.7× / 1.0×
kat_Geor 1.55 2.49 2.10 3.04 16.8 11.3 4.2 2.0 8.0× / 5.5× 5.4× / 3.7× 2.0× / 1.4× 1.0× / 0.7×
kor_Hang 1.23 2.72 2.52 4.00 29.4 21.0 7.3 3.7 11.7× / 7.3× 8.3× / 5.2× 2.9× / 1.8× 1.5× / 0.9×
rus_Cyrl 1.16 2.90 2.15 3.89 25.6 16.6 5.9 2.5 11.9× / 6.6× 7.7× / 4.3× 2.7× / 1.5× 1.2× / 0.6×
tam_Taml 0.93 2.94 2.72 4.74 42.4 26.6 9.6 3.4 15.6× / 8.9× 9.8× / 5.6× 3.5× / 2.0× 1.3× / 0.7×
tha_Thai 1.53 2.59 2.62 3.67 27.2 16.1 6.6 3.0 10.4× / 7.4× 6.2× / 4.4× 2.5× / 1.8× 1.1× / 0.8×
added_normalized_dense 0.06 1.47 1.12 2.53 22.5 17.1 6.5 1.8 20.2× / 8.9× 15.3× / 6.8× 5.8× / 2.6× 1.6× / 0.7×
added_normalized_sparse 0.06 1.77 2.07 3.77 30.0 22.8 8.9 2.5 14.5× / 7.9× 11.1× / 6.0× 4.3× / 2.4× 1.2× / 0.7×
added_special_dense 0.06 1.47 4.77 6.19 90.5 74.5 20.9 3.1 19.0× / 14.6× 15.6× / 12.0× 4.4× / 3.4× 0.7× / 0.5×
added_special_sparse 0.06 1.77 4.54 6.25 58.3 46.8 15.0 3.6 12.8× / 9.3× 10.3× / 7.5× 3.3× / 2.4× 0.8× / 0.6×
agentic-traces 0.71 1.47 3.72 4.48 49.9 43.5 14.6 4.5 13.4× / 11.1× 11.7× / 9.7× 3.9× / 3.3× 1.2× / 1.0×
agentic_swe 0.66 1.45 2.88 3.67 52.8 51.7 15.5 3.4 18.3× / 14.4× 17.9× / 14.1× 5.4× / 4.2× 1.2× / 0.9×
code_mixed 0.07 1.44 3.28 4.65 50.4 48.9 14.9 4.0 15.3× / 10.8× 14.9× / 10.5× 4.5× / 3.2× 1.2× / 0.9×
math_latex 0.72 1.48 3.44 4.20 48.2 38.9 14.1 4.1 14.0× / 11.5× 11.3× / 9.3× 4.1× / 3.3× 1.2× / 1.0×
llama-2 — model-bounded BPE, no pre-tokenizer · ×3.16 vs v0.23.1 · ×1.02 vs base · decode ×2.84 llama-2 speedup llama-2 stage decomposition llama-2 thread scaling llama-2 decode speedup llama-2 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 18+0 (peak 23) · Pipeline 24+0 (peak 24)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 5.6 41.7 ×7.47 ×0.99 0% (0.0) 12% (2.7) 0% (0.0) 88% (18.9) 0% (0.0) match
arb_Arab lang 12.3 49.4 ×4.02 ×1.02 0% (0.0) 16% (3.1) 0% (0.0) 84% (15.9) 0% (0.0) match
ben_Beng lang 13.5 81.2 ×6.02 ×1.02 0% (0.0) 19% (2.1) 0% (0.0) 81% (9.2) 0% (0.0) match
cmn_Hani lang 11.3 65.9 ×5.83 ×1.05 0% (0.1) 3% (0.4) 0% (0.0) 97% (13.5) 0% (0.0) match
ell_Grek lang 12.6 57.8 ×4.58 ×1.02 0% (0.0) 19% (3.1) 0% (0.0) 81% (13.4) 0% (0.0) match
eng_Latn lang 4.5 6.5 ×1.44 ×1.01 0% (0.1) 4% (6.7) 0% (0.0) 95% (143.4) 0% (0.3) match
heb_Hebr lang 12.3 60.5 ×4.91 ×1.02 0% (0.0) 21% (3.2) 0% (0.0) 79% (12.1) 0% (0.0) match
hin_Deva lang 14.4 78.7 ×5.45 ×1.05 0% (0.0) 24% (2.8) 0% (0.0) 77% (9.0) 0% (0.0) match
jpn_Jpan lang 15.7 82.3 ×5.24 ×1.01 0% (0.0) 3% (0.3) 0% (0.0) 97% (10.3) 0% (0.0) match
kat_Geor lang 17.3 91.6 ×5.28 ×1.04 0% (0.0) 18% (1.8) 0% (0.0) 81% (8.3) 0% (0.0) match
kor_Hang lang 8.6 50.2 ×5.86 ×1.00 0% (0.1) 17% (3.2) 0% (0.0) 82% (15.2) 0% (0.0) match
rus_Cyrl lang 8.9 16.2 ×1.81 ×1.01 0% (0.0) 5% (2.8) 0% (0.0) 95% (57.6) 0% (0.2) match
tam_Taml lang 15.3 89.1 ×5.81 ×1.05 0% (0.0) 17% (1.7) 0% (0.0) 83% (8.5) 0% (0.0) match
tha_Thai lang 19.4 87.4 ×4.51 ×1.05 0% (0.0) 9% (0.9) 0% (0.0) 91% (9.5) 0% (0.0) match
added_normalized_dense modalities 6.1 8.6 ×1.40 ×0.99 0% (0.1) 3% (3.7) 0% (0.0) 98% (109.9) 0% (0.0) match
added_normalized_sparse modalities 5.2 7.4 ×1.42 ×1.01 0% (0.0) 5% (6.0) 0% (0.0) 96% (122.3) 0% (0.0) match
added_special_dense modalities 5.3 20.1 ×3.80 ×1.01 11% (5.2) 32% (15.6) 2% (1.2) 54% (26.2) 0% (0.0) match
added_special_sparse modalities 7.7 10.3 ×1.34 ×1.01 2% (2.2) 14% (13.7) 0% (0.5) 83% (79.8) 0% (0.0) match
agentic-traces modalities 4.7 7.2 ×1.52 ×1.02 0% (0.1) 4% (6.1) 0% (0.0) 95% (129.4) 0% (0.4) match
agentic_swe modalities 4.2 7.6 ×1.80 ×1.01 0% (0.0) 7% (9.7) 0% (0.0) 93% (120.6) 0% (0.0) match
code_mixed modalities 4.4 7.2 ×1.64 ×1.01 0% (0.0) 6% (7.9) 0% (0.0) 95% (127.2) 0% (0.0) match
math_latex modalities 4.6 6.8 ×1.46 ×0.99 0% (0.1) 4% (6.2) 0% (0.0) 95% (136.5) 0% (0.5) match
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×7.00 vs v0.23.1 · ×0.98 vs base · decode ×3.30 llama-3 speedup llama-3 stage decomposition llama-3 thread scaling llama-3 decode speedup llama-3 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 143+0 (peak 199) · Pipeline 145+0 (peak 200)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.4 50.9 ×11.67 ×0.96 4% (0.7) 0% (0.0) 22% (3.7) 75% (12.7) 0% (0.0) match
arb_Arab lang 4.9 17.6 ×3.63 ×0.95 1% (0.6) 0% (0.0) 4% (2.3) 95% (51.4) 0% (0.0) match
ben_Beng lang 4.1 31.1 ×7.52 ×0.97 2% (0.6) 0% (0.0) 10% (3.1) 88% (27.4) 0% (0.0) match
cmn_Hani lang 5.2 16.4 ×3.16 ×0.95 1% (0.6) 0% (0.0) 4% (2.4) 96% (52.9) 0% (0.0) match
ell_Grek lang 5.3 19.3 ×3.68 ×0.98 1% (0.6) 0% (0.0) 4% (2.2) 96% (47.1) 0% (0.0) match
eng_Latn lang 4.0 45.0 ×11.30 ×0.89 11% (2.0) 0% (0.0) 16% (3.0) 73% (13.3) 0% (0.0) match
heb_Hebr lang 4.4 25.9 ×5.92 ×1.01 2% (0.6) 0% (0.0) 6% (2.3) 92% (33.6) 0% (0.0) match
hin_Deva lang 4.5 76.0 ×17.03 ×0.99 5% (0.6) 0% (0.0) 27% (3.2) 68% (7.9) 0% (0.0) match
jpn_Jpan lang 5.8 16.2 ×2.76 ×0.99 1% (0.6) 0% (0.0) 4% (2.2) 91% (53.9) 4% (2.3) match
kat_Geor lang 5.5 38.3 ×6.95 ×1.10 2% (0.6) 0% (0.0) 9% (2.1) 89% (21.6) 0% (0.0) match
kor_Hang lang 4.1 19.0 ×4.61 ×0.99 1% (0.6) 0% (0.0) 5% (2.5) 93% (45.6) 0% (0.2) match
rus_Cyrl lang 5.1 16.5 ×3.22 ×0.97 1% (0.6) 0% (0.0) 4% (2.2) 95% (55.7) 1% (0.4) match
tam_Taml lang 4.0 35.8 ×9.04 ×1.02 2% (0.6) 0% (0.0) 10% (2.8) 88% (24.1) 0% (0.0) match
tha_Thai lang 5.3 20.0 ×3.76 ×0.99 1% (0.6) 0% (0.0) 5% (2.6) 95% (45.7) 0% (0.0) match
added_normalized_dense modalities 6.1 26.8 ×4.38 ×0.98 2% (0.7) 0% (0.0) 3% (1.2) 95% (35.3) 0% (0.0) match
added_normalized_sparse modalities 5.4 42.6 ×7.95 ×0.99 6% (1.3) 0% (0.0) 8% (1.9) 87% (20.0) 0% (0.0) match
added_special_dense modalities 4.1 75.7 ×18.35 ×1.01 39% (5.0) 1% (0.1) 36% (4.6) 24% (3.0) 0% (0.0) match
added_special_sparse modalities 4.4 72.8 ×16.64 ×0.99 25% (3.2) 0% (0.0) 37% (4.7) 38% (5.0) 0% (0.1) match
agentic-traces modalities 3.5 37.4 ×10.55 ×0.99 7% (1.8) 0% (0.0) 15% (3.7) 79% (18.9) 0% (0.0) match
agentic_swe modalities 4.1 28.7 ×7.05 ×1.00 4% (1.3) 0% (0.0) 9% (2.9) 88% (28.8) 0% (0.0) match
code_mixed modalities 4.1 46.7 ×11.39 ×0.98 8% (1.6) 0% (0.0) 17% (3.3) 75% (14.1) 0% (0.0) match
math_latex modalities 3.8 40.6 ×10.74 ×0.98 9% (1.9) 0% (0.0) 16% (3.4) 79% (17.0) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.78 3.43 3.68 4.34 27.6 16.2 5.9 4.8 7.5× / 6.4× 4.4× / 3.7× 1.6× / 1.4× 1.3× / 1.1×
arb_Arab 1.15 2.96 2.32 4.13 31.5 19.3 6.8 5.2 13.6× / 7.6× 8.3× / 4.7× 2.9× / 1.6× 2.2× / 1.2×
ben_Beng 1.62 2.93 3.11 4.43 46.0 27.5 10.2 3.7 14.8× / 10.4× 8.8× / 6.2× 3.3× / 2.3× 1.2× / 0.8×
cmn_Hani 1.12 2.27 2.37 3.52 19.5 11.7 4.7 2.3 8.2× / 5.5× 4.9× / 3.3× 2.0× / 1.3× 1.0× / 0.7×
ell_Grek 0.59 2.96 2.20 4.58 28.8 17.2 6.2 4.8 13.1× / 6.3× 7.8× / 3.7× 2.8× / 1.4× 2.2× / 1.0×
eng_Latn 0.09 1.45 3.00 4.36 44.9 32.7 12.3 3.7 15.0× / 10.3× 10.9× / 7.5× 4.1× / 2.8× 1.2× / 0.8×
heb_Hebr 1.14 3.00 2.31 4.17 31.6 19.7 6.9 3.0 13.7× / 7.6× 8.5× / 4.7× 3.0× / 1.7× 1.3× / 0.7×
hin_Deva 1.51 3.11 3.21 4.80 48.9 30.5 11.1 4.0 15.3× / 10.2× 9.5× / 6.4× 3.5× / 2.3× 1.3× / 0.8×
jpn_Jpan 1.69 3.41 2.18 3.90 18.2 9.9 4.0 3.7 8.4× / 4.7× 4.6× / 2.5× 1.9× / 1.0× 1.7× / 0.9×
kat_Geor 1.54 2.55 2.09 3.10 17.5 11.3 4.2 2.0 8.4× / 5.6× 5.4× / 3.7× 2.0× / 1.4× 1.0× / 0.6×
kor_Hang 1.23 2.73 2.51 4.02 31.8 21.2 7.5 3.7 12.7× / 7.9× 8.5× / 5.3× 3.0× / 1.9× 1.5× / 0.9×
rus_Cyrl 1.16 2.93 2.15 3.92 27.3 17.0 5.9 2.5 12.7× / 7.0× 7.9× / 4.3× 2.8× / 1.5× 1.1× / 0.6×
tam_Taml 0.93 2.96 2.75 4.79 45.0 27.0 9.9 3.4 16.3× / 9.4× 9.8× / 5.6× 3.6× / 2.1× 1.2× / 0.7×
tha_Thai 1.52 2.59 2.60 3.67 28.6 16.6 6.7 3.0 11.0× / 7.8× 6.4× / 4.5× 2.6× / 1.8× 1.2× / 0.8×
added_normalized_dense 0.06 1.48 1.20 2.62 24.1 16.8 6.4 1.8 20.1× / 9.2× 14.0× / 6.4× 5.4× / 2.5× 1.5× / 0.7×
added_normalized_sparse 0.06 1.77 1.92 3.63 31.9 22.5 8.8 2.6 16.6× / 8.8× 11.7× / 6.2× 4.6× / 2.4× 1.3× / 0.7×
added_special_dense 0.06 1.48 4.56 5.98 97.1 73.3 20.9 3.2 21.3× / 16.2× 16.1× / 12.3× 4.6× / 3.5× 0.7× / 0.5×
added_special_sparse 0.06 1.48 4.73 6.15 62.0 45.4 15.1 3.3 13.1× / 10.1× 9.6× / 7.4× 3.2× / 2.5× 0.7× / 0.5×
agentic-traces 0.74 1.54 3.70 4.51 54.2 44.5 15.1 4.5 14.6× / 12.0× 12.0× / 9.9× 4.1× / 3.3× 1.2× / 1.0×
agentic_swe 0.68 1.46 2.87 3.66 56.9 53.0 16.1 3.4 19.8× / 15.5× 18.5× / 14.5× 5.6× / 4.4× 1.2× / 0.9×
code_mixed 0.10 1.45 3.27 4.63 54.0 50.8 15.5 3.9 16.5× / 11.7× 15.5× / 11.0× 4.7× / 3.4× 1.2× / 0.8×
math_latex 0.71 1.48 3.42 4.20 52.1 40.2 14.1 4.1 15.2× / 12.4× 11.7× / 9.6× 4.1× / 3.4× 1.2× / 1.0×
Not yet supported: t5-base
t5-base not supported

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Jul 22, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode-impl branch from fb2ddcb to f85971b Compare July 22, 2026 15:39
@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode-impl branch from 65406e4 to 037511e Compare July 22, 2026 20:51
@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode branch from 1ab19db to df3fd86 Compare July 24, 2026 10:08
Base automatically changed from feat/pipeline-decode to feat/train_encode_split July 24, 2026 10:35
@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode-impl branch 2 times, most recently from d5e3014 to 13fd07b Compare July 24, 2026 10:38
@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode-impl branch from f481829 to 9c13d97 Compare July 25, 2026 17:10
@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode-impl branch from 7d1e59e to 9fd3cf2 Compare July 26, 2026 20:27
@SBrandeis
SBrandeis force-pushed the feat/pipeline-decode-impl branch from 6810fa5 to 5845e8e Compare July 31, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SBrandeis @HuggingFaceDocBuilderDev