perf: bundle all ongoing optimization work - #2272

Closed

SBrandeis wants to merge 13 commits into

feat/train_encode_splitfrom

perf/all-optims

Commits on Jul 31, 2026

perf(cache): serve repeated words from a shared WordCache (#2262)
SBrandeis
committed
bb360a4
perf(atomsplit): SIMD literal pattern search (#2271)
SBrandeis
committed
a2bde64
perf(bpe): cut whole-chunk SentencePiece text into words, proven from the vocabulary (#2266)
SBrandeis
committed
9769d67
perf(pipeline): fuse Metaspace prepend and replace into one normalizer pass
SBrandeis
committed
0484ea4
test(pipeline): pin that the fused normalizer and proven cuts both fire
SBrandeis
committed
820d436
perf(bpe): seed byte-level words with proven whole-character tokens
SBrandeis
committed
8cb43d2
bench: fold_ab example for two-build process-level A/B throughput
SBrandeis
committed
8da5522
perf(normalize): route every literal cut through atomsplit::Literal

SBrandeis
and
claude
committed
2cb2e0e
perf(cache): pack the word key with overlapping reads, not a buffer copy

SBrandeis
and
claude
committed
8092462

Commits on Aug 1, 2026

perf(pipeline): fuse the native FSM scans into the model step

SBrandeis
and
claude
committed
459be6c
perf(pretok): boundary-mask scanners for the regex-shaped FSMs

SBrandeis
and
claude
committed
24813cf
perf(pipeline): read pending rewrites off the raw text instead of writing them

SBrandeis
and
claude
committed
ac50768
fix(bench): make the stage ladder measure the route encode actually takes

SBrandeis
and
claude
committed
972adc0