perf(pipeline): synthesise the metaspace splitter, with its exceptions baked in

[ArthurZucker]

Stacked on #2296.

Two changes, both removing an all-or-nothing check that a single vocabulary entry could defeat.

1. A declared pre-tokenizer that cannot cut is the None case

gemma-3 declares Split on the literal " ", and its normalizer replaces every space with ▁ before that splitter runs. So it is installed, executes, and matches nothing — the model gets the whole document as one span:

model	normalizer	pre_tokenizer	spans on 2000 chars
gpt2	no	ByteLevel	465
llama-2	yes	None	1
gemma-3	yes	Split	1

tokenizers 0.23.1 does the same, so the ids were already right — but every word merges inside one span, and the WordCache can only hit on a byte-identical repeat of the whole document. #2296 fixed this for llama-2 by matching on pre_tokenizer: null; gemma-3 declares a no-op splitter instead and slipped through.

It cannot be decided from the pre-tokenizer alone, because whether it can match depends on what the normalizer did first. So probe the pair: a declared splitter that yields one span for a multi-word normalized sample cannot cut anything. Errs toward "it cuts", so a failed probe leaves the declared config exactly as written.

2. The cut guard is a set of bytes, not a yes/no

metaspace_cuts_are_safe asked whether cutting was safe anywhere, so one piece took the answer away from the whole vocabulary. gemma-3 holds exactly one piece with an interior ▁ out of 262,144:

'>▁</'

llama-2 has zero, which is the only reason it worked there.

metaspace_cut_guard now returns the 256-bit set of bytes that may precede an interior ▁, and the synthesised splitter carries it, skipping a cut after one of those bytes. For gemma-3 that is the single byte > — so it cuts everywhere except immediately after a >.

Skipping a cut is always sound. Not cutting is what the reference does — it merges the whole span — so any subset of the safe cuts reproduces the reference's ids. Only cutting where a piece spans the boundary can change them, and that is exactly what the set forbids.

Numbers

10 kB documents each encoded once with the cache carried across them:

cell	before	after	gain
gemma-3/chat-mistral	17	76 MB/s	4.48×
gemma-3/english	17	58 MB/s	3.39×
gemma-3/xnli	29	73 MB/s	2.48×
gemma-3/code	19	41 MB/s	2.17×
gemma-3/chinese	70	80 MB/s	1.14×

llama-2 and gpt2 are unchanged. llama-2 medians over three runs: 60/52/164/88 MB/s on english/code/chinese/xnli against 58/54/163/89 before — spread ≤ 3, so the one 0.89× I first measured was a single-sample artifact, not a regression. Their guard sets are empty, so the cost is one bitmap probe per ▁.

Exactness

Full benchmark matrix, 8 models × 30 corpora: 203 cells byte-exact against tokenizers 0.23.1, 29 mismatched — the same albert (Unigram) cells that already fail. No new mismatch. 352 tests pass.

the_synthesised_splitter_never_changes_the_ids pins ids for text containing > before a space, which a splitter that ignored the guard would cut straight through.

Prior art

gigatoken reaches the same conclusion by a different route: it pattern-matches gemma's config specifically (its comment names "gemma-3/4"), resolves it to "no metaspace config", then independently picks WordSplit::SpaceRuns from the vocab — and keeps a cross_prev bitset of predecessor bytes plus up to 32 crossing pieces, guarding those boundaries with a memcmp. This PR is the same idea with the exception baked into the splitter at load rather than checked as a special case, and byte-level rather than piece-level, which is more conservative and needs no memcmp.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.