feat(tokenization): refactor and improve tokenization support - #685

Draft

binaryaaron wants to merge 18 commits into

mainfrom

feature/full-tokenizer

Commits on Aug 10, 2026

feat(tokenization): add extensible tokenizer contracts
binaryaaron
committed
6f7cd76
feat(tokenization): migrate records and token cache
binaryaaron
committed
eeb1137
test(tokenization): allowlist revision fixture
binaryaaron
committed
4baec4a
test(smoke): provide tokenizer provenance
binaryaaron
committed
05f8407
feat(tokenization): own full tokenizer lifecycle
binaryaaron
committed
6050a47
style(tokenization): format persistence files
binaryaaron
committed
d55b400
feat(generation): dispatch canonical token prompts
binaryaaron
committed
6a4808b
test(tokenization): align prefill contract with generation
binaryaaron
committed
c153ca9
feat: gate vLLM generation on tokenizer parity
binaryaaron
committed
b126557
fix: fail closed on tokenizer initialization
binaryaaron
committed
05d2544
refactor: type tokenizer workload boundaries
binaryaaron
committed
9ef83a8
refactor(tokenization): replace tokenizer framework
binaryaaron
committed
43f1474
refactor(tokenization): clarify boundary parsing
binaryaaron
committed
e5a552d
test(data-processing): make shuffle cache state explicit
binaryaaron
committed
733faa6
refactor(tokenization): centralize mask validation
binaryaaron
committed
33fa702
ci: allowlist deterministic cache digest
binaryaaron
committed
99bd7a9
fix(tokenization): preserve cross-stage token identity
binaryaaron
committed
12746d9
fix(tokenization): enforce production boundary parity
binaryaaron
committed
9cee2aa