feat(thicktoken): replace tiktoken with in-repo Rust→WASM tokenizer engine#691

Merged

slvnperron merged 6 commits into

chore/thicktoken-dep-updates

Jul 2, 2026

Commits on Jul 1, 2026

feat(thicktoken): replace tiktoken with in-repo Rust→WASM tokenizer engine

slvnperron
and
claude
committed
fix(thicktoken): browser-compatible wasm inlining (atob fallback for Buffer)

slvnperron
and
claude
committed
feat(thicktoken): expose truncate modes (head/tail/middle) on TextTokenizer

slvnperron
and
claude
committed
chore: gitignore .gstack local tooling state

slvnperron
and
claude
committed
feat(thicktoken): vocab variants for edge — lite (cl50k) and micro (cl25k)

slvnperron
and
claude
committed
fix(thicktoken): address review — single-pass slice, dedupe CountOptions

slvnperron
and
claude
committed