Skip to content

Pull requests: huggingface/tokenizers

New pull request New
144 Open 951 Closed
144 Open 951 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

chore(deps): bump brace-expansion from 1.1.13 to 1.1.18 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2273 opened Aug 1, 2026 by dependabot Bot Loading…
1
perf: bundle all ongoing optimization work
#2272 opened Jul 31, 2026 by SBrandeis Contributor • Draft
1
fast atomsplit::literal
#2271 opened Jul 31, 2026 by SBrandeis Contributor • Draft
1
experiment(ai): faster cache
#2267 opened Jul 30, 2026 by SBrandeis Contributor • Draft
1
perf: chunk sentencepiece
#2266 opened Jul 29, 2026 by SBrandeis Contributor • Draft
1
feat(atomsplit): run deepseek's pre-tokenizer with no regex backend
#2265 opened Jul 29, 2026 by SBrandeis Contributor • Draft
1
perf(bpe): WordCache
#2262 opened Jul 29, 2026 by SBrandeis Contributor Loading…
26
perf(scratch): reuse Scratch buffers across encode() calls
#2261 opened Jul 29, 2026 by SBrandeis Contributor Loading…
3
chore(deps): bump braces from 3.0.2 to 3.0.3 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2251 opened Jul 27, 2026 by dependabot Bot Loading…
1
chore(deps-dev): bump http-proxy-middleware from 2.0.9 to 2.0.10 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2250 opened Jul 27, 2026 by dependabot Bot Loading…
1
chore(deps): bump uuid and webpack-dev-server in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2249 opened Jul 27, 2026 by dependabot Bot Loading…
1
Fix usize underflow in BPE / Strip decoders on empty / short input
#2245 opened Jul 27, 2026 by weixlu Loading…
2
chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2244 opened Jul 25, 2026 by dependabot Bot Loading…
1
Update fancy-regex dependency to 0.18
#2243 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
Update monostate dependency to v1
#2242 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
Perf/bpe merge
#2241 opened Jul 24, 2026 by ArthurZucker Collaborator Loading…
1
perf(pipeline): BPE word cache: bench different cache architectures
#2240 opened Jul 24, 2026 by SBrandeis Contributor • Draft
1
Fix panic in Strip decoder on short all-content tokens
#2239 opened Jul 24, 2026 by v-code01 Loading…
feat(pipeline+bindings): Support encoding pairs of sequences
#2237 opened Jul 23, 2026 by SBrandeis Contributor • Draft
2
perf(tk-encode): SIMD-first encode pipeline — fused split cache + de-virtualized BPE
#2234 opened Jul 23, 2026 by ArthurZucker Collaborator • Draft
bench: gigatoken per-step profile + take/have/skip analysis
#2233 opened Jul 23, 2026 by ArthurZucker Collaborator • Draft
1
implement ultra-basic, erroneous decode
#2232 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
feat(bindings): Python bindings rework, plugging PipelineTokenizer
#2229 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
chore(deps): bump shell-quote from 1.8.1 to 1.10.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2224 opened Jul 20, 2026 by dependabot Bot Loading…
1
bench(pipeline): add sebpop/upstream as a second reference series
#2221 opened Jul 20, 2026 by SBrandeis Contributor • Draft
2
ProTip! Updated in the last three days: updated:>2026-07-31.