@@ -13,80 +14,120 @@
+
+ -Provides an implementation of today's most used tokenizers, with a focus on performance and -versatility. +# β‘ faster-whitespace-pretok -## Main features: +**This is a performance fork of Hugging Faceβs `tokenizers`**, focused on optimizing the `Whitespace` PreTokenizer. +It preserves all original functionality and directory layout of `tokenizers/tokenizers` for compatibility β including benchmark support and test coverage. - - Train new vocabularies and tokenize, using today's most used tokenizers. - - Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes - less than 20 seconds to tokenize a GB of text on a server's CPU. - - Easy to use, but also extremely versatile. - - Designed for research and production. - - Normalization comes with alignments tracking. It's always possible to get the part of the - original sentence that corresponds to a given token. - - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs. +> π§ Pull Request: [huggingface/tokenizers#1822](https://github.com/huggingface/tokenizers/pull/1822) -## Performances -Performances can vary depending on hardware, but running the [~/bindings/python/benches/test_tiktoken.py](bindings/python/benches/test_tiktoken.py) should give the following on a g6 aws instance: - +--- +## π Whatβs New in This Fork? -## Bindings +### β Optimized `Whitespace` PreTokenizer +- Replaced regex-based logic with a cache-efficient manual traversal using `char_indices()`. +- No change to output behavior β identical span offsets and splits. +- Drop-in compatible with all existing pipelines. -We provide bindings to the following languages (more to come!): - - [Rust](https://github.com/huggingface/tokenizers/tree/main/tokenizers) (Original implementation) - - [Python](https://github.com/huggingface/tokenizers/tree/main/bindings/python) - - [Node.js](https://github.com/huggingface/tokenizers/tree/main/bindings/node) - - [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo) +### β Criterion Benchmark Added +- Added `benches/whitespace_bench.rs` +- Measures short, medium, and long inputs +- Registered in `Cargo.toml`: -## Installation - -You can install from source using: -```bash -pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python +```toml +[[bench]] +name = "whitespace_bench" +harness = false ``` -our install the released versions with +### β Additional Variant: `WhitespaceSplit` + +* Lightweight alternative that only splits on whitespace (no span tracking). +* Useful for standalone benchmarking or ultra-fast preprocessing. + +--- + +## π Benchmarks + +Benchmarked using Criterion across 5 test cycles: + +| Input Type | Avg. Time (Original) | Avg. Time (Optimized) | Speedup | +| ---------- | -------------------- | --------------------- | -------- | +| Short | \~620 ns | \~555 ns | β 10β15% | +| Medium | 4.3 Β΅s | 3.7β4.0 Β΅s | β 5β30% | +| Long | \~60β74 Β΅s | \~50β63 Β΅s | β 5β15% | + +--- + +## β‘ Visual Benchmark + + +* π¬ Output remains identical to the original `Whitespace` implementation. +* π§ͺ Verified with robust unit tests. +* π Consistent results across runs. + +--- + +## π§ Technical Highlights + +* β No regex (avoids unnecessary overhead) +* β Manual `char_indices()` loop for precision and cache-friendliness +* π§ Inline span classification +* π‘ Zero additional dependencies +* π Fully backwards-compatible with `impl_serde_type!` + +--- + +## π Related Issue + +Improves local benchmarking infrastructure and test coverage related to: +[#1820](https://github.com/huggingface/tokenizers/issues/1820) + +This PR does not fix dataset download issues directly, but **adds independent, reproducible local benchmarking support**. + +--- + +## π§ Installation & Usage + +Clone the fork and use it as a **drop-in `tokenizers/tokenizers` replacement**: ```bash -pip install tokenizers +git clone --branch faster-whitespace-pretok https://github.com/8ria/tokenizers.git +cd tokenizers/tokenizers +cargo bench --bench whitespace_bench ``` - -## Quick example using Python: -Choose your model between Byte-Pair Encoding, WordPiece or Unigram and instantiate a tokenizer: +Use your own sample inputs by editing `whitespace_bench.rs`. + +--- -```python -from tokenizers import Tokenizer -from tokenizers.models import BPE +## π¦ Python Installation (from this fork) -tokenizer = Tokenizer(BPE()) +To use the Python bindings with the optimized version: + +```bash +pip install git+https://github.com/8ria/faster-whitespace-pretok.git#subdirectory=bindings/python ``` -You can customize how pre-tokenization (e.g., splitting into words) is done: +> All Python-facing behavior remains identical to upstream `tokenizers`. -```python -from tokenizers.pre_tokenizers import Whitespace +--- -tokenizer.pre_tokenizer = Whitespace() -``` +## π Why This Matters -Then training your tokenizer on a set of files just takes two lines of codes: +Whitespace pre-tokenization is executed millions of times in ML workflows: -```python -from tokenizers.trainers import BpeTrainer +* LLM inference +* Prompt batching +* Offline training pipelines -trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]) -tokenizer.train(files=["wiki.train.raw", "wiki.valid.raw", "wiki.test.raw"], trainer=trainer) -``` +Even small improvements in this phase **compound at scale** β especially when parallelized. -Once your tokenizer is trained, encode any text with just one line: -```python -output = tokenizer.encode("Hello, y'all! How are you π ?") -print(output.tokens) -# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"] -``` +This fork improves efficiency **without changing outputs or APIs**. + +--- + +## π« Contact -Check the [documentation](https://huggingface.co/docs/tokenizers/index) -or the [quicktour](https://huggingface.co/docs/tokenizers/quicktour) to learn more! +**AndriaK** - [hey@andriaK.com](mailto:hey@andriaK.com) - [GitHub](https://github.com/8ria) diff --git a/comparison.png b/comparison.png new file mode 100644 index 000000000..9912efe31 Binary files /dev/null and b/comparison.png differ diff --git a/tokenizers/Cargo.toml b/tokenizers/Cargo.toml index 6ed8498cf..c8fd6274d 100644 --- a/tokenizers/Cargo.toml +++ b/tokenizers/Cargo.toml @@ -40,6 +40,10 @@ harness = false name = "llama3_benchmark" harness = false +[[bench]] +name = "whitespace_bench" +harness = false + [dependencies] rand = "0.9" onig = { version = "6.5.1", default-features = false, optional = true } diff --git a/tokenizers/benches/whitespace_bench.rs b/tokenizers/benches/whitespace_bench.rs new file mode 100644 index 000000000..eafece259 --- /dev/null +++ b/tokenizers/benches/whitespace_bench.rs @@ -0,0 +1,32 @@ +use std::hint::black_box; +use criterion::{criterion_group, criterion_main, Criterion}; +use tokenizers::pre_tokenizers::whitespace::Whitespace; +use tokenizers::tokenizer::PreTokenizedString; +use tokenizers::PreTokenizer; + +fn bench_whitespace(c: &mut Criterion) { + let tokenizer = Whitespace; + + let short_text = "Hello world!"; + let medium_text = "This is a sentence with multiple spaces. And\tsome\nnewlines. Also, punctuation! Like.this? And unicode: γγγ«γ‘γ―δΈη ππ½"; + let long_text = "The quick brown fox jumps over the lazy dog. This is a much longer piece of text designed to test the performance of the whitespace pre-tokenizer on a substantial input. It includes various forms of whitespace, such as multiple consecutive spaces, tabs\tbetween words, and newlines.\nIt also mixes in different types of characters, including numbers (123), special symbols (!@#$%^&*()), and some common emojis like ππβ¨. The goal is to ensure that the tokenizer correctly splits the text into tokens while maintaining its performance characteristics across a diverse set of linguistic features. This paragraph will be repeated several times to create a truly long input for comprehensive benchmarking. The quick brown fox jumps over the lazy dog. This is a much longer piece of text designed to test the performance of the whitespace pre-tokenizer on a substantial input. It includes various forms of whitespace, such as multiple consecutive spaces, tabs\tbetween words, and newlines.\nIt also mixes in different types of characters, including numbers (123), special symbols (!@#$%^&*()), and some common emojis like ππβ¨. The goal is to ensure that the tokenizer correctly splits the text into tokens while maintaining its performance characteristics across a diverse set of linguistic features. This paragraph will be repeated several times to create a truly long input for comprehensive benchmarking. The quick brown fox jumps over the lazy dog. This is a much longer piece of text designed to test the performance of the whitespace pre-tokenizer on a substantial input. It includes various forms of whitespace, such as multiple consecutive spaces, tabs\tbetween words, and newlines.\nIt also mixes in different types of characters, including numbers (123), special symbols (!@#$%^&*()), and some common emojis like ππβ¨. The goal is to ensure that the tokenizer correctly splits the text into tokens while maintaining its performance characteristics across a diverse set of linguistic features."; + + let samples = vec![ + ("short_unique", short_text), + ("medium_unique", medium_text), + ("long_unique", long_text), + ]; + + for (label, text) in samples { + c.bench_function(&format!("whitespace_pretokenizer_{}", label), |b| { + b.iter(|| { + let mut s = PreTokenizedString::from(black_box(text)); + tokenizer.pre_tokenize(&mut s).unwrap(); + black_box(&s); + }); + }); + } +} + +criterion_group!(benches, bench_whitespace); +criterion_main!(benches); diff --git a/tokenizers/src/pre_tokenizers/whitespace.rs b/tokenizers/src/pre_tokenizers/whitespace.rs index 20cfb6519..9fd8bb9b2 100644 --- a/tokenizers/src/pre_tokenizers/whitespace.rs +++ b/tokenizers/src/pre_tokenizers/whitespace.rs @@ -1,12 +1,18 @@ -use std::sync::LazyLock; - -use regex::Regex; - +use std::iter::Peekable; +use std::str::CharIndices; use crate::tokenizer::{ - pattern::Invert, PreTokenizedString, PreTokenizer, Result, SplitDelimiterBehavior, + PreTokenizedString, + PreTokenizer, + Result, + SplitDelimiterBehavior, }; use crate::utils::macro_rules_attribute; +/// A pre-tokenizer that splits text into tokens by whitespace while separating +/// word characters (alphanumeric + underscore) from punctuation characters. +/// +/// This tokenizer groups consecutive word characters together and consecutive +/// punctuation characters together, removing all whitespace in the process. #[derive(Clone, Debug, PartialEq, Eq)] #[macro_rules_attribute(impl_serde_type!)] pub struct Whitespace; @@ -17,17 +23,74 @@ impl Default for Whitespace { } } +// Helper function to check if a character is a word character (alphanumeric or underscore) +fn is_word_char(c: char) -> bool { + c.is_alphanumeric() || c == '_' +} + +/// Helper function to extend the end index while a predicate holds true +fn extend_while