Faster Whitespace PreTokenizer (Drop-in Replacement)

[8ria]

🚀 Faster Whitespace PreTokenizer (Drop-in Replacement)

This PR replaces the current Whitespace pre-tokenizer implementation with an optimized version that achieves consistent 10–30% performance improvements across short, medium, and long inputs — with identical output behavior.

🔧 Changes

• ✅ Replaced whitespace.rs with a new implementation using manual char_indices() traversal (no regex).

• ✅ Added a WhitespaceSplit variant for simpler whitespace-only tokenization.

• ✅ Updated unit tests to verify correctness and output compatibility.

• ✅ Added whitespace_bench.rs in benches/, using Criterion.

• ✅ Updated Cargo.toml to register the benchmark:

  [[bench]]
  name = "whitespace_bench"
  harness = false
  

---

⚡ Benchmarks (Criterion)

Benchmarks were run across five full test cycles to minimize outliers and assess stability.

🧪 Inputs

• Short: "Hello world!" (~10–20 chars)

• Medium: Sentences with spaces, tabs, punctuation (~100–150 chars)

• Long: Large paragraphs repeated 3× (~5,000+ chars)

✅ Optimized Version (New)

Input Type	Avg. Time	Change
Short	555 ns	10–15% faster
Medium	3.78–4.28 µs	5–30% faster
Long	50.1–63 µs	5–15% faster

Across repeated runs, the optimized implementation consistently showed faster or equivalent performance with no regressions. Variance in outliers decreased as well.

---

🧬 Output Compatibility

• Produces the exact same pre-tokenization splits as the current version.

• Word boundaries, punctuation, and whitespace are handled identically.

• Includes robust unit tests verifying span offsets and output strings.

---

🧠 Technical Improvements

• No regex: replaced with a simple and cache-efficient char_indices() iterator loop.

• Span classification is done in-place: word, whitespace, punctuation.

• Avoids unnecessary allocations or dependencies.

• Fully backward-compatible and implements impl_serde_type!.

---

📎 Related Issue

Addresses the motivation in #1820:

Cannot download test data: 'make test' and direct links fail with "Repository not found" / 404

While this PR doesn't solve that issue directly, it improves local testing coverage and adds Criterion-based benchmarks so others can independently validate behavior and performance — without needing external test datasets.

---

🙌 Closing

Whitespace is used everywhere in tokenization — from LLM pretraining to inference. Optimizing its performance has cascading effects at scale, especially in multithreaded and batched pipelines.

Thank you for maintaining this incredible library. Let me know if you'd like additional changes — such as splitting this into a side-by-side version (WhitespaceFast) for testing — but this PR is designed as a safe drop-in upgrade.

Best,
AndriaK