HuggingFace Tokenizers on Android (HFTA)

Reference implementation using HuggingFace's (HF) tokenizers in Android.

Demo Video

UI to show text to tokens via the tokenizers library in real-time on Android at demo/demo.mp4:

demo.mp4

Try a Tokenizer

1. Find a model you want to test on HF, e.g., Google's gemma-3-4b-it
2. Download and add the tokenizer.json to app/src/main/assets named gemma-3-4b-it.json
3. Modify SELECTED_TOKENIZER in app/build.gradle.kts

Features

• Run any HuggingFace's (HF) tokenizers on-device in Android.
• rust to java NDK bindings of HF's tokenizers in rs-hfta
• Use of JNI bindings between rust and Android
• Parameterized instrumentation tests (runs on-device)
• compiler optimizations to reduce lib filesize

Implementation Details

Run any HF's tokenizer on Android using the associated tokenizers.json from huggingface.co. To achieve that, the HF library is built via rust into a shared library and uses Java Native Interface (JNI) to load the library.

Thanks to

• Hugging Face's tokenizers library
• Qualcomm's Genie Library has a rust to C++ static library implementation of HF's tokenizers at qairt/2.34.0.250424/examples/Genie/Genie/src/qualla/tokenizers/rust
• Shubham Panchal's Sentence-Embeddings-Android
• Rust's profile docs