Appreciation for Kitoken – Excellent Performance and Clean Implementation

[sfallah]

Dear @Systemcluster,

Thank you for providing this excellent library.
I have previously worked with HuggingFace Tokenizers in Rust and am fairly familiar with both the Tokenizers and Tiktoken source code.
The performance of your implementation is outstanding, and I am particularly impressed by how clean and straightforward the code is.
I will be adopting Kitoken in my project as a replacement for Tokenizers.

I sincerely appreciate your excellent work.

[Systemcluster]

Thank you for the kind words @sfallah, I really appreciate it!

While I haven't had much time to work on Kitoken recently, I'm available in case there are any issues or questions. If there's something that needs to be done to support a new model let me know!

Best of luck for your project!