Wordpiece tokenization directly in Java?

[lintool]

Apparently, there's already an implementation in djl: https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/modality/nlp/bert/WordpieceTokenizer.java

Javadoc:
https://javadoc.io/doc/ai.djl/api/latest/ai/djl/modality/nlp/bert/WordpieceTokenizer.html

[ToluClassics]

@lintool Apparently DJL has an implementation for loading Huggingface Tokenizers that works. Wondering if we can explore that instead?

https://github.com/deepjavalibrary/djl/tree/master/extensions/tokenizers

[lintool]

Yup, that's a promising lead.

[lintool]

We've done this a while ago... closing!