2024.1.0.2

What's Changed

• Fixed prebuild tokenizers on Windows by @ilya-lavrenov in #141

Full Changelog: 2024.1.0.1...2024.1.0.2

2024.1.0.1

What's Changed

• Llama3 Tokenizer Support
• Add not-add-special-tokens flag to CLI conversion tool

Full Changelog: 2024.1.0.0...2024.1.0.1

2024.1.0.0

What's Changed

• New operations:
  • TrieTokenizer
  • VocabEncoder
  • EqualStr
  • RaggedToSparse
  • RaggedToRagged
  • FuzeRagged
• Update existing operations:
  • Add max_splits argument to RegexSplit
  • Add encoding argument to CaseFold
• Add new and update existing TensorFlow translators for TextVectorization layer partial support.
• RWKV tokenizer support.
• New way to get OpenVINO Tokenizers - build from files. Supports RWKV tokenizer.
• Update tokenizer operation caching mechanism for OpenVINO model caching support
• SentencePiece tokenizer changes and fixes:
  • Update to 0.2.0 version
  • Use constant 0 as mask hide token by @as-suvorov in #90
  • Sentencepiece BOS Token Detection
• Fix multi-input model merging by @yas-sim in #53

New Contributors

• @dependabot made their first contribution in #30
• @yas-sim made their first contribution in #53
• @as-suvorov made their first contribution in #90
• @akladiev made their first contribution in #102

Full Changelog: 2024.0.0.0...2024.1.0.0

2024.0.0.0

What's Changed

• Improve Regex Support - filter lookarounds, unsupported by re2
• Improve model coverage - T5 tokenizers, QWEN2
• Add tokenizer metadata to rt_info - EOS token id
• Support TensorFlow Text MUSE model conversion and inference

New Contributors

• @Wovchena made their first contribution in #20