Fix usize underflow in BPE / Strip decoders on empty / short input

[weixlu]

Describe the Bug

decode_chain() of BPE / Strip decoders compute an index with an unchecked usize subtraction that may underflow.

• bpe.rs: tokens.len() - 1 underflows when the token list is empty.
• strip.rs: chars.len() - i - 1 underflows when stop exceeds the token length

Reproduce

from tokenizers.decoders import BPEDecoder, Strip

BPEDecoder().decode([])
Strip(content="H", right=3).decode(["HH"])

Run the code above, and PanicException will be throw on debug build.

thread '<unnamed>' (3152788) panicked at /root/tokenizers/tokenizers/src/decoders/bpe.rs:28:17:
attempt to subtract with overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<string>", line 1, in <module>
pyo3_runtime.PanicException: attempt to subtract with overflow

Fix

The fix is quite straightforward, please take a look at the patch, it just contains 2 lines.

Notes

This issue was first reported by @devdan via fusil, the original report can be found here: https://github.com/devdanzin/fusil-extensions-findings/blob/main/tokenizers/reports/TOKENIZERS-0001-bpe-decoder-empty-underflow/report.md

[Sanjays2402]

the min() stops the underflow but the same short-token input still panics one line down: with Strip::new('H', 2, 1) and token "HH", start_cut ends at 2 and stop_cut at 1, so chars[start_cut..stop_cut] blows up with "slice index starts at 2 but ends at 1" (checked against the patched loop). needs a stop_cut.max(start_cut) before the slice.

[weixlu]

Thanks for your review! Sure, I have fix the case you mentioned. Also I tested the following:

from tokenizers.decoders import Strip
print(repr(Strip(content='H', right=3).decode(['HH'])))
print(repr(Strip(content='H', left=2, right=1).decode(['HH'])))
print(repr(Strip(content='H', left=1).decode(['xHy'])))
print(repr(Strip(content='H', left=1, right=1).decode(['HHHH'])))

Now it print '' '' 'xHy' 'HH', looks good