Support splitting to chunks during encode

[xrl1]

Hello,
I use the Rust library directly and call encode on a given text.
Because I use BERT with a limited token window, very often the encoded tokens are larger than the max sequence_length.
Currently, my code splits the tokens, but it has a few drawbacks:

1. There is no API to read and split the encodings into chunks, so the use needs to slice into and copy all the required variables from the Encoding struct.
2. The caller needs to reserve BERT's special tokens and append them.

In addition, I have a custom logic to find where to split the tokens (for a classification task), so passing to encode a callback that also finds the splits can be an additional improvement that can be useful.

I want to contribute this feature request myself and introduce a new parameter to encode, or a new function - encode_with_chunks, but I'd like to get feedback on this approach first :)

[ArthurZucker]

I can review, we will accept it if this issue is voted a lot by community!

[xrl1]

Thanks!
I dug into the code to understand how this feature could be implemented, but I eventually concluded that adding it inside the library would be inefficient. It’s still more practical to handle the splitting on the caller side. Here’s why:

1. The encode function accepts both Single and Dual EncodeInput types.
   My use case only involves a single sequence, but for pair-encoding, I believe there’s no “correct” way to split two sequences into aligned chunks. For example, if one input in the pair is significantly longer and produces more chunks than the other, the merging logic becomes use-case-dependent.
   I can overcome this by changing the input trait bound to Into<InputSequence<'s>>, but I'm not sure this will get accepted.

2. Because handling of add_special_tokens happens separately from encode_single_sequence, implementing this feature generically would require two copy operations on the Encoding struct:
   one copy to split the content into chunks before post-processing, and another copy inside the BERT process_encodings step.
   On the caller side, it’s easier to anticipate the post-processing behavior and apply it inline while copying the chunks from the initial Encoding. This avoids the second copy entirely.

@ArthurZucker, please let me know if I’m overlooking something. Otherwise, we can close the issue.
I’d be happy to check out other contribution opportunities! :)