AutoTokenizer / Process does not support Magistral model

[benglewis]

System Info

Transformers version: 4.57.1
Python: 3.13
Linux and macOS (seems irrelevant)

Who can help?

It looks like the same issue is also breaking the AutoTokenizer for Magistral:

Traceback (most recent call last):
File "./debug_mistral.py", line 23, in
tokenizer = AutoTokenizer.from_pretrained("mistralai/Magistral-Small-2509")
File "./.pixi/envs/dev/lib/python3.13/site-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained
tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "./.pixi/envs/dev/lib/python3.13/site-packages/transformers/models/auto/auto_factory.py", line 815, in getitem
raise KeyError(key)
KeyError: <class 'transformers.models.mistral3.configuration_mistral3.Mistral3Config'>

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Magistral-Small-2509")

Expected behavior

It should work and not crash.

[benglewis]

This appears similar to #36968 and I believe that I can fix it similarly

[vasqu]

#42198 should fixes your issue no? Magistral is also mistral3. Note that there hasn't been a patch or release so you need to be on main.

Note that you will need mistral-common package as well. Other tokenizers don't work for now

[benglewis]

@vasqu Ohh, yeah, you are right, it seems to work with the latest code from main. My apologies 🙏

[benglewis]

Is there any chance that this fix can be backported to a transformers 2.x release? I ask since I see that main is for transformers 3.x already

[vasqu]

Ehm, we are on version v4 and go to the next major v5 soon. I think we planned backporting certain PRs for v4 as patch release cc @ArthurZucker @Cyrilvallez

[ArthurZucker]

This one will be patch for v4 this week!