Is transformers necessary or tokenizers is enough?

[felipemello1]

Hi guys, thanks for open sourcing this! I worked in Torchtune and now i am in TorchTitan. Usually we want to keep the requirements lightweight, and transformers is a heavy one, so we try to import tokenizers only.

noob question: Is there a real need to depend on transformers or maybe it could be made optional and the library could lean more on tokenizers?

[hallerite]

for VLMs we unfortunately need transformers, but I am open to making it an optional dep for text-only renderers

[felipemello1]

thanks @hallerite , that would be awesome! We would love to use it in Titan.

my 2c: I took a very brief look and it seems that all of the transformers imports are just for type hinting and tokenizer loading. Maybe the type hint could be relaxed, and for tokenizer loading, its up to people to provide one that satisfies the expected protocol.

edit: i missed these two

renderers/renderers/client.py

Line 260 in 01aeb00

from transformers.feature_extraction_utils import BatchFeature

renderers/renderers/qwen35.py

Line 157 in 01aeb00

from transformers import AutoProcessor

[hallerite]

yeah, though I believe the client could also be structured in a way that it is more agnostic & not tailored to vLLM, though for VLMs it will be hard to entirely get away without transformers, I'm afraid

[felipemello1]

though for VLMs it will be hard to entirely get away without transformers, I'm afraid

The use seems to be self-contained. For qwen35.py, i saw:

proc = self._get_processor()
out = proc.image_processor(images=[pil], return_tensors="np")
merge_size = proc.image_processor.merge_size

If more people from the community express interest, it seems that it could be replaced with a simple protocol, and then its up to the user to import transformers or provide their own processor in self._get_processor().

But we are prioritizing only text for now, so it wouldn't be a p0 for us.