🚨 Clean up `image-text-to-text` pipeline #42430

ebezzam · 2025-11-26T18:08:51Z

What does this PR do?

This PR standardizes / cleans up image-to-text-text pipeline to use the chat template logic from the base pipeline class.

TODO:

add function to convert from OpenAI format to HF-format to address this failing test after 🚨 Image-text pipeline expects correctly formatted chat #42359 removed add_images_to_messages (see here)

Co-authored-by: Anton Vlasjuk <[email protected]>

github-actions · 2025-11-26T18:09:51Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: bark

ebezzam

@zucchini-nlp I started a PR on how image-text-to-text can be further cleaned up! What do you think about altogether removing the images input to __call__? That could significantly clean up the logic there

~~don't mind the changes on the other files, I'm waiting for #42326 to be merged, but currently blocked because of unrelated failing test~~ it's merged!

ebezzam · 2025-11-26T18:10:15Z

src/transformers/pipelines/image_text_to_text.py


        # encourage the user to use the chat format if supported
        if getattr(self.processor, "chat_template", None) is not None:
-            logger.warning_once(


be stricter to force chat template usage?

ebezzam · 2025-11-26T18:10:35Z

src/transformers/pipelines/image_text_to_text.py

-            # We have one or more prompts in list-of-dicts format, so this is chat mode
-            if isinstance(text[0], dict):
-                return super().__call__(Chat(text), **kwargs)
-            else:
-                chats = [Chat(chat) for chat in text]  # 🐈 🐈 🐈
-                return super().__call__(chats, **kwargs)


Can let base class handle casting to Chat

ebezzam · 2025-11-26T18:10:45Z

src/transformers/pipelines/image_text_to_text.py

-            # We have one or more prompts in list-of-dicts format, so this is chat mode
-            if isinstance(images[0], dict):
-                return super().__call__(Chat(images), **kwargs)
-            else:
-                chats = [Chat(image) for image in images]  # 🐈 🐈 🐈
-                return super().__call__(chats, **kwargs)


Can let base class handle casting to Chat

HuggingFaceDocBuilderDev · 2025-11-26T18:17:05Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ebezzam

@zucchini-nlp I merged with main after #42326 was merged, and did some more cleanup/fixes 🙈

including reintroducing the OpenAI chat format conversion. Btw latest version seems to be different than what was currently in the tests? (explanations/links in my comments)

I also fixed some tests 🙂

ebezzam · 2025-11-27T14:00:17Z

tests/pipelines/test_pipelines_any_to_any.py

    @slow
    def test_small_model_pt_token_text_only(self):
        pipe = pipeline("any-to-any", model="google/gemma-3n-E4B-it")
-        text = "What is the capital of France? Assistant:"


switched to chat template usage

we still need a test to check if the pipe can work in text-only mode imo. Users might want to pass only text sometimes and continue a multi-turn conversation

was there already a test for this? or did I remove it accidentally? 🙈

ebezzam · 2025-11-27T14:00:41Z

tests/pipelines/test_pipelines_image_text_to_text.py

    @require_torch
    def test_small_model_pt_token_text_only(self):
        pipe = pipeline("image-text-to-text", model="llava-hf/llava-interleave-qwen-0.5b-hf")
-        text = "What is the capital of France? Assistant:"


switched to chat template usage

ebezzam · 2025-11-27T14:01:14Z

tests/pipelines/test_pipelines_image_text_to_text.py

-        image = "./tests/fixtures/tests_samples/COCO/000000039769.png"
-        text = "<image> What this is? Assistant: This is"


switched to chat template usage

ebezzam · 2025-11-27T14:02:01Z

tests/pipelines/test_pipelines_image_text_to_text.py

+                        "type": "input_image",
+                        "image_url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg",
                    },
-                    {"type": "text", "text": "Describe this image in one sentence."},
+                    {"type": "input_text", "text": "Describe this image in one sentence."},


Perhaps OpenAI structure changed since the test was created? https://platform.openai.com/docs/guides/images-vision?api-mode=responses#giving-a-model-images-as-input

hmm, right! W never got bug reports on this, and I just realized we don't promote openAI format in docs. Makes me wonder, should we keep supporting it ?

I would lean towards no for less maintenance overhead? (esp if indeed OpenAI changes their structure)

If people didn't complain about it not working, I can only guess that it wasn't used much.

@Rocketknight1 may have thoughts on this?

tests/pipelines/test_pipelines_image_text_to_text.py

ebezzam · 2025-11-27T14:14:05Z

src/transformers/utils/chat_template_utils.py

+            # Convert OpenAI fields to Transformers fields
+            for content in message["content"]:
+                if isinstance(content, dict):
+                    content_type = content.get("type")
+                    # (27 Nov 2025) Image/vision fields: https://platform.openai.com/docs/guides/images-vision
+                    if content_type == "input_image":
+                        content["type"] = "image"
+                        content["image"] = content.pop("image_url")
+                    if content_type == "input_text":
+                        content["type"] = "text"


re-introduced the conversion (deleted in #42359 (review)) according to structure of input dict on this page: https://platform.openai.com/docs/guides/images-vision?api-mode=responses#giving-a-model-images-as-input

Linked to this thread if we keep OpenAI conversion: https://github.com/huggingface/transformers/pull/42430/files#r2568843132

ebezzam · 2025-11-27T15:10:13Z

The last commit is more breaking to enforce chat template usage if the model supports it.

zucchini-nlp · 2025-12-02T13:41:16Z

src/transformers/pipelines/image_text_to_text.py

        self,
-        image: Union[str, "Image.Image"] | None = None,
        text: str | None = None,
+        images: Union[str, "Image.Image"] | None = None,


can be a bit breaking if someone used to pass positional args, but I see the reason behind. Prob can be squeezed in v5-release

Also, we'd need to update docs, afair there are examples with positional args for the pipe

Definitely breaking!

It was an idea for phasing out the images argument and nudging chat template usage (linked to this thread). Because I thought it makes more sense as the text argument rather than the images argument for usage like so?

outputs = pipe(chat_template)

rather than:

outputs = pipe(text=chat_template)

But could also keep the original order and only support chat templates passed to images and not to text? Namely adjusting this logic to raise an error if images is not a chat and text is not None.

Which would be similar to this code path in the original:

transformers/src/transformers/pipelines/image_text_to_text.py

Line 291 in 2e93004

elif text is None and _is_chat(images):

zucchini-nlp · 2025-12-02T13:44:49Z

src/transformers/pipelines/image_text_to_text.py

+        if getattr(self.processor, "chat_template", None) is not None:
            if images is not None:
                raise ValueError(
-                    "Invalid input: you passed `chat` and `images` as separate input arguments. "
-                    "Images must be placed inside the chat message's `content`. For example, "
+                    "The model supports chat templates and you passed an `images` argument. A chat template must be "
+                    "passed with images placed inside the chat message's `content`. For example, "


As discussed internally, not super sure about this. Pipes are designed for beginners so I wouldn't expect them to pass custom formatted prompts

It'd be nice to get more opinions here, maybe @molbap @Rocketknight1 ?

I understand, and yeah it's as a balance between convenience (for users) and maintenance (for us).

For TTS, we decided (for now) to not have audio inputs (e.g. for voice cloning) so that the __call__ method is much leaner:

transformers/src/transformers/pipelines/text_to_audio.py

Line 238 in 2e93004

def __call__(self, text_inputs, **forward_params):

and let model's chat template do it's work to properly extract the audio/text before calling the processor.

But we also never had audio as input so it wasn't a breaking change 🙏

If the chat template usage is well-documented, I guess it should still be fine for beginners? But I leave for you to decide, as you are way more knowledgeable about the image-text-to-text models and how the community prefers using them!

zucchini-nlp · 2025-12-02T13:51:01Z

tests/pipelines/test_pipelines_any_to_any.py

    @slow
    def test_small_model_pt_token_text_only(self):
        pipe = pipeline("any-to-any", model="google/gemma-3n-E4B-it")
-        text = "What is the capital of France? Assistant:"


we still need a test to check if the pipe can work in text-only mode imo. Users might want to pass only text sometimes and continue a multi-turn conversation

zucchini-nlp · 2025-12-02T13:52:12Z

tests/pipelines/test_pipelines_image_text_to_text.py

+                        "type": "input_image",
+                        "image_url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg",
                    },
-                    {"type": "text", "text": "Describe this image in one sentence."},
+                    {"type": "input_text", "text": "Describe this image in one sentence."},


hmm, right! W never got bug reports on this, and I just realized we don't promote openAI format in docs. Makes me wonder, should we keep supporting it ?

tests/pipelines/test_pipelines_image_text_to_text.py

ebezzam and others added 28 commits November 21, 2025 14:27

Fix processor usage and add chat_template support to TTS pipeline.

d4f89a0

Fallback to tokenizer for musicgen.

f4dbca1

Fallback to tokenizer for musicgen.

25f4816

Make style

29185d4

style/quality after update?

27a56cf

FIx copied from

a654122

Smaller things.

13234d9

Update src/transformers/pipelines/text_to_audio.py

729844b

Co-authored-by: Anton Vlasjuk <[email protected]>

Shift common utilities to chat template utils.

bb96d9b

Merge branch 'main' into fix/tts_pipepine

7da8240

Type nits

f801b96

Remove tied weights.

716c3cf

Keep seamless error

c0060d1

Better audio output object.

f8e90cf

Properly handle DIa and add test.

f5fb635

Shift chat template prep to base, test dia batch.

ed25458

Backward compatibility for dicts passed to pipelines

97158c6

Simplify postprocessing and tests.

5b80b03

Merge branch 'main' into fix/tts_pipepine

0b17759

Make quality/style

838daf7

Remove chat from image text to text.

e3b882d

Only check first item, to not consume first item of generator inputs.

661cafb

Nit

94410f4

Simplify

79d7e73

Add checks for bark/musicgen to ensure output is audio.

7a438ff

Better var

1428092

Merge branch 'main' into fix/tts_pipepine

4fde390

Rely on base class for chat casting.

11fd7ab

ebezzam marked this pull request as draft November 26, 2025 18:09

ebezzam commented Nov 26, 2025

View reviewed changes

ebezzam changed the title ~~Clean up image-text-to-text pipeline~~ 🚨 Clean up image-text-to-text pipeline Nov 26, 2025

ebezzam added 6 commits November 27, 2025 11:20

Merge branch 'main' into cleanup/image_text_text_pipeline

9ee8050

Fix test accordingly, and cleanup.

47dec31

Use chat template in test, and fix expected output.

eb4b0bd

Address any-to-any test.

2788165

Reintroduce OpenAI chat conversion, fix some expected outputs.

e513b53

FIx openai conversion for text generation.

86af935

ebezzam commented Nov 27, 2025

View reviewed changes

🚨 enforce chat template usage if it exists

fdc4db1

ebezzam marked this pull request as ready for review November 27, 2025 15:10

zucchini-nlp reviewed Dec 2, 2025

View reviewed changes

ebezzam added 2 commits December 4, 2025 10:22

Merge branch 'main' into cleanup/image_text_text_pipeline

ec5347f

Revert to onevision.

7ffcc35

		image = "./tests/fixtures/tests_samples/COCO/000000039769.png"
		text = "<image> What this is? Assistant: This is"

🚨 Clean up image-text-to-text pipeline #42430

Are you sure you want to change the base?

🚨 Clean up image-text-to-text pipeline #42430

Conversation

ebezzam commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

github-actions bot commented Nov 26, 2025

Uh oh!

ebezzam left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Nov 26, 2025

Uh oh!

ebezzam left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ebezzam Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam commented Nov 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

🚨 Clean up `image-text-to-text` pipeline #42430

🚨 Clean up `image-text-to-text` pipeline #42430

ebezzam commented Nov 26, 2025 •

edited

Loading

ebezzam left a comment •

edited

Loading

ebezzam Nov 26, 2025 •

edited

Loading

ebezzam left a comment •

edited

Loading

ebezzam Nov 27, 2025 •

edited

Loading