padding='max_length'

[Hummer12007]

Can I specify padding=max_length for an embedding model?

Trying to run https://huggingface.co/Marqo/marqo-fashionSigLIP/ and with default padding the embedding values are different and less useful.

Aha, yeah, huggingface/transformers#28569 <- seems like all the siglip models expect padding to be set to max_length

[michaelfeil]

max_length padding is default, what are you expecting?

[Hummer12007]

Hi. True padding is the default (and hardcoded across the infinity codebase), which is distinct from max_length. Citing huggingface docs:

The padding argument controls padding. It can be a boolean or a string:

True or 'longest': pad to the longest sequence in the batch (no padding is applied if you only provide a single sequence).
'max_length': pad to a length specified by the max_length argument or the maximum length accepted by the model if no max_length is provided (max_length=None). Padding will still be applied if you only provide a single sequence.
False or 'do_not_pad': no padding is applied. This is the default behavior.

[Hummer12007]

Code snippet:

from transformers import AutoModel, AutoProcessor
model = AutoModel.from_pretrained('Marqo/marqo-fashionSigLIP', trust_remote_code=True)
processor = AutoProcessor.from_pretrained('Marqo/marqo-fashionSigLIP', trust_remote_code=True)

import torch
from PIL import Image

# 64 tokens
kan = "ASLDKl;sdalkjkdjasljljdlkajswou3908uoidjkajwkjdklsanmdwbnjkehqu8934082uj3knwemdsj"
text = ["black t-shirt", kan]

processed_true = processor(text=text, padding=True, return_tensors="pt")
processed_true_nopad = processor(text=text[:1], padding=True, return_tensors="pt")
processed_max = processor(text=text[:1], padding="max_length", return_tensors="pt")

with torch.no_grad():
    text_features = model.get_text_features(processed_true['input_ids'], normalize=True)
    text_features_nopad = model.get_text_features(processed_true_nopad['input_ids'], normalize=True)
    text_features_max = model.get_text_features(processed_max['input_ids'], normalize=True)


print("text norm/b")
print(text_features[0][:10]) # <- hacky workaround that makes padding correct
print(text_features_nopad[0][:10]) # <- infinity returnes this one
print(text_features_max[0][:10]) # <- the correct one, same as first

[Hummer12007]

bump @michaelfeil added relevant context for why this is actually a problem in my use case