Utilizing SINQ for non-LLM architectures

[ogencoglu]

Thanks for the interesting work. What would be needed to use SINQ in general deep neural net setups, not only LLMs, for example CNNs ?

[wsbagnsv1]

Yeah i have a similar question, i tried to use it with vibevoice (which is based on a llm) but got "Quantization failed: block must divide W. Using original weights." errors, and even with those tensors skipped i dont get everything, could be a config issue though since i might be setting up the device map wrong.
Device map used: full model -> cuda
Starting SINQ quantization (incompatible weights will be saved as full precision)...
Default model structure not supported. Make sure you feed device as dictionary as {name_block: device}

[philippebich]

We are working on CNN support (also for non LLM models) that should be available (hopefully) next week 🥳

[ogencoglu]

We are working on CNN support (also for non LLM models) that should be available (hopefully) next week 🥳

Fantastic. Looking forward to it. Pure torch or onnx support would be amazing without the bloat called HuggingFace.

[tllewellynn1]

I would be interested to test out CNN model support and provide feedback as TensorRT does not support NVFP4 for convolutions.
NVIDIA/Model-Optimizer#362 (comment)

As well as visual language models : https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking

[philippebich]

I haven’t forgotten about this, I just need a bit more time to make sure everything works correctly. I’m currently working on it.

[wsbagnsv1]

You are a legend, a quantization framework that works well without first defining everything would be amazing (;
Can you while you are at it, make sure that 5d tensors and similar wont be quantized, because when i tried that it gave errors, but we can basically just skip those anyways (;

[philippebich]

@wsbagnsv1 which model did you try? If it is a public one I can consider testing also on that model before pushing the code

[wsbagnsv1]

@philippebich

I tried with multiple ones, vibevoice1.5b and ovis2.5 2b primarily (;

[wsbagnsv1]

okay I and claude code went over it with glm4.6 and I think ive managed to get it working for vibevoice and similar models at least the quantizing part seems to work fine and keep the tensors with 5d in high precision, though i didnt have time to check inference yet, i hope i can do that today. If it works ill share the code (;
(tried to also implement a configuration method so you can keep specifically names tensors in high precision, which can help some models im sure (; )

[wsbagnsv1]

okay i found an issue with the inference part, since it initiates normally with
model = AutoModel.from_pretrained(save_dir)
for
from_quantized_safetensors
vibevoice doesnt work since it needs to be initiated with
model = VibeVoiceForConditionalGenerationInference(config)
Im trying to get it working with such custom loaders now (;

[wsbagnsv1]

Well vibevoice has some weird loading mechanism so ill just switch to qwen3 vl 2b now that should work relatively easy with that custom loader patch since it has a standard config and tokenizer

[wsbagnsv1]

okay the inference part im struggling with, its really really slow but it works (;

Loading quantized model from G:/testingRing/Qwen3-VL-4B-Instruct-4bit-sinq...
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 266/266 [00:00<00:00, 106397.56it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 357/357 [00:00<00:00, 37577.90it/s]
preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████| 390/390 [00:00<?, ?B/s]
chat_template.json: 5.50kB [00:00, 11.0MB/s]
tokenizer_config.json: 10.9kB [00:00, 21.8MB/s]
vocab.json: 2.78MB [00:00, 29.7MB/s]
merges.txt: 1.67MB [00:00, 35.2MB/s]
tokenizer.json: 7.03MB [00:00, 51.9MB/s]
video_preprocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████| 385/385 [00:00<00:00, 771kB/s]
Model and processor loaded successfully.
Running inference...
Setting pad_token_id to eos_token_id:151645 for open-end generation.
Inference complete.
Output: This is a heartwarming, sun-drenched photograph capturing a tender moment between a woman and her dog on a beach at sunset.

Key Elements:

• The Subjects: A woman with long, dark hair, wearing a plaid shirt and dark pants, sits on the sand. Beside her, a large, light-colored Labrador Retriever sits attentively, wearing a harness with a colorful pattern.
• The Action: The dog is giving the woman a high-five with its paw. The woman is smiling broadly, looking at the dog with joy and affection, and is holding out her hand to meet the

So it can succesfully quantize multimodal llms with my patch (;

[wsbagnsv1]

This one takes ages though
Loading quantized model from G:/testingRing/Qwen3-VL-4B-Instruct-4bit-sinq...

[wsbagnsv1]

This is the demo picture btw so it seems inference works fine with the quantized weights and the 4bit is like 3.6gb and the original is 9gb (i think some parts are not quantized gonna look into that later)