[GenAI 2025.3] Accuracy drop in chat mode for Llama-3.2-1B-Instruct (regression check for ID 167065, 168564, …)

[e950280]

---

Environment

• OS: Ubuntu 24.04
• OpenVINO GenAI: 2025.3
• CPU: Intel Core Ultra 5 125H

---

Background

In the 2025.2 release notes, the following known issue was documented:

Component: OpenVINO GenAI
ID: 167065, 168564, 168360, 168339, 168361
Description:
Models such as Qwen-7B-Chat, Phi4-Reasoning, Llama-3.2-1B-Instruct, Qwen3-8B, and DeepSeek-R1-Distill* show reduced accuracy in chat scenarios compared to regular generation requests. Currently no workaround is available; a fix is planned for future releases.

---

Current Observation

With OpenVINO GenAI 2025.3, I tested Llama-3.2-1B-Instruct in chat scenarios, and still observed noticeable accuracy degradation compared to:

• The same model used in regular generation
• Running via optimum.intel.openvino (OVModelForCausalLM)

This suggests the issue might still persist.

---

Question

• Has this issue already been fixed in 2025.3?
• If not yet fixed, is there an ETA for when the fix will be released?
• Until then, is using optimum.intel.openvino's OVModelForCausalLM the recommended workaround to avoid accuracy degradation?

---

[yatarkan]

Hi @e950280 ,

In 2025.3 reduced accuracy issue was fixed for models DeepSeek-R1-Distill*, Qwen3-0.6B, Qwen3-8B, Llama-3.2-1B-Instruct by switching to minja chat template engine.
Phi4-Reasoning accuracy issue is also fixed now and will be available in the upcoming 2025.4 release.

I have also tested Llama-3.2-1B-Instruct (FP16) via our whowhatbench tool and I see the same accuracy as in optimum-intel :

INFO:whowhatbench.wwb:Metrics for model: ./models/llama_3_2_1b_instruct_fp16
INFO:whowhatbench.wwb:   similarity
0         1.0

Could you please share which versions of openvino, openvino_tokenizers, openvino_genai, optimum-intel and transformers are you using?
And how exactly you are observing "noticeable accuracy degradation"? Do you have a script/reproducer, how do you run the generation in GenAI and how are you comparing the output?