System instruction not work

[shijiecao-01]

I am using lmms-eval to evaluate Qwen2.5-VL-7B on refCOCO dataset.

I want to change the system prompt by --system_instruction command, but it didn't work. I checked the DEBUG output, the system prompt is always 'You are a helpful assistant.'

Has anyone encountered the same question? How to solve this?

[ngquangtrung57]

Can you try adding --apply_chat_template to your command, for example:

lmms_eval --model hf \
  --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct \
  --tasks refcoco_bbox_test \
  --batch_size 1 \
  --system_instruction "Your custom instruction here" \
  --apply_chat_template

[shijiecao-01]

Can you try adding --apply_chat_template to your command, for example:

lmms_eval --model hf \
  --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct \
  --tasks refcoco_bbox_test \
  --batch_size 1 \
  --system_instruction "Your custom instruction here" \
  --apply_chat_template

Hi @ngquangtrung57. I added --apply_chat_template:

accelerate launch --num_processes=2 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=xxx \
    --tasks refcocog_bbox_rec_test \
    --batch_size 1 \
    --log_samples \
    --system_instruction "test" \
    --apply_chat_template \
    --limit=10 \
    --verbosity=DEBUG \
    --output_path ./results 

But it didn't work:

Exploded dataset from 5023 to 9602 rows
[rank1]: Traceback (most recent call last):
[rank1]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank1]:   File "<frozen runpy>", line 88, in _run_code
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 539, in <module>
[rank1]:     cli_evaluate()
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 366, in cli_evaluate
[rank1]:     raise e
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 347, in cli_evaluate
[rank1]:     results, samples = cli_evaluate_single(args)
[rank1]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 474, in cli_evaluate_single
[rank1]:     results = evaluator.simple_evaluate(
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/utils.py", line 533, in _wrapper
[rank1]:     return fn(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/evaluator.py", line 259, in simple_evaluate
[rank1]:     chat_template=lm.chat_template if apply_chat_template else None,
[rank1]:                   ^^^^^^^^^^^^^^^^
[rank1]: AttributeError: 'Qwen2_5_VL' object has no attribute 'chat_template'

[ngquangtrung57]

Can you try this one instead?

lmms-eval \
  --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,system_prompt='Your custom system instruction here' \
  --tasks refcocog_bbox_rec_test \
  --batch_size 1 \
  --log_samples \
  --output_path ./results

[shijiecao-01]

Still not working.

accelerate launch --num_processes=2 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=/Qwen/Qwen2.5-VL-7B-Instruct,system_prompt="test" \
    --tasks refcocog_bbox_rec_test \
    --batch_size 1 \
    --limit=10 \
    --log_samples \
    --verbosity=DEBUG \
    --output_path ./results 

DEBUG output:

Model Responding:  60%|███████████████████████████████████████████████████▌                                  | 3/5 [00:16<00:10,  5.21s/it]2025-09-08 07:43:28 | DEBUG    | lmms_eval.models.chat.qwen2_5_vl:generate_until:132 - Question: <|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>Please locate the region this sentence describes in the image using bounding box with format [num, num, num, num], you should think first before answer the question. Stick to the format: <think>your think process<   hink><answer>[num, num, num, num]<nswer>: There is red colored truck in between the other trucks<|im_end|>
<|im_start|>assistant