Qwen demo can't run successfully

[qzheng527]

The demo qwen couldn't run successfully. Error log as below.

Without bigdl optimization, the default demo code in Qwen can run successfully with the same model Qwen-7B-Chat.

[rnwang04]

Hi @qzheng527 , this error is caused by qwen modeling update, and we are fixing it. When it is solved, I will update here.

[rnwang04]

This error is fixed, could you please try it again with latest version tomorrow (so that new bigdl-llm package is released)?

[qzheng527]

Ok, I will give a try tomorrow.

[qzheng527]

@rnwang04 Now the simple demo below runs well.

python3 ./qwen/generate.py --repo-id-or-model-path /models/Qwen-7B-Chat

But if I specify the parameter --n-predict 512, it has werid output.

[rnwang04]

@qzheng527 , I have reproduced this issue with --n-predict 512.
But I found that even setting optimize_model=False or comment load_in_4bit (which means using original model), there is still werid output. Maybe it's a problem of model itself or it needs some prompt tuning.
In summary, I believe this is a problem inherent in the original model and is not related to the optimization of bigdl-llm.