When bigdl-llm processes long prompts, the lead latency is long

[Storm0921]

Test environment: AMD7453 60GB mem Baichuan13B-chat bigdl llm20230903

Task: Knowledge enhancement QA, with a prompt length of 780 and a waiting time of approximately 100 seconds for program output, which is not as good as the bigdl llm (approximately 27 seconds) from an old version in July (20230706).

Using a very short prompt for QA only requires a delay time of about 5 seconds, which is not as good as the previous version's 2 seconds.

As a comparison, using the GPU version of Baihuan13B 8bit only requires a delay time of 0.2 seconds.

I'm surprised, why does a long prompt require such a long waiting time?

[hkvision]

Hi @fengenbao

We have recently added many Intel-specific optimizations for bigdl-llm. These updates haven't been tested on AMD platforms, and it is possible that the latest version may not run that fast or even get wrong results on AMD.
You are highly recommended to try bigdl-llm on Intel platforms to get optimal performance.
Feel free to tell us if you have further issues.