Can you package your dependency package?

[zhangwzhang]

There have been many incompatibility issues. My CUDA version is 12.1. Follow your steps to report various errors.

[hubertyoo]

Hi, zhang, what is your GPU and could you share more details about the error? It should not have any issue with GPU that has computation power greater than 7. If your GPU's computation power is lower than 7, you may need to manually compile the xformers or vllm. Could you try skipping the vllm installation and use the huggingface inference and see if there are any issues running the codes?