Adapting Vidur to vLLM and Profiling CPU Overhead

[duanzhaol]

Hi team,

I’m working on adapting Vidur, the LLM inference system simulator, to vLLM. Currently, Vidur’s profiling is based on Sarathi-Serve, but I’d like to explore how to make it compatible with vLLM. From my understanding, simply setting vLLM’s scheduler may not accurately simulate the performance of the latest version of vLLM. Could you provide guidance on how to properly adapt Vidur to vLLM?

Additionally, I noticed that the documentation mentions profiling CPU overhead, but the linked file for this section seems to be missing. Could you clarify how to profile CPU overhead in Vidur or point me to the correct resource?

Any insights or suggestions would be greatly appreciated!

Thanks,
Zhaolin Duan

[RKai025]

Hi team,

I’m working on adapting Vidur, the LLM inference system simulator, to vLLM. Currently, Vidur’s profiling is based on Sarathi-Serve, but I’d like to explore how to make it compatible with vLLM. From my understanding, simply setting vLLM’s scheduler may not accurately simulate the performance of the latest version of vLLM. Could you provide guidance on how to properly adapt Vidur to vLLM?

Additionally, I noticed that the documentation mentions profiling CPU overhead, but the linked file for this section seems to be missing. Could you clarify how to profile CPU overhead in Vidur or point me to the correct resource?

Any insights or suggestions would be greatly appreciated!

Thanks, Zhaolin Duan

Same issue, did you solve it?

[manav-aggarwal4]

Me too!

[AgrawalAmey]

Hello @duanzhaol @PhantomKid @manav-aggarwal4 -- We don't plan to officially support vLLM v1 in Vidur. There is some code in the canary branch that you can check out, @nitinkedia7 can share more details about the supported features and limitations. We are working on a new runtime modeling system that would support both vLLM and SGLang.

[zhuango]

Hi @AgrawalAmey, the new runtime modeling system is "REVATI: Transparent GPU-Free Time-Warp Emulation for LLM Serving"? That is an excellent work! Would you mind sharing the timeline of making REVATI opensource?

[AgrawalAmey]

Hi @zhuango, yes, this is the new framework that I had mentioned. It is currently under peer review. We are still figuring out the open sourcing strategy, but we might do that only after the paper is through the review process.