[Performance]: [AutoDeploy] Benchmark and analyze AD-vLLM perf gap for Nemotron MoE FP8 tp=1

[galagam]

Proposal to improve performance

Nemotron MoE FP8 tp=1. Compare to vLLM perf on H100/B200.
Sweep over max concurrency and prepare output tok/s vs tok/user/s pareto curves.
Dump traces for both vLLM and AD.
Analyze traces and identify possible performance optimizations for AD.

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[galagam]

Curves in Nemotron nano v3 perf - ad vs vllm .

Initial vLLM measurements were done https://github.com/amirkl94/vllm/tree/round-1-optimizations
It was integrated to vLLM main branch along with additional optimizations. The new vLLM baseline is main ToT (251118, commit #5bb1da5). Precompiled in bvllm nightly container:
docker pull vllm/vllm-openai:nightly-5bb1da5190b54aefb08478c6b1170f97722b8bdb

Possible optimizations identified based on round-1-optimizations + max concurrency 1:

1. multi-stream MoE - major
2. Integrate FlashAttention as attention backend - minor

Need to analyze main tot vLLM trace + additional concurrencies to identify additional opportunities.