mxfp8 params without dp comm overlap will cause NaN

[kunlunl]

Describe the bug

using --fp8-param-gather and --reuse-grad-buf-for-mxfp8-param-ag without --overlap-grad-reduce and --overlap-param-gather will cause NaN loss.

If both --fp8-param-gather, --reuse-grad-buf-for-mxfp8-param-ag,--overlap-grad-reduce and --overlap-param-gather are enabled, it's fine.

Steps/Code to reproduce bug

Enable --fp8-recipe=mxfp8 --fp8-param-gather --reuse-grad-buf-for-mxfp8-param-ag.

Don't use --overlap-grad-reduce and --overlap-param-gather

Expected behavior

Converge correctly without dp comm overlap

Additional context

Created MRs to temporarily prevent using mxfp8 params without dp comm overlap

• [dev] Add assertion for mxfp8 params without dp overlap #2270
• Add assertion for mxfp8 params without dp overlap #2271

[BestJuly]

Add more info here: we've met this issue in DeepSeek-V3 and Qwen3-235B. However, when using a single-node Qwen3-like proxy, I do not meet this issue (at lease in few iterations, in comparison, NaN appears in the 2nd iteration of Qwen3-235B).