Skip to content

[Issue]: enable NCCL_GRAPH_MIXING_SUPPORT will launch a large number of stream in cudagraph #1901

@MARD1NO

Description

@MARD1NO

How is this issue impacting you?

Lower performance than expected

Share Your Debug Logs

No response

Steps to Reproduce the Issue

No response

NCCL Version

2.28.3+cu129

Your platform details

I find when enable NCCL_GRAPH_MIXING_SUPPORT, it will launch multiple stream in cuda graph:

Image

when I turn off, it seems reasonable:

Image

Error Message & Behavior

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions