[Question]: why PXN graph is prefered than crossNic ?

[fishautumn]

Question

when NIC bandwidth is low, the channel search algorithm will end with PXN channels. so ring alteration wron't execute.

I prefer ring alteration for rail optimization. Also compared with PXN, crossNic channels has less hops.

after some investigation, I found ncclTopoCompareGraphs() doesn't check hop count when graphs have different crossNic value:

nccl/src/graph/search.cc

Line 436 in b17addf

if (graph->pattern == refGraph->pattern && graph->crossNic == refGraph->crossNic && graph->nHops < refGraph->nHops) *copy = 1;

as a result, it prefers PXN graph than crossNic graph here.

so the questions are:

1. why PXN is preferred than crossNic ?
2. should we prefer less hops here, especially for ring pattern ?

[sjeaugey]

Alternate rings have some limitations, like they don't work well with odd number of nodes. But you're right to point out that PXN adds an extra step, hence a bit of latency.

Are you seeing a noticeable latency difference between PXN and alternate rings?

[fishautumn]

Thanks a lot for your quick answer. I didn't see noticeable latency difference.

about odd node count ring alter, I guess it will only cross rail between first and last node. it still seems better than full PXN.

[fishautumn]

... I didn't see noticeable latency difference.

my previous comment was wrong: after testing, I found providing correct NIC bandwidth (which enable ringl alteration) brings in average 2.5% (0.5% ~ 5%) improvement in latency. @sjeaugey