prov/lnx: Device to Device cudaMemcpy not synchronized

[angainor]

Describe the bug
cudaMemcpy operations between devices are not synchronized when using the lnx provider. This results in ongoing transfers reported to the host as completed before they actually complete. In particular, osu_bibw with validation -c often fails, or reports unreasonably high bandwidth. This is related to results discussed with @amirshehataornl in open-mpi/ompi#13156

To Reproduce
Compiled libfabric with lnx and cuda support, then run osu_bibw -c D D:

mpirun -np 2 -x FI_SHM_USE_XPMEM=1 -x FI_HMEM_CUDA_USE_GDRCOPY=1 -x FI_LNX_PROV_LINKS="shm" -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "lnx" -map-by numa -prtemca ras_base_launch_orted_on_hn 1 -mca mtl_ofi_av table ~/gpubind_pmix.sh ./osu_bibw -c D D
[x1000c1s0b0n0:668142] SET FI_SHM_USE_XPMEM=1
[x1000c1s0b0n0:668142] SET FI_HMEM_CUDA_USE_GDRCOPY=1
[x1000c1s0b0n0:668142] SET FI_LNX_PROV_LINKS=shm
rank 0 local 0 gpu 0
rank 1 local 1 gpu 1

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
1                       0.58              Pass
2                       1.18              Fail

The above Fails may be hard to reproduce, depending on the system. In case when there is no failure the reported bandwidths for large messages are too high (max 200GB/s for the GH200 system tested here):

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
1                       0.58              Pass
2                       1.18              Pass
4                       2.35              Pass
8                       4.70              Pass
16                      9.41              Pass
32                     18.73              Pass
64                     37.69              Pass
128                    75.20              Pass
256                   150.05              Pass
512                   297.85              Pass
1024                  600.33              Pass
2048                 1193.13              Pass
4096                 2384.77              Pass
8192                 4768.02              Pass
16384                9550.22              Pass
32768               19117.57              Pass
65536               38159.45              Pass
131072              77169.28              Pass
262144             157666.12              Pass
524288             318422.65              Pass
1048576            638673.26              Pass
2097152           1281063.50              Pass
4194304           2554071.38              Pass

Expected behavior
After adding calls to cuda_set_sync_memops i n lnx_trecv and lnx_tsenddata (see patch) the results validate and the bandwidth for large messages is more reasonable:

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
1                       0.22              Pass
2                       0.43              Pass
4                       0.87              Pass
8                       1.73              Pass
16                      3.45              Pass
32                      6.90              Pass
64                     13.84              Pass
128                    27.57              Pass
256                    54.94              Pass
512                   110.11              Pass
1024                  219.93              Pass
2048                  437.83              Pass
4096                  871.59              Pass
8192                 1717.63              Pass
16384                3415.92              Pass
32768                6793.29              Pass
65536               13068.11              Pass
131072              24836.90              Pass
262144              45147.18              Pass
524288              75769.13              Pass
1048576            114856.73              Pass
2097152            155360.10              Pass
4194304            191511.74              Pass

The attached patch is most likely wrong/incomplete in general, but does demonstrate the point for Device - Device transfers with Cuda.

diff --git a/prov/lnx/src/lnx_ops.c b/prov/lnx/src/lnx_ops.c
index 41879b086..b6120bf38 100644
--- a/prov/lnx/src/lnx_ops.c
+++ b/prov/lnx/src/lnx_ops.c
@@ -453,6 +453,7 @@ ssize_t lnx_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
        struct lnx_ep *lep;
        const struct iovec iov = {.iov_base = buf, .iov_len = len};
 
+       cuda_set_sync_memops(buf);
        lep = container_of(ep, struct lnx_ep, le_ep.ep_fid.fid);
        if (!lep)
                return -FI_ENOSYS;
@@ -664,6 +665,7 @@ ssize_t lnx_tsenddata(struct fid_ep *ep, const void *buf, size_t len, void *desc
        fi_addr_t core_addr;
        void *core_desc = desc;
 
+       cuda_set_sync_memops(buf);
        lep = container_of(ep, struct lnx_ep, le_ep.ep_fid.fid);
        if (!lep)
                return -FI_ENOSYS;

[msimberg]

Are there any updates on this issue? Do you think the above patch could work in main until a more thorough patch is available? Or in other words: how "wrong/incomplete" do you think the patch is?

[angainor]

@msimberg I'd also like to know :) On our system we do not use it (yet) - I did not test it beyond what's described here. My libfabric code base knowledge is essentially NULL, so I imagine this patch could be very incomplete. I don't think it can cause any data correctness errors though - after all it's just a sync.

Let me know if you have any further info about this - we're also eagerly waiting for any developments.

[msimberg]

Thanks @angainor, even that's useful to know! In that case we'll take this patch as is for now and try it a bit more at CSCS.

[amirshehataornl]

We are attempting to reproduce the issue locally here and we'll get back on this ticket as soon as we have an update.

[aingerson]

@amirshehataornl This looks like more of a shm issue rather than just lnx. I'm going to try to reproduce as well.

[amirshehataornl]

great. We were going to try with just lnx+shm. And if we're able to reproduce we were going to modify openMPI so it can select shm directly (currently MTL doesn't) and try that.

[angainor]

great. We were going to try with just lnx+shm. And if we're able to reproduce we were going to modify openMPI so it can select shm directly (currently MTL doesn't) and try that.

If I remember correctly, I did try that also. In ompi/mca/mtl/ofi/mtl_ofi_component.c:ompi_mtl_ofi_component_init you have to remove FI_REMOTE_COMM from hints->caps |= FI_MSG | FI_TAGGED | FI_LOCAL_COMM | FI_REMOTE_COMM | FI_DIRECTED_RECV;, otherwise the lnx+shm transport is not accepted. I think it was working after the change.

But I think I was unable to use just shm provider with OpenMPI. That's why my patch-work only addressed the lnx part.

[angainor]

@amirshehataornl actually, my memory failed me. Removal of FI_REMOTE_COMM from caps makes it possible to run OpenMPI with shm provider:

mpirun -np 2 -x FI_PROVIDER=shm -x FI_SHM_USE_XPMEM=1 -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "shm" -mca mtl_ofi_av table -prtemca ras_base_launch_orted_on_hn 1 ./app

while it seems no change is needed to run lnx+shm:

mpirun -np 2 -x FI_PROVIDER=lnx -x FI_SHM_USE_XPMEM=1 -x FI_LNX_PROV_LINKS="shm" -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "lnx" -mca mtl_ofi_av table -prtemca ras_base_launch_orted_on_hn 1 ./app

[amirshehataornl]

@angainor, when you tried shm directly from openMPI did you see the problem? In your previous response it didn't seem that you were 100% certain. Is it easy for you re-verify?

[zachdworkin]

@angainor can you provide your gpu binding from the pmix script youre running? I am trying to recreate this on my machine and want to get as close as possible to your failing case

[zachdworkin]

@angainor I have completed 100 runs using modified ompi to run shm only. None of my runs displayed any failure or the issue that you are seeing. Can you provide more information about the system that you are running on?

I am using
Intel Icelake CPU
Single NVIDIA A100 card
Cuda version12.6, V12.6.85
Driver Build cuda_12.6.r12.6/compiler.35059454_0

[angainor]

I am running my tests on a HPE ex254n blade with 4 GH200 cards and an ARM cpu. This is a production system now, previous tests were ran on a test HPE blancapeak system (@mflough-at-hpe) with same hardware.

For today's results I used OpenMPI 5.0.8 with FI_REMOTE_COMM removed, as discussed above, and libfabric master:f08ec6a6c549c83de576bfcf85b92189364d38d7. I ran 3 tests, all using osu_bibw:

• provider: lnx+shm, osu_bibw D D (no verification) - test runs, but bandwidth is unreasonably high
• provider: lnx+shm, osu_bibw -c D D (with verification) - test fails verification most times. If it runs, the bandwidth is even more unreasonably high.
• provider: shm, osu_bibw -c D D (with verification) - test segfaults for buffers > 4096 bytes

Here are the exact outputs:
Teset 1

mpirun -np 2 -x FI_SHM_USE_XPMEM=1 -x FI_HMEM_CUDA_USE_GDRCOPY=1 -x FI_LNX_PROV_LINKS="shm" -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "lnx" -map-by numa -prtemca ras_base_launch_orted_on_hn 1 -mca mtl_ofi_av table ~/gpubind.sh ./osu_bibw D D

[gpu-1-57:3406118] SET FI_SHM_USE_XPMEM=1
[gpu-1-57:3406118] SET FI_HMEM_CUDA_USE_GDRCOPY=1
[gpu-1-57:3406118] SET FI_LNX_PROV_LINKS=shm
rank 1 local 1 gpu 1
rank 0 local 0 gpu 0

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
1                       0.53
2                       1.06
4                       2.13
8                       4.27
16                      8.53
32                     17.06
64                     34.11
128                    68.23
256                   136.50
512                   272.94
1024                  546.15
2048                 1090.42
4096                 2181.70
8192                 4379.22
16384                8744.32
32768               17519.04
65536               35147.29
131072              70572.27
262144             144109.43
524288             290514.56
1048576            325891.30
2097152            318890.69
4194304            349287.10

The maximum bandwidth should be around 200GB/s, so this is too much. This is already an indication that something is wrong.

Teset 2

mpirun -np 2 -x FI_SHM_USE_XPMEM=1 -x FI_HMEM_CUDA_USE_GDRCOPY=1 -x FI_LNX_PROV_LINKS="shm" -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "lnx" -map-by numa -prtemca ras_base_launch_orted_on_hn 1 -mca mtl_ofi_av table ~/gpubind.sh ./osu_bibw -c D D

[gpu-1-57:3407193] SET FI_SHM_USE_XPMEM=1
[gpu-1-57:3407193] SET FI_HMEM_CUDA_USE_GDRCOPY=1
[gpu-1-57:3407193] SET FI_LNX_PROV_LINKS=shm
rank 1 local 1 gpu 1
rank 0 local 0 gpu 0

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
1                       0.53              Pass
2                       1.07              Pass
4                       2.13              Pass
8                       4.25              Pass
16                      8.52              Pass
32                     17.04              Pass
64                     34.03              Pass
128                    68.13              Pass
256                   136.32              Pass
512                   271.92              Pass
1024                  545.22              Pass
2048                 1089.46              Fail
DATA VALIDATION ERROR: ./osu_bibw exited with status 1 on message size 2048.

Here the tests fails at random points, indicating some race. I believe it's because - as described before - completion events arrive before the actual data transfer is completed. Sometimes the test runs through, but the bandwidth obtained in this case is completely off:

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
1                       0.54              Pass
2                       1.08              Pass
4                       2.15              Pass
8                       4.30              Pass
16                      8.59              Pass
32                     17.19              Pass
64                     34.41              Pass
128                    68.87              Pass
256                   137.53              Pass
512                   275.34              Pass
1024                  549.72              Pass
2048                 1097.16              Pass
4096                 2194.60              Pass
8192                 4389.13              Pass
16384                8739.25              Pass
32768               17576.37              Pass
65536               35153.43              Pass
131072              70564.64              Pass
262144             144519.57              Pass
524288             292389.72              Pass
1048576            576572.30              Pass
2097152           1177703.35              Pass
4194304           2355803.07              Pass

Teset 3

mpirun -np 2 -x FI_SHM_USE_XPMEM=1 -x FI_HMEM_CUDA_USE_GDRCOPY=1 -x FI_PROVIDER="shm" -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "shm" -map-by numa -prtemca ras_base_launch_orted_on_hn 1 -mca mtl_ofi_av table ~/gpubind.sh ./osu_bibw D D

[gpu-1-57:3408047] SET FI_SHM_USE_XPMEM=1
[gpu-1-57:3408047] SET FI_HMEM_CUDA_USE_GDRCOPY=1
[gpu-1-57:3408047] SET FI_PROVIDER=shm
rank 1 local 1 gpu 1
rank 0 local 0 gpu 0

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
1                       0.11
2                       0.23
4                       0.46
8                       0.91
16                      1.83
32                      3.66
64                      7.26
128                    14.62
256                    28.50
512                    56.51
1024                  112.26
2048                  216.34
4096                  421.78
[gpu-1-57:3408878] *** Process received signal ***
[gpu-1-57:3408878] Signal: Segmentation fault (11)
[gpu-1-57:3408878] Signal code: Invalid permissions (2)
[gpu-1-57:3408878] Failing at address: 0x40051a400000
[gpu-1-57:3408878] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x4002a7ff07c0]
[gpu-1-57:3408878] [ 1] /lib64/libc.so.6(+0xa2c60)[0x4002ab7b2c60]
[gpu-1-57:3408878] [ 2] /cluster/projects/nn9999k/marcink/software/libfabric-master-gpu/lib/libfabric.so.1(+0x32030)[0x4002abad2030]
[gpu-1-57:3408878] *** End of error message ***

This tests, using shm provider, always segfaults for messages > 4096 bytes when using xpmem. When I turn xpmem off, the test runs with quite bad performance (explicit cudaCopy?):

mpirun -np 2 -x FI_SHM_USE_XPMEM=0 -x FI_HMEM_CUDA_USE_GDRCOPY=1 -x FI_PROVIDER="shm" -mca pml cm -mca mtl ofi --mca opal_common_ofi_provider_include "shm" -map-by numa -prtemca ras_base_launch_orted_on_hn 1 -mca mtl_ofi_av table ~/gpubind.sh ./osu_bibw D D

[gpu-1-59:3251532] SET FI_SHM_USE_XPMEM=0
[gpu-1-59:3251532] SET FI_HMEM_CUDA_USE_GDRCOPY=1
[gpu-1-59:3251532] SET FI_PROVIDER=shm
rank 1 local 1 gpu 1
rank 0 local 0 gpu 0

# OSU MPI-CUDA Bi-Directional Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
1                       0.11
2                       0.23
4                       0.46
8                       0.92
16                      1.84
32                      3.67
64                      7.33
128                    14.62
256                    28.35
512                    55.71
1024                  111.23
2048                  213.41
4096                  424.56
8192                  741.52
16384                1419.19
32768                2964.51
65536                3321.38
131072               3536.58
262144               3659.12
524288               3736.94
1048576              3760.48
2097152              3747.38
4194304              3757.41

Binding
The binding script is trivial really - bind rank 0 to GPU 0, rank 1 to GPU 1:

#!/bin/bash

if [ -z "$OMPI_COMM_WORLD_LOCAL_RANK" ]; then
    local=${SLURM_LOCALID}
    rank=${PMIX_RANK}
else
    local=${OMPI_COMM_WORLD_LOCAL_RANK}
    rank=${OMPI_COMM_WORLD_RANK}
fi
gpu=$((0+${local}))

echo rank $rank local $local gpu $gpu
export CUDA_VISIBLE_DEVICES=$gpu

"$@"