Numerical Discrepancies Between CUDA-Aware MPI and Standard MPI on Multi-GPU Runs

[ia267]

We're observing numerical discrepancies when running PyFR with CUDA-aware MPI compared to standard MPI. These differences persist across different machines and configurations, raising concerns about the consistency of numerical results when using CUDA-aware MP

Expected Behavior:

Numerical results should be consistent between CUDA-aware MPI and standard MPI, with differences limited to floating-point round-off error (e.g., last few digits).

Actual Behavior:

• Differences are observed beyond typical floating-point round-off tolerance.
• Discrepancies are more pronounced in multi-GPU runs.
• The differences persist across multiple machines with different hardware configurations.

Suggested Next Steps:

• Investigate whether different code paths are taken in PyFR when using CUDA-aware MPI vs. standard MPI.
• Check if asynchronous communication or stream handling differs between the two MPI modes.
• Confirm if the issue is related to order of operations or numerical reductions performed across GPUs.
• Provide guidance on recommended MPI configurations for reproducible numerical results with CUDA.

[ia267]

Need to run the following:

• both versions with aliasing on single core - result: no difference
• both versions without aliasing on single core - result: differences
• both versions with aliasing on 2 cores - result: cuda-aware one doesn't run
• standard MPI with aliasing on multiple cores (2 runs) - no difference
• standard MPI without aliasing on multiple cores (2 runs) - differences

[ia267]

With anti-aliasing switched on (uncommented lines necessary for anti-aliasing):
No difference between CUDA-aware and standard MPI (on a single core)
No difference in multiple runs for both CUDA-aware and standard MPI (on a single core)
No difference in multiple runs between standard MPI (GPU oversubscribed). I think this contradicts with your results.

Without anti-aliasing:
Differences between CUDA-aware and standard MPI (on a single core)
No difference in multiple runs for standard MPI (on a single core)
Differences in multiple runs for CUDA-aware MPI (on a single core)
Differences in multiple runs between standard MPI (GPU oversubscribed). I could not get the same results even when using self.backend.wait()

Few issues:
CUDA-aware MPI runs don't work with oversubscribed GPUs on my machine (I'm getting a segmentation fault). Is this a known issue?
Setting self.backend.wait() in rhs function pyfr/solvers/base/system.py doesn't seem to resolve the differences for me