Cray MPICH Issue - OFI poll failed

[saisandeepdammati]

Howdy!

I am running a job using a parallel adaptive mesh refinement based CFD code on a Cray machine, Carpenter at ERDC. The code is written in C++ and is compiled using cray-mpich (version 8.1.26).

The job is performed with 4224 mpi processes on 22 nodes with 192 cores per node. The job runs for sometime (15-20 minutes) and then crashes with signal 9 error with the following MPICH error (complete error file is attached as text file):

x1001c1s2b0n3.hsn.ex4000.erdc.hpc.mil: rank 2984 died from signal 9
MPICH ERROR [Rank 1024] [job id 3d8b8d04-c5cb-4e69-b621-75c176d4ce2c] [Wed Jul 10 11:12:37 2024] [x1001c0s3b0n2] - Abort(70375567) (rank 1024 in comm 0): Fatal error in PMPI_Send: Other MPI error, error stack:
PMPI_Send(163).................: MPI_Send(buf=0x25c1bbf0, count=65536, MPI_BYTE, dest=0, tag=12341, comm=0x84000003) failed
PMPI_Send(143).................: 
MPIR_Wait_impl(41).............: 
MPID_Progress_wait(201)........: 
MPIDI_Progress_test(97)........: 
MPIDI_OFI_handle_cq_error(1067): OFI poll failed (ofi_events.c:1069:MPIDI_OFI_handle_cq_error:Input/output error - PTLTE_NOT_FOUND)

I have looked at all the MPI_Send commands in the code and they look sensible, however, the runs crash with this error. Is this a familiar issue? Can you please provide me with a workaround or a fix for it?

Thanks in advance.

cray_mpich_error_carpenter.txt

[hzhou]

Is this with HPE slingshot network?

[saisandeepdammati]

Hi,

Yes the machine uses HPE Slingshot network.

[saisandeepdammati]

Hi,

Yes the machine uses HPE Slingshot network.