Interactive node connection closed by remote host

[maituoy]

If you notice an issue on arjuna, please first search the existing issues and ensure that it has not been reported. If you notice a similar example, please comment on that issue.

Please update the issue's title with something informative

Please provide the following information to help us help you:

Basic Info

Your Name: Maituo Yu
Your Andrew ID: maituoy

Where it happened

Job Ids: 8427
Node(s) on which the problem occurred: e001 (But on other nodes as well)

What Happened

Expected Behavior: The node will keep connected when running a interactive node
Observed Behavior: The connection to the granted node was closed by remote host after a certain amount of time.

Log Files

Location of Log file Showing the Error: No log file
Location of Script showing Minimum Working Example: /home/maituoy/software/interdispipe/example/Ni2MnIn/HSE/band

Please also attach any logs and the submission script to this issue.

Submit log files like this (Notice the ``` before and after this block)
this ensures logs are formated correctly (Ie. Comments don't become headings)

salloc -N 1 -n 54 -p cpu --mem-per-cpu=0 --time=7-00:00:00

What I've Tried

Please List what you've tried to debug the issue. Please include commands and resulting output

1. I googled but nothing have been found similar so far.

> awesome_command
Output from awesome_command

If you do not have any of the above, please explain why you do not have it, and submit the issue, however, the more information you give us, the better we can help you.

Screenshot showing the disconnection.

[awadell1]

My guess is you're doing something like:

salloc ....
ssh e001

The problem is ssh e001 is outside of slurm's control, making it an unauthorized process, and thus it gets terminated. This is a new feature to ensure jobs get the resources requested and automatically clean up wayward jobs. The check happens on an hourly basis which is why your terminal doesn't get killed right away.

I would recommend switching to something like: srun -N 1 -n 54 -p cpu --pty bash.

Looking at e001's logs I'm guessing the disconnect happened around 20210925 17:39:27. Can you confirm?

[maituoy]

Yeah, that's exactly what I was doing. I'm trying srun -N 1 -n 54 -p cpu --pty bash. But I'm not able to run executables with mpirun. It stuck at some point and nothing was output. I do it in mpirun -np 54 vasp_ncl > vasp.out. I check the mpirun module and I think it's well loaded. The single core calculation is not affected.

[awadell1]

Maybe try: #59 (comment)

[maituoy]

You mean run like srun -n 54 vasp_ncl > vasp.out?

[awadell1]

Roughly. I think he was running in an sbatch allocation so maybe not exactly the interactive environment you want. And idk how srun within srun behaves.

[maituoy]

No, it does not work. Job is terminated after a while.

[awadell1]

Did you also install vasp linked to openmpi schedulers=slurm pmi=true? But I would try reducing your MWE and trying to isolate the problem. My openmpi / vasp knowledge is pretty limited geneally.

Definitely don't try to start an interactive job with salloc and then sshing in, slurm will kill your processes. If you do find a solid way to do an interactive mpi job, we'd love PR to update the docs (#4).

[maituoy]

I did not use the Spack to install VASP since it was not able to find the local package mirror.

[maituoy]

I'm not sure if this is due to the Slurm version since I see a similar issue (mpirun hang) posted in the open-mpi git repo. I have checked that we are currently using the slurm 20.11.8 and the latest version is slurm 21.08.0. Is it possible to upgrade the slurm without affecting others' running job?

[awadell1]

No and slurm will not be updated any time soon. Slurm 21.08.0 was released on August 26th right at the tail end of the upgrade. Previously we were running a roughly 5-year-old version.

The linked issue suggests that 21.11.03 has the fixed so I don't think that's your issue
open-mpi/ompi#8378 (comment)

[maituoy]

Did you also install vasp linked to openmpi schedulers=slurm pmi=true? But I would try reducing your MWE and trying to isolate the problem. My openmpi / vasp knowledge is pretty limited geneally.

Definitely don't try to start an interactive job with salloc and then sshing in, slurm will kill your processes. If you do find a solid way to do an interactive mpi job, we'd love PR to update the docs (#4).

It does not have to be the VASP, since the same issue exists for other software as well like Quantum Espresso. I think it's more like a system issue, not the software.

[awadell1]

We have encountered some OpenMPI growing pains and would appreciate a minimal working example of this issue. The system OpenMPI is not Slurm aware, and that does seem to be causing problems.

Again have you tried following #59 and installing a slurm-aware openmpi? There have been reports of this working for py-gpaw.

To clarify, if you ssh into a node that you have an salloc (or any other job) on, the expected behavior is for that terminal (and all related processes) to be killed by slurm. This is to ensure users get the resources they request and will not be changing.

[awadell1]

Correction: the System OpenMPI is slurm aware. Apologies for the confusion

[maituoy]

Unfortunately, none of the scientific packages we usually use are able to be installed via spack. That means I was not able to test a package linked to OpenMPI and slurm.

The good news is I found out that we can run the parallel jobs on the interactive nodes by not login into the nodes. Basically, I just submit an interactive job using the salloc and run executable binaries with srun right after allocation is granted. I'm still testing this approach. So far, it works well if your job only involves a scientific package. However, if you would like to embed your calculation in the subprocess of python script (such as using an ASE calculator), it can easily encounter a memory issue on complicated calculations (I compared the PBE and HSE functional in DFT).