-
Notifications
You must be signed in to change notification settings - Fork 0
Lesson 1: getting to know the cluster
(This section is based on material from this slurm tutorial
In this lesson you will experiment with the difference between the head node and compute nodes and learn some basic commands for seeing information about the cluster as a whole.
If you are are not familiar with how computer clusters work, you can read more here.
In order to use this tutorial, you will need access to a computer cluster using the Slurm job management system. This access consists of a username and password. You will need to obtain these from the cluster administrator.
Also, in order to connect to the cluster you need a terminal program. If you are on a Linux computer you most likely already know how to access the command terminal. If you are on a Macintosh computer, you can run the builtin program "Terminal" or the superior free replacement "Iterm2." If you are using a computer running Windows, install MobaXTerm.
For this tutorial, we will assume your cluster has a head node called clusterhead1, and nodes named clusternode-#-#.
In the terminal type:
ssh clusthead1
You will receive a prompt asking you for your username and password. If this is your first time logging in, you will also see some stuff about your rsa fingerprint. Just say "yes."
Once you are logged in, you will have a command line on the head node. To confirm this, type the linux command hostname.
balter@clusthead1:~$ hostname
clusthead1
Let's find out a bit about the head node, for instance what kind of CPU it has. The following command first gets a long list of information about all of the many CPUs on the head node and then filters for one piece of information. Type:
cat /proc/cpuinfo | grep "model name" | uniq
balter@clusthead1:~$ cat /proc/cpuinfo | grep "model name" | uniq
model name : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Now let's submit those same commands to the job scheduler, and see what happens. Just type the commands entered at the terminal prompt "$".
balter@clusthead1:~$ srun hostname
clustnode-3-56.local
The command srun told the head node "please have the job scheduler find me a place to run the command hostname." So this time, we ran hostname on a compute node, not the head node. As we can see, the processor on this node (clustnode-3-56.local) is clearly different from the one on the head node.
balter@clusthead1:~$ srun cat /proc/cpuinfo | grep "model name" | uniq
model name : Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
Slurm offers many commands you can use to interact with the system. For instance, the sinfo command gives an overview of the resources offered by the cluster, while the squeue command shows to which jobs those resources are currently allocated.
By default, sinfo lists the partitions that are available. A partition is a set of compute nodes grouped logically. Typical examples include partitions dedicated to batch processing, debugging, post processing, or visualization.
balter@clusthead1:~$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
exacloud* up 1-12:00:00 3 idle clustnode-2-[8,47],clustnode-3-56
short_jobs up 1:00:00 1 down* clustnode-3-1
short_jobs up 1:00:00 5 idle clustnode-2-[8,47],clustnode-3-56,clustnode-4-[12,44]
long_jobs up 10-00:00:0 3 idle clustnode-2-[8,47],clustnode-3-56
very_long_jobs up 30-00:00:0 3 idle clustnode-2-[8,47],clustnode-3-56
In the above example, we see partitions dedicated to different length jobs, and exacloud, which is the default partition as it is marked with an asterisk. All nodes of the debug partition are idle, while one is down.
The command sinfo can output the information in a node-oriented fashion, with the argument -N
balter@clusthead1:~$ sinfo -N -l
Thu Jun 1 19:57:29 2017
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
clustnode-2-8 1 short_jobs idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-8 1 exacloud* idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-8 1 long_jobs idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-8 1 very_long_jobs idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-47 1 short_jobs idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-47 1 exacloud* idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-47 1 long_jobs idle 24 2:12:1 61440 0 1 (null) none
clustnode-2-47 1 very_long_jobs idle 24 2:12:1 61440 0 1 (null) none
clustnode-3-1 1 short_jobs down* 24 2:12:1 61440 0 1 (null) Not responding
clustnode-3-56 1 short_jobs idle 24 2:12:1 126976 0 1 (null) none
clustnode-3-56 1 exacloud* idle 24 2:12:1 126976 0 1 (null) none
clustnode-3-56 1 long_jobs idle 24 2:12:1 126976 0 1 (null) none
clustnode-3-56 1 very_long_jobs idle 24 2:12:1 126976 0 1 (null) none
clustnode-4-12 1 short_jobs idle 36 2:18:1 245760 0 1 (null) none
clustnode-4-44 1 short_jobs idle 36 2:18:1 245760 0 1 (null) none
Note that with the additional -l argument, more information about the nodes is provided: number of CPUs, memory, temporary disk (also called scratch space), node weight (an internal parameter specifying preferences in nodes for allocations when there are multiple possibilities), features of the nodes (such as processor type for instance) and the reason, if any, for which a node is down.
You can actually specify precisely what information you would like sinfo to output by using its --format argument. For more details, have a look at the sinfo manpage with man sinfo.
The squeue command shows the list of jobs which are currently running (they are in the RUNNING state, noted as 'R') or waiting for resources (noted as 'PD').
balter@clusthead1:~/slurm_tutorial$ squeue -l
Mon Jun 12 10:34:42 2017
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
66 exacloud hostname nds COMPLETI 0:52 1:00:00 1 clustnode-2-47
65 exacloud hostname nds COMPLETI 0:52 1:00:00 1 clustnode-2-8
70_60 exacloud sleepyti balter COMPLETI 0:00 1:00:00 1 clustnode-3-56
70_120 exacloud sleepyti balter RUNNING 26:12 1:00:00 1 clustnode-4-12
70_180 exacloud sleepyti balter RUNNING 26:12 1:00:00 1 clustnode-4-12
70_240 exacloud sleepyti balter RUNNING 26:12 1:00:00 1 clustnode-4-12
70_300 exacloud sleepyti balter RUNNING 26:25 1:00:00 1 clustnode-4-12
The above output show that user nds has two jobs that are in the "Completing" state. One whose Slurm job ID is 66 is on clustnode-2-47 and the other with Slurm job ID 65 is on clustnode-2-8. User balter has 5 jobs with job IDs 70-[60,120,180,240,300]. The job ID is a unique identifier that is used by many Slurm commands when actions must be taken about one particular job. For instance, to cancel one of these jobs, you oculd use scancel 70-180. Time is the time the job has been running until now. Node is the number of nodes which are allocated to the job, while the Nodelist column lists the nodes which have been allocated for running jobs. For pending jobs, that column gives the reason why the job is pending.
There are many switches you can use to filter the output by user (--user), by partition (--partition) by state (--state), etc.
To learn to use the cluster, you can now progress to Lesson 2: Submitting Jobs.