Slurm

From CAC Documentation wiki
Revision as of 16:40, 18 June 2019 by Srl6 (talk | contribs)
Jump to navigation Jump to search

Some of the CAC's Private Clusters are managed with OpenHPC, which includes the Slurm Workload Manager (Slurm for short). Slurm (originally the Simple Linux Utility for Resource Management) is a group of utilities used for managing workloads on compute clusters.

This page is intended to give users an overview of Slurm. Some of the information on this page has been adapted from the Cornell Virtual Workshop topics on the Stampede2 Environment and Advanced Slurm. For a more in-depth tutorial, please review these topics directly.

One important feature of CAC's typical Slurm configuration is that scheduling is done by CPU, not by node. This means that by default, a node may be shared among multiple users.

Overview

Some clusters use Slurm as the batch queuing system and the scheduling mechanism. This means that jobs are submitted to Slurm from a login node and Slurm handles scheduling these jobs on nodes as resources becomes available. Users submit jobs to the batch component which is responsible for maintaining one or more queues (also known as "partitions"). These jobs include information about themselves as well as a set of resource requests. Resource requests include anything from the number of CPUs or nodes to specific node requirements (e.g. only use nodes with > 2GB RAM). A separate component, called the scheduler, is responsible for figuring out when and where these jobs can be run on the cluster. The scheduler needs to take into account the priority of the job, any reservations that may exist, when currently running jobs are likely to end, etc. Once informed of scheduling information, the batch system will handle starting your job at the appropriate time and place. Slurm handles both of these components, so you don't have to think of them as separate processes, you just need to know how to submit jobs to the batch queue(s).

Note: Refer to the documentation for your cluster to determine what queues/partitions are available.

Running Jobs

This section covers general job submission and job script composition; for more specific details on how to run jobs or job scripts and use queues on your particular system, see the documentation for the Private Cluster you are working on. Also note, many of the following commands have several options. For full details, see the man page for the command, or the Slurm Docs.

Display Info

Common commands used to display information:

  • sinfo displays information about nodes and partitions/queues. Use -l for more detailed information.
  • scontrol show nodes views the state of the nodes.
  • scontrol show partition views the state of the partition/queue.

Job Control

Here are some common Job Control commands:

  • sbatch testjob.sh submits a job where testjob.sh is the script you want to run. Also see the Job Scripts section and the sbatch documentation.
  • srun -p <partition> --pty /bin/bash -l starts an interactive job and opens a login shell for you to enter commands into. Also see the srun documentation.
    • Note: remember to exit the session once you are done to free resources for other users.
  • squeue -u my_userid shows state of jobs for user my_userid. Also see the squeue documentation.
  • scontrol show job <job id> views the state of a job. Also see the scontrol documentation.
  • scancel <job id> cancels a job. Also see the scancel documentation.
  • squeue with no arguments retrieves summary information on all jobs scheduled.

Once the job has completed, the stdout and stderr streams will be put in your $HOME directory named with the job id. To verify the job ran successfully, examine these output files.

Key Arguments

The following table shows key directives that may be specified for each job. If these directives are not supplied, then default values go into effect, which could influence how your job runs. Some clusters may require one or more of these arguments to be provided.

Meaning Flag Allowed Value Example Default
Submission queue -p Queue/partition name
(valid names are cluster dependent)
-p normal (cluster dependent)
Job walltime -t hh:mm:ss
(not to exceed time limit of queue)
-t 00:05:00 time limit of queue
Number of tasks -n 1 ... number of CPUs on N nodes
(Slurm calculates N, if -N is not present)
-n 16 2*N
(2, if -N is also not present)
Number of nodes -N 1 ... NP = number of nodes in partition
(if N > NP, job is queued but never runs)
-N 2 enough to satisfy -n
(1, if -n is also not present)

Specifying a time limit is helpful to you, the scheduler, and your fellow cluster users. If you know in advance that your job will not take the maximum running time allowed in the queue, use the -t option to give an accurate expectation. This encourages the scheduler to "backfill" your job so it might run sooner that it otherwise would. As a nice bonus, your job may get out of the way of other waiting jobs.

As you can see, the maximum number of tasks that is feasible for a job depends on the hardware that is available in the chosen queue. In the absence of any other information, Slurm allocates the minimum number of nodes that can accommodate -n tasks with each task occupying one CPU. (You must be careful that this number of nodes does not exceed the total number in the partition!)

It is important to understand that Slurm typically counts each physical core of a multi-core processor as two CPUs. This is due to Intel's hyperthreading technology, which makes each physical core appear to be two hardware threads to the OS. For this reason, your job will always be assigned an even number of CPUs. Even a job with one task is assigned two CPUs.

Accordingly, Slurm calculates the total number of CPUs per node as follows:

CPUs/node = (boards/node) * (sockets/board) * (cores/socket) * (hardware threads/core)
          =       1       *        2        * (cores/socket) *            2

Most of the above factors are fixed, because nearly all CAC clusters consist of dual-socket nodes, i.e., each of the 2 sockets in the node holds one Intel multi-core processor. The number of cores per processor is generally the only variable, and it can vary quite a lot, even within a single queue. Check your cluster's documentation for details.

It is possible to assign fewer process to each node via the -N argument (see the Optional Arguments section), which specifies the desired number of nodes.

Example Command-line Job Submission

All of the required options can be specified on the command-line with sbatch. For example, say you had the following script "simple_cmd.sh" to run:

#!/bin/bash
#Ensures that the node can sleep

#print date
date
#verify that sleep 5 works
time sleep 5

In order to run this on the command-line, you could issue (where development is an available queue on the system):

$  sbatch -p development -t 00:01:00 -n 1 simple_cmd.sh

There is also an easier way, as demonstrated in Job Scripts.

Optional Arguments

Meaning Flag Value Example
Name of Job -J any string -J SimpleJob
Stdout -o absolute path -o $HOME/project1/%j.out
Sterr -e absolute path -e $HOME/project1/%j.err
Job Dependency -d type:job_id -d=after:1234
Email address --mail-user email@domain --mail-user=genius@gmail.com
Email notification type --mail-type BEGIN, END, FAIL, REQUEUE, or ALL --mail-type=ALL
Specify Environment Variable --export varname=varvalue, ALL (default), NONE --export LOC=$SCRATCH/x/foo.cdf

Of particular importance to those who run MPI applications, the -N and -n arguments may be used together in order to determine the number of processes running on each node. Running fewer processes on a given node means that each process can safely consume a greater percentage of a node's memory.

If -N is specified along with -n, Slurm will allocate the number of nodes implied by -N, then evenly divide the total number of processes specified with -n amongst the nodes. For example, -N 8 -n 32 will specify 32 processes to be launched, divided evenly among the 8 allocated nodes (i.e. 32 processes ÷ 8 nodes = 4 processes per node).

Job Scripts

For more specific examples on how to write job scripts and use queues on your particular system, see the documentation for the Private Cluster you are working on.

Simple Job Script

For the same example from above, the same commands can be put into the batch script itself. This makes it easy to copy and paste to new scripts as well as be confident that a job is submitted the same way over and over again. We'll modify the previous script so that it includes all of the required directives.

All that is required is to place the command line options in the batch script and prepend them with #SBATCH. They appear as comments to the shell, but Slurm parses them for you and applies them. Here is the end result:

#!/bin/bash
#Ensures that the node can sleep

#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -p development

#print date
date
#verify that sleep 5 works
time sleep 5

References