- pool is a joint cluster by two departments: Chemical and Biomolecular Engineering and Chemistry
- PIs are Fernando Escobedo, Don Koch, Yong Joo, Robert DiStasio, and Nandini Ananth
- pool is a private cluster with restricted access to the following groups: fe13_0001, dlk15_0001, ylj2_0001, rad332_0001, na346_0001
- Head node: pool.cac.cornell.edu (access via ssh)
- OpenHPC deployment running Centos 7
- Scheduler: slurm 18
How To Login
- There is a 1.8TB local /scratch disk on the head node only.
- Pool Hardware Details
- All nodes have a 1GB ethernet connection for eth0 on a private net served out from the pool head node.
Running Jobs / Slurm Scheduler
CAC's Slurm page explains what Slurm is and how to use it to run your jobs on CAC clusters. Please take the time to read this page, giving special attention to the parts that pertain to the types of jobs you want to run.
A few slurm commands to initially get familiar with: sinfo -l scontrol show nodes scontrol show partition Submit a job: sbatch testjob.sh Interactive Job: srun -p normal --pty /bin/bash scontrol show job [job id] scancel [job id] squeue -u userid
("Partition" is the term used by slurm for "Queues")
- hyperthreading is turned on for ALL nodes - Slurm considers each core to consist of 2 logical CPUs
- all partitions have a default time of 1 hour and are set to OverSubscribe
Partitions on the pool cluster:'
Queue/Partition Number of nodes Node Names Limits Group Access common (default) 18 c00[17,19,20,22,29-38,50-53] walltime limit: 168 hours (i.e. 7 days) All Groups plato 1 c0009 walltime limit: 168 hours (i.e. 7 days) limited access per fe13 fe13 23 c00[01-08,18,21,23,24,40-49,54] walltime limit: 168 hours (i.e. 7 days) fe13_0001 dlk15 7 c00[10-14,25,39] walltime limit: 168 hours (i.e. 31 days) dlk15_0001 ylj2 5 c00[15-16,26-28] walltime limit: 168 hours (i.e. 7 days) ylj2_0001 vega 7 c00??-c0???] walltime limit: 168 hours (i.e. 7 days) rad332_0001
Slurm Examples & Tips
- NOTE: All lines begining with "#SBATCH" are a directive for the scheduler to read. If you want the line ignored (i.e. a comment), you must place 2 "##" at the beginning of your line.
Example batch job to run in the partition: common
Example sbatch script to run a job with one task (default) in the 'common' partition (i.e. queue):
#!/bin/bash ## -J sets the name of job #SBATCH -J TestJob ## -p sets the partition (queue) #SBATCH -p common ## 10 min #SBATCH --time=00:10:00 ## sets the tasks per core (default=2 for hyperthreading: cores are oversubscribed) ## set to 1 if one task by itself is enough to keep a core busy #SBATCH --ntasks-per-core=1 ## request 4GB per CPU (may limit # of tasks, depending on total memory) #SBATCH --mem-per-cpu=4GB ## define job stdout file #SBATCH -o testcommon-%j.out ## define job stderr file #SBATCH -e testcomon%j.err echo "starting at `date` on `hostname`" # Print the Slurm job ID echo "SLURM_JOB_ID=$SLURM_JOB_ID" echo "hello world `hostname`" echo "ended at `date` on `hostname`" exit 0
Submit/Run your job:
View your job:
scontrol show job <job_id>
Example MPI batch job to run in the partition: common
Example sbatch script to run a job with 60 tasks in the 'common' partition (i.e. queue):
#!/bin/bash ## -J sets the name of job #SBATCH -J TestJob ## -p sets the partition (queue) #SBATCH -p common ## 10 min #SBATCH --time=00:10:00 ## the number of slots (CPUs) to reserve #SBATCH -n 60 ## the number of nodes to use (min and max can be set separately) #SBATCH -N 3 ## typically an MPI job needs exclusive access to nodes for good load balancing #SBATCH --exclusive ## don't worry about hyperthreading, Slurm should distribute tasks evenly ##SBATCH --ntasks-per-core=1 ## define job stdout file #SBATCH -o testcommon-%j.out ## define job stderr file #SBATCH -e testcommon-%j.err echo "starting at `date` on `hostname`" # Print Slurm job properties echo "SLURM_JOB_ID = $SLURM_JOB_ID" echo "SLURM_NTASKS = $SLURM_NTASKS" echo "SLURM_JOB_NUM_NODES = $SLURM_JOB_NUM_NODES" echo "SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST" echo "SLURM_JOB_CPUS_PER_NODE = $SLURM_JOB_CPUS_PER_NODE" mpiexec -n $SLURM_NTASKS ./hello_mpi echo "ended at `date` on `hostname`" exit 0
To include or exclude specific nodes in your batch script
To run on a specific node only, add the following line to your batch script:
#SBATCH -w, --nodelist=c0009
To include one or more nodes that you specifically want, add the following line to your batch script:
#SBATCH --nodelist=<node_names_you_want_to_include> ## e.g., to include c0006: #SBATCH --nodelist=c0006 ## to include c0006 and c0007 (also illustrates shorter syntax): #SBATCH -w c000[6,7]
To exclude one or more nodes, add the following line to your batch script:
#SBATCH -exclude=<node_names_you_want_to_exclude> ## e.g., to avoid c0006 through c0008, and c0013: #SBATCH -exclude=c00[06-08,13] ## to exclude c0006 (also illustrates shorter syntax): #SBATCH -x c0006
Environment variables defined for tasks that are started with srun
If you submit a batch job in which you run the following script with "srun -n $SLURM_NTASKS", you will see how the various environment variables are defined.
#!/bin/bash echo "Hello from `hostname`," \ "$SLURM_CPUS_ON_NODE CPUs are allocated here," \ "I am rank $SLURM_PROCID on node $SLURM_NODEID," \ "my task ID on this node is $SLURM_LOCALID"
These variables are not defined in the same useful way in the environments of tasks that are started with mpiexec or mpirun.
Use $HOME within your script rather than the full path to your home directory
In order to access files in your home directory, you should use $HOME rather than the full path . To test, you could add to your batch script:
echo "my home dir is $HOME"
Then view the output file you set in your batch script to get the result.
Copy your data to /tmp to avoid heavy I/O from your nfs mounted $HOME !!!
- We cannot stress enough how important this is to avoid delays on the file systems.
#!/bin/bash ## -J sets the name of job #SBATCH -J TestJob ## -p sets the partition (queue) #SBATCH -p common ## time is HH:MM:SS #SBATCH --time=00:01:30 #SBATCH --cpus-per-task=15 ## define job stdout file #SBATCH -o testcommon-%j.out ## define job stderr file #SBATCH -e testcommon-%j.err echo "starting $SLURM_JOBID at `date` on `hostname`" echo "my home dir is $HOME" ## copying my data to a local tmp space on the compute node to reduce I/O MYTMP=/tmp/$USER/$SLURM_JOB_ID /usr/bin/mkdir -p $MYTMP || exit $? echo "Copying my data over..." cp -rp $SLURM_SUBMIT_DIR/mydatadir $MYTMP || exit $? ## run your job executables here... echo "ended at `date` on `hostname`" echo "copy your data back to your $HOME" /usr/bin/mkdir -p $SLURM_SUBMIT_DIR/newdatadir || exit $? cp -rp $MYTMP $SLURM_SUBMIT_DIR/newdatadir || exit $? ## remove your data from the compute node /tmp space rm -rf $MYTMP exit 0
Explanation: /tmp refers to a local directory that is found on each compute node. It is faster to use /tmp because when you read and write to it, the I/O does not have to go across the network, and it does not have to compete with the other users of a shared network drive (such as the one that holds everyone's /home).
To look at files in /tmp while your job is running, you can ssh to the login node, then do a further ssh to the compute node that you were assigned. Then you can cd to /tmp on that node and inspect the files in there with
Note, if your application is producing 1000's of output files that you need to save, then it is far more efficient to put them all into a single tar or zip file before copying them into $HOME as the final step.
The 'lmod module' system is implemented for your use with listing and loading modules that will put you in the software environment needed. (For more information, type: module help)
To list the available software and the software environment you can put yourself in, type:
(to get a more complete listing, type: module spider) The software that is listed with "(L)" references what you have loaded. EXAMPLE: To be sure you are using the environment setup for gromacs, you would type:
module load gromacs/2019.1 module list (you will see gromacs is loaded (L)) * when done, either logout and log back in or type: module unload gromacs/2019.1
You can create your own modules and place them in your $HOME. Once created, type: module use $HOME/path/to/personal/modulefiles This will prepend the path to $MODULEPATH [type echo $MODULEPATH to confirm]
Reference: User Created Modules
- It is usually possible to install software in your home directory.
- List installed software via rpms: rpm -qa. Use grep to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]
Build software from source into your home directory ($HOME)
* download and extract your source * cd to your extracted source directory ./configure --./configure --prefix=$HOME/appdir [You need to refer to your source documentation to get the full list of options you can provide 'configure' with.] make make install The binary would then be located in ~/appdir/bin. * Add the following to your $HOME/.bashrc: export PATH="$HOME/appdir/bin:$PATH" * Reload the .bashrc file with source ~/.bashrc. (or logout and log back in)