Pool Cluster
Getting Started
General Information
- pool is a private cluster with restricted access to the following groups:
- Head node: pool.cac.cornell.edu (access via ssh)
- Open HPC deployment running Centos 7.6
- Cluster scheduler: slurm 17.11.10
/home
15TB directory server (nfs exported to all cluster nodes)
-
- 9 compute nodes c00[01-9]
- Current Cluster Status: Ganglia.
- Please send any questions and report problems to: cac-help@cornell.edu
How To Login
- To get started, login to the head node pool.cac.cornell.edu via ssh.
- If you are unfamiliar with Linux and ssh, we suggest reading the Linux Tutorial and looking into how to Connect to Linux before proceeding.
- You will be prompted for your CAC account password
Hardware
c000[1-9] hyperthreading ON
Node Names Memory per node Model name Processor count per node Core(s) per socket Sockets Thread(s) per core
Networking
- All nodes have a 1GB ethernet connection for eth0 on a private net served out from the pool head node.
Running Jobs
Slurm
Queues/Partitions
("Partition" is the term used by slurm)
- hyperthreading is turned on for ALL nodes
- all partitions have a default time of 1 hour
- pool has X separate queues:
Queue/Partition Number of nodes Node Names Limits normal (default) 9 c000[1-9] walltime limit: 4 hours
Common Slurm Commands
Command/Option Summary (two page PDF)
Slurm HELP
Slurm Workload Manager Quick Start User Guide - this page lists all of the available Slurm commands
Slurm Workload Manager Frequently Asked Questions includes FAQs for Management, Users and Administrators
Convenient SLURM Commands has examples for getting information on jobs and controlling jobs
Slurm Workload Manager - sbatch - used to submit a job script for later execution. The script will typically contain one or more srun commands to launch parallel tasks.
A few slurm commands to initially get familiar with: scontrol show nodes scontrol show partition Submit a job: sbatch testjob.sh Interactive Job: srun -p short --pty /bin/bash scontrol show job [job id] scancel [job id] sinfo -l
Example in Short Partition/Queue
Example sbatch file to run a job in the short partition/queue; save as example.sh:
#!/bin/bash ## J sets the name of job #SBATCH -J TestJob ## -p sets the partition (queue) #SBATCH -p normal ## 10 min #SBATCH --time=00:10:00 ## sets the tasks per core (default=2; keep default if you want to take advantage of hyperthreading) ## 2 will take whole cores, but will divide by 2 with hyperthreading #SBATCH --ntasks-per-core=1 ## request 300MB per core #SBATCH --mem-per-cpu=4GB ## define jobs stdout file #SBATCH -o testnormal-%j.out ## define jobs stderr file #SBATCH -e testnormal-%j.err echo "starting at `date` on `hostname`" # Print the SLURM job ID. echo "SLURM_JOBID=$SLURM_JOBID" echo "hello world `hostname`" echo "ended at `date` on `hostname`" exit 0
Submit/Run your job:
sbatch example.sh
View your job:
scontrol show job 9
Software
Installed Software
The 'lmod module' system is implemented. To list an environment you can put yourself in:
module avail
(to get a more complete listing, type: module spider)
EXAMPLE: To be sure you are using the environment setup for gromacs, you would type:
* module load gromacs/2019.1 - when done, either logout and log back in or type: * module unload gromacs/2019.1
You can create your own modules and place them in your $HOME. Once created, type: module use $HOME/path/to/personal/modulefiles This will prepend the path to $MODULEPATH [type echo $MODULEPATH to confirm]
Reference: User Created Modules
-sh-4.2$ module avail -------------------------------------------------------- /opt/ohpc/pub/moduledeps/gnu8-openmpi3 -------------- netcdf-fortran/4.4.4 netcdf/4.6.1 phdf5/1.10.3 ------------------------------------------------------------ /opt/ohpc/pub/moduledeps/gnu8 --------------------- mpich/3.2.1 mvapich2/2.3 openmpi3/3.1.2 (L) -------------------------------------------------------------- /opt/ohpc/pub/modulefiles -------------------------- autotools (L) gnu/5.4.0 gnu8/8.2.0 (L) intel/19.0.2.187 matlab/R2018a pmix/2.1.4 spack/0.11.2 cmake/3.12.2 gnu7/7.3.0 gromacs/2019.1 intel19 ohpc (L) prun/1.2 (L) valgrind/3.13.0 Where: L: Module is loaded Use "module spider" to find all possible modules.
- It is usually possible to install software in your home directory.
- List installed software via rpms: 'rpm -qa'. Use grep to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]
Build software from source into your home directory ($HOME)
* download and extract your source * cd to your extracted source directory * ./configure --./configure --prefix=$HOME/appdir [You need to refer to your source documentation to get the full list of options you can provide 'configure' with.] * make * make install The binary would then be located in ~/appdir/bin. * Add the following to your $HOME/.bashrc: export PATH="$HOME/appdir/bin:$PATH" * Reload the .bashrc file with source ~/.bashrc. (or logout and log back in)
How to Install R packages in your home directory
Reference: http://cran.r-project.org/doc/manuals/R-admin.html#Managing-libraries
************************************************************************************
NOTE: Steps 1) through 4) need to be done once, after your Rlibs directory
has been created and your R_LIBS environment is set, you can install additional
packages using step 5).
************************************************************************************
Know your R library search path:
Start R and run .libPaths() Sample output is shown below:
> .libPaths()
[1] "/usr/lib64/R/library"
Now we will create a local Rlibs directory and add this to the library search path.
NOTE: Make sure R is NOT running before you proceed.
1) Create a directory in your home directory you would like to install the R packages, e.g. Rlibs
mkdir ~/Rlibs
2) Create a .profile file in your home directory (or modify existing) using your favorite editor (emacs, vim, nano, etc)
Add the following to your .profile
#!/bin/sh
if [ -n $R_LIBS ]; then
export R_LIBS=~/Rlibs:$R_LIBS
else
export R_LIBS=~/Rlibs
fi
3) To reset the R_LIBS path we need to run the following: "source ~/.profile" (or logout and log back in)
4) Confirm the change is in your library path:
start R
> .libPaths()
[1] "$HOME/Rlibs"
[2] "/usr/lib64/R/library"
5) Install the package in your local directory
>install.packages("packagename","~/Rlibs","https://cran.r-project.org/")
i.e. to install the package:snow
>install.packages("snow","~/Rlibs","https://cran.r-project.org/")
6) For more help with install.packages() use
>?install.packages( )
7) To see which libraries are available in your R library path, run library()
The output will show your local packages and the system wide packages
>library()
How to Install Python Anaconda (miniconda) home directory
- Anaconda can be used to maintain custom environments for R (as well as other software).
- Reference to help decide if miniconda is enough: https://conda.io/docs/user-guide/install/download.html
- NOTE: Consider starting with miniconda if you do not need a multitude of packages for it will be smaller, faster to install as well as update.
- Reference for Anaconda R Essentials: https://conda.io/docs/user-guide/tasks/use-r-with-conda.html
- Reference for linux install: https://conda.io/docs/user-guide/install/linux.html
- Please take the tutorials to assist you with your management of conda packages: