Difference between revisions of "DSS1 Cluster"

From CAC Documentation wiki
Jump to navigation Jump to search
Line 241: Line 241:
====Running a Job Using 1 Task Per Node====
====Running a Job Using 1 Task Per Node====
Use showq to see how many cores are available. It may be less than 208 (26x8) if a node is down.<br />
Use showq to see how many cores are available. It may be less than 192 (24x8) if a node is down.<br />
<source lang="bash">
<source lang="bash">
#PBS -l nodes=26:ppn=8,walltime=10:00
#PBS -l nodes=24:ppn=8,walltime=10:00
#PBS -N test
#PBS -N test
#PBS -j oe
#PBS -j oe
Line 259: Line 259:
ntasks=$(wc -l < nodefile.1way)
ntasks=$(wc -l < nodefile.1way)
# Run 1-way on 26 nodes
# Run 1-way on 24 nodes
# Substitute your application name for "hostname -v" if desired
# Substitute your application name for "hostname -v" if desired
mpiexec -np $ntasks --hostfile nodefile.1way --prefix /opt/openmpi hostname -v
mpiexec -np $ntasks --hostfile nodefile.1way --prefix /opt/openmpi hostname -v

Revision as of 12:35, 12 June 2017

DSS Cluster General Information

  • Data on the dss Cluster is ***NOT*** backed up!!!! In the event of a compute node crash, the compute node will be rebuilt and all data will be lost!!!!
  • User's home directories are ***NOT*** backed up!!!! The storage is on a RAID-6 volume so there is some degree of safety, but you should keep a second copy of important files somewhere else.
  • dss is a private cluster for the DSS group - jab18_0002
  • dss has one head node and 24 compute nodes (compute-1-[1-8,12,14]; compute-2-[1-2,4-15])
    • dss1.cac.cornell.edu head node: Dell PowerEdge 2650 w/ 16GB RAM, RAID 5
    • All Compute nodes: Dell PowerEdge 2650 w/ 16GB RAM, 8 cpus, RAID 0 w/ 2TB disk space in /tmp
  • Head node: dss1.cac.cornell.edu (access via ssh) serves as:
    • the cluster scheduler
    • Rocks Cluster server for compute node deployment
    • /home/fs01/[login](nfs exported to all cluster nodes)
  • Rocks 6.0 with CentOS 6.2
  • Cluster Status: Ganglia.
  • Submit HELP requests: help OR by sending email to: help@cac.cornell.edu



  • Default python on dss cluster: 2.6.6
  • 2.7 is installed in: /opt/python2.7
  • For Python, most packages with a setup.py file will install into your home directory
    • cd into the package directory and issue the command:
python setup.py install --prefix=$HOME

(Reminder, the default python might not be the one you wish to use)

  • For a specific version, you will need more options to find the necessary compilers/libraries:
    • i.e. Python 2.7:
 /usr/local/bin/python2.7 setup.py build_ext --library-dirs=/opt/python2.7/lib config_fc --fcompiler=gnu95 install --prefix=$HOME 


  • Use software VirtualEnv creating a sandbox to install and use Python packages.

Further reading: docs.python.org

R (v3.3.0)

Most of the information that follows is taken from the R Installation and Administration Managing libraries section at http://cran.r-project.org/doc/manuals/R-admin.html#Managing-libraries

NOTE: Steps 1) through 4) need to be done once, after your Rlibs directory
has been created and your R_LIBS environment is set, you can install additional 
packages using step 5).

Before you begin you may want to check your R library search path:
    Start R and run .libPaths()  Sample output is shown below:
    > .libPaths()
     [1] "/usr/lib64/R/library"

Now we will create a local Rlibs directory and add this to the library search path.
NOTE: Make sure R is NOT running before you proceed.

1) Create a directory in your home directory you would like to install the R packages, e.g. Rlibs 
mkdir  ~/Rlibs

2) Create a .profile file in your home directory (or modify existing) using your favorite editor (emacs, vim, etc)  
   Note: The default shell for DSS users is /bin/bash (alias to /bin/sh) and the .profile  
          is read and executed every time you login. 
     Add the following to your .profile
     if [ -n $R_LIBS ]; then
        export R_LIBS=~/Rlibs:$R_LIBS
        export R_LIBS=~/Rlibs

3) To reset the R_LIBS path we need to run the following: "source ~/.profile" (or logout and log back in) 

4) Start R and rerun the .libPaths() command and you should see something similar to:
> .libPaths()
[1] "/home/fs01/username/Rlibs"     (where username is your netid)
[2] "/usr/lib64/R/library"   

5) Now you can install a package in your local directory 
NOTE: Replace "packagename" with the actual name of the package you want installed.  
        If you leave "packagename" blank, you'll get a list of all available packages. 
         The example uses the CRAN site at Iowa, if you leave the site blank, 
         you'll get a list of all available CRAN repository sites.

6) For more help with install.packages() use
>?install.packages( )  

7) To see which libraries are available in your R library path, run library() 
The output will show your local packages and the system wide packages


  • MATLAB R2012b is on the head node, and the R2012b runtimes are on the compute nodes.
  • Compile your MATLAB script on the head node by starting up MATLAB and running "mcc -m matlabexecutable.m".
  • How to run compiled MATLAB code on the compute nodes

To properly set up the environment to run MATLAB compiled code add the following line to your job script (case sensitive):

module load MCR2012b

Here is a sample job script. You can create a file called job.sh and copy the following text into it:

#PBS -l nodes=1,walltime=30:00
#PBS -N compiledmatlab
#PBS -j oe
#PBS -S /bin/bash
set -x


# load MCR2012b environment module
module load MCR2012b
# run your compiled matlab code

In place of "username" and "matlabexecutable", put your username and the name of your compiled code. If your code takes any arguments, just place these arguments following the name of your executable in the usual way. You do not need the script "run_matlabexecutable.sh" that was created by mcc.

To run the above script in batch, just enter "qsub job.sh".


  • In addition to the system default gcc, you can find /opt/gcc-4.6.4/bin/gcc

How to run jobs using the scheduler


The dss cluster has only one queue at this time:

  • default
Number of nodes: 24 compute nodes
Node Names: compute-1-[1-8,14,15]; compute-2-[1-2,4-15]
Memory: 16GB ram/server
Limits: Maximum of 24 nodes, walltime limit: None

Quick Tutorial

The batch system on dss treats each server as a node, provided ppn (i.e., processes per node) is greater than 1.
If ppn=1 or is absent, scheduling is done by core. Thus, nodes=16:ppn=1 actually puts 8 tasks on each of 2 nodes!
This can make it a bit tricky to get jobs to run on the intended cores. In general, you must use a hostfile.
If you don't specify a hostfile, all processes will run on "node 0" (that is, the first node assigned to your job).

Jobs are scheduled by the Maui scheduler with the Torque resource manager. We suggest you use a job submission batch file utilizing PBS Directives ('Options' section).

If you have any experience with PBS/Torque or SGE, Maui Commands may be recognizable.

Creating and Submitting a Batch Script

A batch script on the cluster head node is a script that informs the scheduler what to do. (#PBS without a space gives the scheduler a directive; # followed by a space and wording is a comment)

Create the script using your favorite editor:

# job is requesting 5 minutes, 30 seconds, one node, 8 processors (all on the node)
#PBS -l walltime=00:05:30,nodes=1:ppn=8
# join your output and error for easier debugging
#PBS -j oe
# make a comment
#PBS -N testdefaultqueue 

# Turn on echo of shell commands
set -x
# Because jobs start in the HOME directory, move to where we submitted.
echo 'pwd'
echo "PBS_O_WORKDIR is `pwd`"
echo "env is `env`"
# copy the binary and data files to a local directory on node job is executing on
cp $HOME/bin/helloworld.sh $TMPDIR
# cp $HOME/values $TMPDIR/
# run your executable from the local disk on the node the job was placed on
./helloworld.sh >&hello.stdout
# Copy output files to your output folder       
cp -f $TMPDIR/hello.stdout $HOME/output

Submit your script:

qsub scriptname

checkjob -v jobid (to see the details of your job)

showq (to show all jobs in the queue) canceljob jobid (cancels your job)

More reference:

Running an interactive job

  • Be sure to have logged in w/ X11 forwarding enabled (if using ssh, use ssh -X or ssh -Y; if using putting, you need to be sure to check the X11 forwarding box)
  • You do not have to specify a specific node as below

Running an interactive job on a specific node

To run on a specific node use the '-l' with the'host='option; to run Interactively, use the '-I' (capital I) option. Example below is requesting you get the node for 5 days:

#PBS -l host=compute-1-5, walltime=120:10:00       
#PBS -N test
#PBS -j oe
#PBS -q default

set -x

Running a Job on the Whole Cluster

NOTE: this example is for 24 nodes up:

First use showq to see how many cores are available. It may be less than 192 (24x8) if a node is down.
Because the nodes have 8 physical cores, it is often best to have your job run exactly 8 tasks per node.
The node file constructed by Maui/Torque lists each node 8 times, so you can just hand that file to MPI.
Your job doesn't actually have to use MPI at all; the script just uses mpiexec as a process launcher.

#PBS -l nodes=24:ppn=8,walltime=10:00
#PBS -N test
#PBS -j oe
#PBS -q default
#PBS -S /bin/bash

set -x

# Run 8-way on 24 nodes
# Substitute your application name for "hostname -v" if desired
mpiexec --hostfile $PBS_NODEFILE --prefix /opt/openmpi hostname -v

Running a Job Using 1 Task Per Node

Use showq to see how many cores are available. It may be less than 192 (24x8) if a node is down.

#PBS -l nodes=24:ppn=8,walltime=10:00
#PBS -N test
#PBS -j oe
#PBS -q default
#PBS -S /bin/bash

set -x

# Construct a copy of the hostfile with only 1 entry per node.
# MPI can use this to run 1 task on each node.
uniq "$PBS_NODEFILE"|awk '{for(i=0;i<1;i+=1) print}'>nodefile.1way
ntasks=$(wc -l < nodefile.1way)

# Run 1-way on 24 nodes
# Substitute your application name for "hostname -v" if desired
mpiexec -np $ntasks --hostfile nodefile.1way --prefix /opt/openmpi hostname -v

In this script, the for-loop can be modified to run 2 tasks per node, i<2 would run 2 per node, etc.

Running a job on a specific nodes


#PBS -l walltime=00:03:00,host=compute-2-9+compute-2-10+compute-2-11,nodes=3
#PBS -N testnodes
#PBS -j oe
#PBS -q default

set -x
echo "Run my job."

Cluster Tips

  • Monitor with ganglia Ganglia
  • For individual job/node monitoring use ganglia or type: top [press '1' to view all cpu's]
  • Use the local /tmp rather than the nfs-mounted home directory!