Difference between revisions of "ATLAS2 Cluster"

From CAC Documentation wiki
Jump to navigation Jump to search
(fixed bad comment in example script)
 
(81 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== ATLAS2 General Information ==
+
== Getting Started ==
:* ATLAS2 is a private cluster with restricted access to the bs54_0001 group and is currently available for *TESTING ONLY*
 
:* ATLAS2 currently has one head node (atlas2.cac.cornell.edu) and 9 compute nodes [c00[06,10-12,15-16,34,44,46] (will have the full 56 compute nodes upon completion)
 
  
:* Head node:  '''atlas2.cac.cornell.edu'''  (access via ssh)
+
=== General Information ===
:** Open HPC deployment running Centos 7.3.1611
 
:** Cluster scheduler: slurm 16.05
 
:** /home (15TB) directory server (nfs exported to all cluster nodes)
 
  
:* Cluster Status: [http://atlas2.cac.cornell.edu/ganglia/ Ganglia].
+
:* ATLAS2 is a private cluster with restricted access to the bs54_0001 group.
 +
:* Head node:  '''atlas2.cac.cornell.edu'''  ([[#How To Login|access via ssh]])
 +
:** [https://github.com/openhpc/ohpc/wiki OpenHPC] deployment running Centos 7.6
 +
:** Cluster scheduler: [[Slurm]] 17.11.10
 +
:** <code>/home</code> 15TB directory server (nfs exported to all cluster nodes)
 +
:** Intel(R) Xeon(R) E5-2637 v4 @ 3.5GHz; supports vector extensions up to AVX2
 +
:* 55 compute nodes [[#Hardware|c00[01-16, 31-48,50-70]]]
 +
:* Current Cluster Status: [http://atlas2.cac.cornell.edu/ganglia/ Ganglia].
 
:* Please send any questions and report problems to:  [mailto:cac-help@cornell.edu cac-help@cornell.edu]  
 
:* Please send any questions and report problems to:  [mailto:cac-help@cornell.edu cac-help@cornell.edu]  
  
====Getting Started with the ATLAS cluster====
+
=== How To Login ===
  
==Queues/Partitions ("Partition" is the term used by slurm) ==
+
:* To get started, login to the head node <tt>atlas2.cac.cornell.edu</tt> via ssh.
 +
:* If you are unfamiliar with Linux and ssh, we suggest reading the [[Linux Tutorial]] and looking into how to [[Connect to Linux]] before proceeding.
 +
:* You will be prompted for your [https://www.cac.cornell.edu/services/myacct.aspx CAC account] password
  
ATLAS2 has 5 separate queues:  [NOTE: ABOVE COMMENT THAT THIS CLUSTER CURRENTLY IS RUNNING WITH ONLY 8 COMPUTE NODES)
+
=== Hardware ===
WE NEED USERS TESTING!
 
  
:*'''short''' (default)
+
'''All nodes''' have hyperthreading turned on and are Xeon generations that supports vector extensions: SSE4.2.
::: Number of nodes: 28 servers (total: 56 cpu, 336 cores)
 
::: Node Names: c00[17-44] **CURRENTLY ONLY c0034 & c0044 are up for testing of the short queue! **
 
::: Memory per node: 48GB
 
::: /tmp per node: 409GB
 
::: Limits: Maximum of 28 nodes, walltime limit: 4 hours
 
  
:*'''long'''
+
:{| class="wikitable" border="1" cellpadding="5" style="width: auto"
::: Number of nodes: 18 servers (total: 36 cpu, 216 cores)
+
! style="background:#e9e9e9;" | Node Names
::: Node Names: c00[17-34] **CURRENTLY ONLY c0034 is up for testing of the long queue! **
+
! style="background:#e9e9e9;" | Memory per node
::: Memory per node: 48GB
+
! style="background:#e9e9e9;" | Model name
::: /tmp per node: 409GB
+
! style="background:#e9e9e9;" | Processor count per node
::: Limits: Maximum of 18 nodes, walltime limit: 504 hours
+
! style="background:#e9e9e9;" | Core(s) per socket
 +
! style="background:#e9e9e9;" | Sockets
 +
! style="background:#e9e9e9;" | Thread(s) per core
 +
|-
 +
| c00[01-12]
 +
| align="center" | 94GB
 +
| align="center" | Intel(R) Xeon(R) CPU X5690  @ 3.47GHz
 +
| align="center" | 24
 +
| align="center" | 6
 +
| align="center" | 2
 +
| align="center" | 2
 +
|-
 +
| c00[13-16]
 +
| align="center" | 47GB
 +
| align="center" | Intel(R) Xeon(R) CPU X5670  @ 2.93GHz
 +
| align="center" | 24
 +
| align="center" | 6
 +
| align="center" | 2
 +
| align="center" | 2
 +
|-
 +
| c00[31-48,50-58]
 +
| align="center" | 47GB
 +
| align="center" | Intel(R) Xeon(R) CPU X5670  @ 2.93GHz
 +
| align="center" | 24
 +
| align="center" | 6
 +
| align="center" | 2
 +
| align="center" | 2
 +
|-
 +
| c00[59-70]
 +
| align="center" | 47GB
 +
| align="center" | Intel(R) Xeon(R) CPU X5690 @ 3.47GHz
 +
| align="center" | 24
 +
| align="center" | 6
 +
| align="center" | 2
 +
| align="center" | 2
 +
|-
 +
|}
 +
 
 +
=== Networking ===
 +
:* All nodes have a 1GB ethernet connection for eth0 on a private net served out from the atlas2 head node.
 +
:* All nodes have an Infiniband connection:
 +
:** InfiniPath_QLE7340n  (QDR speed, 8Gbits/sec)
 +
 
 +
== Running Jobs with Slurm ==
 +
 
 +
'''For detailed information and a quick-start guide, see the [[Slurm]] page.'''
 +
 
 +
=== ATLAS2 Queues/Partitions  ===
 +
 
 +
("Partition" is the term used by Slurm)
 +
 
 +
:* '''hyperthreading is turned on for ALL nodes'''
 +
:* '''all partitions have a default time of 1 hour'''
 +
:* ATLAS2 has 5 separate queues:
 +
 
 +
:{| class="wikitable" border="1" cellpadding="4" style="width: auto"
 +
! style="background:#e9e9e9;" | Queue/Partition
 +
! style="background:#e9e9e9;" | Number of nodes
 +
! style="background:#e9e9e9;" | Node Names
 +
! style="background:#e9e9e9;" | Limits
 +
|-
 +
| '''short''' (default)
 +
| align="center" | 31
 +
| align="center" | c00[13-16,31-48,50-58]
 +
| align="center" | walltime limit: 4 hours
 +
|-
 +
| '''long'''
 +
| align="center" | 22
 +
| align="center" | c00[13-16,31-48]
 +
| align="center" | walltime limit: 504 hours
 +
|-
 +
| '''inter''' ~Interactive
 +
| align="center" | 12
 +
| align="center" | c00[59-70]
 +
| align="center" | walltime limit: 168 hours
 +
|-
 +
| '''bigmem'''
 +
| align="center" | 12 servers
 +
| align="center" | c00[01-12]   
 +
| align="center" | Maximum of 12 nodes, walltime limit: 168 hours 
 +
|-
 +
| '''normal'''
 +
| align="center" | 55 servers
 +
| align="center" | c00[01-16, 31-48,50-70] 
 +
| align="center" | walltime limit: 4 hours
 +
|-
 +
|}
 +
 
 +
=== Example in Short Partition/Queue ===
 +
 
 +
Example sbatch file to run a job in the short partition/queue; save as example.sh:
 +
 
 +
<pre>
 +
#!/bin/bash
 +
## J sets the name of job
 +
#SBATCH -J TestJob
 +
 
 +
## -p sets the partition (queue)
 +
#SBATCH -p long
 +
 
 +
## 10 min
 +
#SBATCH --time=00:10:00
 +
 
 +
## sets the tasks per core (default=2; keep default if you want to take advantage of hyperthreading)
 +
## 2 will take whole cores, but will divide by 2 with hyperthreading
 +
#SBATCH --ntasks-per-core=1
  
:*'''inter''' ~Interactive
+
## request 4GB per CPU (may limit # of tasks, depending on total memory)
::: Number of nodes: 12 servers (total: 24 cpu, 144 cores)
+
#SBATCH --mem-per-cpu=4GB
::: Node Names: c00[45-56] .  **CURRENTLY ONLY c0046 is up for testing of the inter queue! **
 
::: Memory per node: 48GB
 
::: /tmp per node: 68GB
 
::: Limits: Maximum of 12 nodes, walltime limit: 168 hours
 
  
:*'''bigmem'''
+
## define jobs stdout file
::: Number of Nodes 12 servers (total: 24 cpu, 144 cores)
+
#SBATCH -o testlong-%j.out
::: Node Names: c00[01-12] .  **CURRENTLY ONLY c00[06,10,11,12] are up for testing of the bigmem queue! **
 
::: HW: Intel x5690 3.46GHZ, 128GB SSD drive,
 
::: Memory per node: 96GB
 
::: /tmp per node: 68GB
 
::: Limits: Maximum of 12 nodes, walltime limit: 168 hours 
 
  
:*'''gpu'''
+
## define jobs stderr file
::: Number of nodes: 4 servers (total: 8 cpu, 48 cores)
+
#SBATCH -e testlong-%j.err
::: Node Names: c00[13-16]  **CURRENTLY ONLY c0015 & c0016 are up for testing of the gpu queue! **
 
::: HW: 2xIntel x5670 2.93GHZ, 500GB SATA drive, 1 Tesla M2090
 
::: Memory per node: 48GB
 
::: /tmp per node: 409GB
 
::: Limits: Maximum of 4 nodes, walltime limit: 168 hours
 
  
 +
echo "starting at `date` on `hostname`"
  
===Common Slurm Commands===
+
# Print the SLURM job ID.
[https://slurm.schedmd.com/quickstart.html Slurm Quickstart Guide]
+
echo "SLURM_JOBID=$SLURM_JOBID"
 +
 
 +
echo "hello world `hostname`"
 +
 
 +
echo "ended at `date` on `hostname`"
 +
exit 0
  
[https://slurm.schedmd.com/pdfs/summary.pdf Command/Option Summary (two page PDF)]
+
</pre>
  
==Software==
+
Submit/Run your job:
* Openhpc uses yum to install available software from the installed repositories.
 
** To view all options of yum, type: man yum
 
** To view installed repositories, type: yum repolist
 
** To view if your requested software is in one of the installed repositories, use: yum search
 
** i.e. To search whether variations of tau are available, you would type:
 
 
<pre>
 
<pre>
yum search tau
+
sbatch example.sh
 
</pre>
 
</pre>
* If the package(s) are listed, you can send a request to cac-help@cornell.edu to have the software installed system-wide; you cannot install this yourself.
 
  
=== Installed Software ===
+
View your job:
 
<pre>
 
<pre>
The  'lmod module' system is implemented. Use: module avail
+
scontrol show job 9
to list what environment you can put yourself in for a software version.
+
</pre>
    (to get a more complete listing, type: module spider)
 
  
EXAMPLE:
+
==Software==
To be sure you are using the environment setup for gdal2, you would type:
 
* module avail
 
* module load gdal2
 
- when done, either logout and log back in or type:
 
* module unload gdal2
 
  
You can create your own modules and place them in your $HOME.
+
The cluster is managed with [https://github.com/openhpc/ohpc/wiki OpenHPC], which uses [https://en.wikipedia.org/wiki/Yum_(software) yum] to install available software from the installed repositories.
Once created, type:
+
:* To view all options of yum, type: <code>man yum</code>
module use $HOME/path/to/personal/modulefiles
+
:* To view installed repositories, type: <code>yum repolist</code>
This will prepend the path to $MODULEPATH
+
:* To view if your requested software package is in one of the installed repositories, use: <code>yum search <package></code>
[type echo $MODULEPATH to confirm]
+
:* i.e. To search whether variations of tau are available, you would type:
 +
<pre>
 +
yum search tau
 
</pre>
 
</pre>
  
----
+
=== Installed Software ===
  
 
:{| class="sortable wikitable" border="1" cellpadding="4" style="width: auto"
 
:{| class="sortable wikitable" border="1" cellpadding="4" style="width: auto"
Line 105: Line 190:
 
  | align="center" | cplex/12.8
 
  | align="center" | cplex/12.8
 
  | align="center" |  
 
  | align="center" |  
 +
|-
 +
| cuda toolkit 9.0
 +
| /opt/ohpc/pub/cuda-9.0
 +
| align="center" |
 +
| cudnn 9.0 in targets/x86_64-linux/lib/
 +
|-
 +
| cuda toolkit 9.1
 +
| /opt/ohpc/pub/cuda-9.1
 +
| align="center" |
 +
| cudnn 9.1 in targets/x86_64-linux/lib/
 +
|-
 +
| cuda toolkit 9.2
 +
| /opt/ohpc/pub/cuda-9.2
 +
| align="center" |
 +
| cudnn 9.2 in targets/x86_64-linux/lib/
 
  |-
 
  |-
  | cuda toolkit 8-0
+
  | cuda toolkit 10.0
  | /usr/local/cuda-8.0
+
  | /opt/ohpc/pub/cuda-10.0
 
  | align="center" |  
 
  | align="center" |  
  | align="center" | c0015 & c0016 (gpus)
+
  | cudnn 7.4.1 for cuda10 in targets/x86_64-linux/lib/
 
  |-
 
  |-
 
  | gcc 7.2.0  
 
  | gcc 7.2.0  
Line 136: Line 236:
 
  | align="center" |  The system-wide installation of packages is no longer supported. See below for Anaconda/miniconda install information.
 
  | align="center" |  The system-wide installation of packages is no longer supported. See below for Anaconda/miniconda install information.
 
  |-
 
  |-
  | R 3.4.3
+
  | R 3.5.1
 
  | /usr/bin/R
 
  | /usr/bin/R
 
  | align="center" |  
 
  | align="center" |  
Line 150: Line 250:
 
:* It is usually possible to install software in your home directory.
 
:* It is usually possible to install software in your home directory.
 
:* List installed software via rpms: ''''rpm -qa''''. Use grep to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]
 
:* List installed software via rpms: ''''rpm -qa''''. Use grep to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]
 +
 +
=== Modules ===
 +
 +
Since this cluster is managed with OpenHPC, the [[Modules (Lmod)| Lmod Module System]] is implemented.  You can see detailed information and instructions at the linked page. 
 +
 +
'''Example:'''
 +
To be sure you are using the environment setup for <code>cplex</code>, you would type:
 +
<pre>
 +
$ module avail
 +
$ module load cplex
 +
</pre>
 +
When done, either logout and log back in or type <code>module unload cplex</code>
 +
 +
You can also ''create your own modules'' and place them in your $HOME.  For instructions, see the [[Modules_(Lmod)#Personal_Modulefiles| Modules (Lmod)]] page.
 +
 +
Once created, type <code>module use $HOME/path/to/personal/modulefiles</code>.  This will prepend the path to <code>$MODULEPATH</code>.  Type <code>echo $MODULEPATH</code> to confirm.
  
 
=== Build software from source into your home directory ($HOME) ===
 
=== Build software from source into your home directory ($HOME) ===
Line 165: Line 281:
 
* Reload the .bashrc file with source ~/.bashrc. (or logout and log back in)
 
* Reload the .bashrc file with source ~/.bashrc. (or logout and log back in)
 
</pre>
 
</pre>
 
 
 
===Slurm HELP===
 
Slurm Workload Manager [https://slurm.schedmd.com/quickstart.html Quick Start User Guide] - this page lists all of the available Slurm commands
 
 
Slurm Workload Manager [https://slurm.schedmd.com/faq.html Frequently Asked Questions] includes FAQs for Management, Users and Administrators
 
 
[https://rc.fas.harvard.edu/resources/documentation/convenient-slurm-commands/ Convenient SLURM Commands] has examples for getting information on jobs and controlling jobs
 
 
Slurm Workload Manager - [https://slurm.schedmd.com/sbatch.html sbatch] - used to submit a job script for later execution. The script will typically contain one or more srun commands to launch parallel tasks.
 
<pre>
 
A few slurm commands to initially get familiar with:
 
scontrol show nodes
 
scontrol show partition
 
 
Submit a job: sbatch testjob.sh
 
Interactive Job: srun -p short --pty /bin/bash
 
 
scontrol show job [job id]
 
scancel [job id]
 
sinfo -l
 
</pre>
 
 
===Running jobs on the ATLAS2 cluster===
 
{{:Running Jobs on the ATLAS2 cluster}}
 
 
 
 
Example sbatch file to run a job in the short partition/queue; save as example.sh:
 
 
<pre>
 
#!/bin/bash
 
## J sets the name of job
 
#SBATCH -J TestJob
 
## -p sets the partition (queue)
 
#SBATCH -p long
 
## 10 min
 
#SBATCH --time=00:10:00
 
## request a single task(core)
 
#SBATCH -n1
 
## request 300MB per core
 
## #SBATCH --mem-per-cpu=300
 
## define jobs stdout file
 
#SBATCH -o testlong-%j.out
 
## define jobs stderr file
 
#SBATCH -e testlong-%j.err
 
 
echo "starting at `date` on `hostname`"
 
 
# Print the SLURM job ID.
 
echo "SLURM_JOBID=$SLURM_JOBID"
 
 
echo "hello world `hostname`"
 
 
echo "ended at `date` on `hostname`"
 
exit 0
 
 
</pre>
 
 
Then run:
 
<pre>
 
sbatch example.sh
 
</pre>
 
 
Then submit:
 
 
<pre>
 
scontrol show job 9
 
</pre>
 
 
You should see the node it ran on and that it was run in the short partition/queue.
 
 
===MATLAB: running MDCS jobs on the ATLAS2 cluster===
 
{{:Running MDCS Jobs on the ATLAS2 cluster}}
 
  
 
===How to Install R packages in your home directory ===
 
===How to Install R packages in your home directory ===
Line 303: Line 344:
 
* Please take the tutorials to assist you with your management of conda packages:
 
* Please take the tutorials to assist you with your management of conda packages:
 
https://conda.io/docs/user-guide/tutorials/index.html
 
https://conda.io/docs/user-guide/tutorials/index.html
 +
 +
===MPI  ===
 +
* To use MPI with Infiniband, use openmpi3
 +
* The mpich default transport is TCP/IP (not ininiband)
 +
* NOTE: mvapich2/2.2 will NOT work on this cluster

Latest revision as of 20:41, 21 January 2020

Getting Started

General Information

  • ATLAS2 is a private cluster with restricted access to the bs54_0001 group.
  • Head node: atlas2.cac.cornell.edu (access via ssh)
    • OpenHPC deployment running Centos 7.6
    • Cluster scheduler: Slurm 17.11.10
    • /home 15TB directory server (nfs exported to all cluster nodes)
    • Intel(R) Xeon(R) E5-2637 v4 @ 3.5GHz; supports vector extensions up to AVX2
  • 55 compute nodes c00[01-16, 31-48,50-70]
  • Current Cluster Status: Ganglia.
  • Please send any questions and report problems to: cac-help@cornell.edu

How To Login

  • To get started, login to the head node atlas2.cac.cornell.edu via ssh.
  • If you are unfamiliar with Linux and ssh, we suggest reading the Linux Tutorial and looking into how to Connect to Linux before proceeding.
  • You will be prompted for your CAC account password

Hardware

All nodes have hyperthreading turned on and are Xeon generations that supports vector extensions: SSE4.2.

Node Names Memory per node Model name Processor count per node Core(s) per socket Sockets Thread(s) per core
c00[01-12] 94GB Intel(R) Xeon(R) CPU X5690 @ 3.47GHz 24 6 2 2
c00[13-16] 47GB Intel(R) Xeon(R) CPU X5670 @ 2.93GHz 24 6 2 2
c00[31-48,50-58] 47GB Intel(R) Xeon(R) CPU X5670 @ 2.93GHz 24 6 2 2
c00[59-70] 47GB Intel(R) Xeon(R) CPU X5690 @ 3.47GHz 24 6 2 2

Networking

  • All nodes have a 1GB ethernet connection for eth0 on a private net served out from the atlas2 head node.
  • All nodes have an Infiniband connection:
    • InfiniPath_QLE7340n (QDR speed, 8Gbits/sec)

Running Jobs with Slurm

For detailed information and a quick-start guide, see the Slurm page.

ATLAS2 Queues/Partitions

("Partition" is the term used by Slurm)

  • hyperthreading is turned on for ALL nodes
  • all partitions have a default time of 1 hour
  • ATLAS2 has 5 separate queues:
Queue/Partition Number of nodes Node Names Limits
short (default) 31 c00[13-16,31-48,50-58] walltime limit: 4 hours
long 22 c00[13-16,31-48] walltime limit: 504 hours
inter ~Interactive 12 c00[59-70] walltime limit: 168 hours
bigmem 12 servers c00[01-12]   Maximum of 12 nodes, walltime limit: 168 hours
normal 55 servers c00[01-16, 31-48,50-70] walltime limit: 4 hours

Example in Short Partition/Queue

Example sbatch file to run a job in the short partition/queue; save as example.sh:

#!/bin/bash
## J sets the name of job
#SBATCH -J TestJob

## -p sets the partition (queue)
#SBATCH -p long 

## 10 min
#SBATCH --time=00:10:00

## sets the tasks per core (default=2; keep default if you want to take advantage of hyperthreading)
## 2 will take whole cores, but will divide by 2 with hyperthreading
#SBATCH --ntasks-per-core=1 

## request 4GB per CPU (may limit # of tasks, depending on total memory)
#SBATCH --mem-per-cpu=4GB

## define jobs stdout file
#SBATCH -o testlong-%j.out

## define jobs stderr file
#SBATCH -e testlong-%j.err

echo "starting at `date` on `hostname`"

# Print the SLURM job ID.
echo "SLURM_JOBID=$SLURM_JOBID"

echo "hello world `hostname`"

echo "ended at `date` on `hostname`"
exit 0

Submit/Run your job:

sbatch example.sh

View your job:

scontrol show job 9

Software

The cluster is managed with OpenHPC, which uses yum to install available software from the installed repositories.

  • To view all options of yum, type: man yum
  • To view installed repositories, type: yum repolist
  • To view if your requested software package is in one of the installed repositories, use: yum search <package>
  • i.e. To search whether variations of tau are available, you would type:
 yum search tau

Installed Software

(sortable table)
Package and Version Location module available Notes
cplex studio 128 /opt/ohpc/pub/ibm/ILOG/CPLEX_Studio128/ cplex/12.8
cuda toolkit 9.0 /opt/ohpc/pub/cuda-9.0 cudnn 9.0 in targets/x86_64-linux/lib/
cuda toolkit 9.1 /opt/ohpc/pub/cuda-9.1 cudnn 9.1 in targets/x86_64-linux/lib/
cuda toolkit 9.2 /opt/ohpc/pub/cuda-9.2 cudnn 9.2 in targets/x86_64-linux/lib/
cuda toolkit 10.0 /opt/ohpc/pub/cuda-10.0 cudnn 7.4.1 for cuda10 in targets/x86_64-linux/lib/
gcc 7.2.0 /opt/ohpc/pub/compiler/gcc/7.2.0/bin/gcc gnu7/7.2.0
gcc 4.8.5 (default) /usr/bin/gcc
gdal 2.2.3 /opt/ohpc/pub/gdal2.2.3 gdal/2.2.3
java openjdk 1.8.0 /usr/bin/java
Python 2.7.5 (default) /usr/bin/python The system-wide installation of packages is no longer supported. See below for Anaconda/miniconda install information.
R 3.5.1 /usr/bin/R The system-wide installation of packages is no longer supported.
Subversion (svn) 1.7 /usr/bin/svn
  • It is usually possible to install software in your home directory.
  • List installed software via rpms: 'rpm -qa'. Use grep to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]

Modules

Since this cluster is managed with OpenHPC, the Lmod Module System is implemented. You can see detailed information and instructions at the linked page.

Example: To be sure you are using the environment setup for cplex, you would type:

$ module avail
$ module load cplex

When done, either logout and log back in or type module unload cplex

You can also create your own modules and place them in your $HOME. For instructions, see the Modules (Lmod) page.

Once created, type module use $HOME/path/to/personal/modulefiles. This will prepend the path to $MODULEPATH. Type echo $MODULEPATH to confirm.

Build software from source into your home directory ($HOME)

* download and extract your source
* cd to your extracted source directory
* ./configure --./configure --prefix=$HOME/appdir
[You need to refer to your source documentation to get the full list of options you can provide 'configure' with.]
* make
* make install

The binary would then be located in ~/appdir/bin. 
* Add the following to your $HOME/.bashrc: 
      export PATH="$HOME/appdir/bin:$PATH"
* Reload the .bashrc file with source ~/.bashrc. (or logout and log back in)

How to Install R packages in your home directory

Reference: http://cran.r-project.org/doc/manuals/R-admin.html#Managing-libraries

************************************************************************************
NOTE: Steps 1) through 4) need to be done once, after your Rlibs directory
has been created and your R_LIBS environment is set, you can install additional 
packages using step 5).
************************************************************************************

Know your R library search path:
    Start R and run .libPaths()  Sample output is shown below:
    > .libPaths()
     [1] "/usr/lib64/R/library"

Now we will create a local Rlibs directory and add this to the library search path.
NOTE: Make sure R is NOT running before you proceed.

1) Create a directory in your home directory you would like to install the R packages, e.g. Rlibs 
mkdir  ~/Rlibs

2) Create a .profile file in your home directory (or modify existing) using your favorite editor (emacs, vim, nano, etc)  
   
     Add the following to your .profile
     #!/bin/sh
     if [ -n $R_LIBS ]; then
        export R_LIBS=~/Rlibs:$R_LIBS
     else
        export R_LIBS=~/Rlibs
     fi

3) To reset the R_LIBS path we need to run the following: "source ~/.profile" (or logout and log back in) 

4) Confirm the change is in your library path:
     start R
> .libPaths()
[1] "$HOME/Rlibs"     
[2] "/usr/lib64/R/library"   

  
5) Install the package in your local directory 
>install.packages("packagename","~/Rlibs","https://cran.r-project.org/")
i.e. to install the package:snow
>install.packages("snow","~/Rlibs","https://cran.r-project.org/")

6) For more help with install.packages() use
>?install.packages( )  

7) To see which libraries are available in your R library path, run library() 
The output will show your local packages and the system wide packages
>library()

How to Install Python Anaconda (miniconda) home directory

https://conda.io/docs/user-guide/tutorials/index.html

MPI

  • To use MPI with Infiniband, use openmpi3
  • The mpich default transport is TCP/IP (not ininiband)
  • NOTE: mvapich2/2.2 will NOT work on this cluster