View source for HD Human Neuroscience Institute (HD-HNI) Computing

== Welcome to the User Documentation for the HD Human Neuroscience Institute Computing Environment ==

== IMPORTANT ANNOUNCEMENT - January 29, 2018 ==
=== '''TEMPORARY Instructions for mounting HD-HNI storage''' ===
The HD-HNI researchers have purchased new disks to store the research data.  In order to do this, CAC staff moved the existing storage to a temporary file server.  
To mount the existing storage please follow the instructions in this document but '''replace the storage server as follows''':

:* replace  \\hd-hni-fs.cac.cornell.edu\<userid>   with '''\\storage01.cac.cornell.edu\hd-hni-home\<userid>'''                 
:* or
:* replace: \\hd-hni-fs.cac.cornell.edu\reyna-fmri  with '''\\storage01.cac.cornell.edu\reyna-fmri'''  (other shared folders are aclab-fmri, brainerd-fmri, HD4630, and spreng-fmri).
== IMPORTANT ANNOUNCEMENT - January 29, 2018 ==


==The HD-HNI computing environment consists of:==
   
   1 Head Node - '''hd-hni.cac.cornell.edu''' - Dell R620 server 
   1 Batch Node - compute-1-1.hd-hni - Dell R820 - 32 cores and 256GB RAM
   1 Interactive Node - '''hd-hni-interactive-1-1.cac.cornell.edu''' - Dell R820 - 32 cores and 256GB RAM
     alias: '''inter1.cac.cornell.edu'''  (easier to type than the full name!)
   1 Cluster file server - hd-hni-fs.cac.cornell.edu - serving 22.5TB of storage capacity
   1 Force10 switch -  connects everything together with 10GB ethernet!

==Installed Software==

{| class="sortable wikitable" border="1" cellpadding="4" style="width: auto"
 |+ (sortable table)<br/>
 ! style="background:#e9e9e9;" | Package and Version
 ! style="background:#e9e9e9;" | Location
 ! style="background:#e9e9e9;" | hd-hni.cac.cornell.edu
 ! style="background:#e9e9e9;" | Compute Node
 ! style="background:#e9e9e9;" | Interactive Node 
 ! style="background:#e9e9e9;" | Notes
 |-
 | Python 2.6.6
 | /usr/bin/python
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
|-
 | Python 2.7.6
 | /usr/local/bin/python (points to /opt/python)
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | to load env variables, type: module load python-2.7.6
 |-
 | PyMVPA2
 | /usr/bin
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 |Matlab R2012b
 | /opt/MATLAB/R2012b
 | align="center" | yes
 | align="center" | no
 | align="center" | no
 | align="center" | 
 |-
 |Matlab R2012b Runtimes
 | /opt/MATLAB/MATLAB_Compiler_Runtime/v80
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 |Matlab R2012b Toolbox: plsgui
 | /opt/MATLAB/R2012b/toolbox
 | align="center" | yes
 | align="center" | no
 | align="center" | no
 | align="center" | 
 |-
 |Matlab R2012b Toolbox: spm5, spm8 spm12
 | /opt/MATLAB/R2012b/toolbox
 | align="center" | yes
 | align="center" | yes 
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2013a (default)
 | /opt/MATLAB/R2013a
 | align="center" | yes
 | align="center" | no
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2013a Runtimes
 | /opt/MATLAB/MATLAB_Compiler_Runtime/v81
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2013a Toolbox: gift
 | /opt/matlab/R2013a/toolbox/GroupICATv3.0a/icatb
 | align="center" | yes
 | align="center" | no
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2013a Toolbox: mvpa
 | /opt/MATLAB/R2013a/toolbox
 | align="center" | yes
 | align="center" | no
 | align="center" | yes
 | align="center" | Princeton
 |-
 | Matlab R2013a Toolbox: plsgui
 | /opt/MATLAB/R2013a/toolbox
 | align="center" | yes
 | align="center" | no
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2013a Toolbox: spm5, spm8, spm12
 | /opt/MATLAB/R2013a/toolbox
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 |Matlab R2013a Toolbox: vbm8
 | /opt/MATLAB/R2013a/toolbox/spm8/toolbox
 | align="center" | yes
 | align="center" | no
 | align="center" | Yes
 | align="center" | 
 |-
 |Matlab R2013a Toolbox: xjview
 | /opt/MATLAB/R2013a/toolbox/spm8/toolbox
 | align="center" | yes
 | align="center" | no
 | align="center" | Yes
 | align="center" | 
 |-
 | Matlab R2014a 
 | /opt/MATLAB/R2014a
 | align="center" | no
 | align="center" | no
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2014a Runtimes
 | /opt/MATLAB/MATLAB_Compiler_Runtime/v83
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2014a Toolbox: spm5, spm8, spm12
 | /opt/MATLAB/R2013a/toolbox
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2016a 
 | /opt/MATLAB/R2016a
 | align="center" | yes
 | align="center" | no
 | align="center" | no
 | align="center" | 
 |-
 | Matlab R2016a Runtimes
 | /opt/MATLAB//MATLAB_Compiler_Runtime/v901
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | Matlab R2016a Toolbox: spm5, spm8, spm12
 | /opt/MATLAB/R2013a/toolbox
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | fsl
 | /opt/fsl
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | freesurfer
 | /opt/freesurfer
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | setup environment, type: source $FREESURFER_HOME/SetUpFreeSurfer.sh
 |-
 | afni
 | /opt/abin
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | meica
 | /opt/meica
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | R 3.2.2
 | /usr/bin/R
 | align="center" | yes
 | align="center" | yes
 | align="center" | yes
 | align="center" | 
 |-
 | xnat
 | http://hd-hni-xnat.cac.cornell.edu/
 | align="center" |  
 | align="center" |  
 | align="center" |  
 | align="center" | 
 |
 |}

:* It is usually possible to install software in your home directory; you can also request through your helpdesk to have the CAC install software, pending the approval from the PI.
:* You can view a list of installed software with: ''''rpm -qa''''. Use the grep option to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]

==Who can access the HD-HNI computing environment?==
:*Users should request access to the HD-HNI project by contacting [mailto:amh352@cornell.edu Allison Hermann].
:*When added, users will receive a welcome letter with their username and initial CAC HD-HNI password.

==What servers can users access and how?==
HD-HNI users can '''directly access two nodes''': the cluster head node and the interactive node.

:*How to [[Connect to Linux]] servers?
:*HD-HNI users submit "jobs" to the [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]].

==First Login==
 The first thing to do is login to the cluster head node and change your password.
 This will require an ssh client.
 How to [[Connect to Linux]] server (head node)?
 If you do not have a ssh client, contact your local [http://www.human.cornell.edu/helpdesk helpdesk].

[https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started#Managing_your_password Managing Your Password]

 After you change your password, you will be prompted for an ssh pass phrase; leave this blank, simply hit the Enter key.
 You are now ready to begin using the HD-HNI computing environment.

==HD-HNI File Server - Moving data from your workstation to the shared file server==

A central file server, hd-hni-fs.cac.cornell.edu, serves all HD-HNI user home directories. '''Users can not ssh to this server but can connect and access files in a variety of other ways outlined below'''.  

Note: by default, your home directory and its contents will be readable and executable by all other users of the HD-HNI systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.

===Attach your home directory to your local computer=== 
You can "mount" your home directory from computers on the Cornell network or if off campus, computers connected to Cornell VPN.
Once your home directory is mounted, you can drag and drop files to it.

====Windows Users - how to mount HD-HNI directories====

'''NOTE: WE RECOMMEND THAT YOU DO NOT SAVE YOUR PASSWORD ON PUBLIC COMPUTERS, DOING SO MAY CAUSE CONNECTION TROUBLES'''
:* Open My Computer
:* Click on Tools -> Map Network Drive
:* Drive H: (if you are already using this drive letter, use another letter)
:* Folder: \\hd-hni-fs.cac.cornell.edu\<userid>                    
:* or
:* Folder: \\hd-hni-fs.cac.cornell.edu\reyna-fmri                  
:* Then:
:** Select "Connect using a different user name:". This will allow you to enter the CAC domain and your HD-HNI userid at CAC, rather than those associated with your own machine.
:** User name: '''''CTC_ITH\your_userid'''''
:** Password: your CAC HD-HNI password
:* Troubleshooting: If you have already mapped the drive and subsequently have problems, disconnect the drive and remap it.

====MacOS X Users - how to mount HD-HNI directories====

'''NOTE: WE RECOMMEND THAT YOU DO NOT SAVE YOUR PASSWORD ON PUBLIC COMPUTERS, DOING SO MAY CAUSE CONNECTION TROUBLES'''
# In the Finder, select '''''Connect to Server...''''' from the '''''Go''''' menu. <br /><br />[[Image:FileAccess1.jpg]]
# Enter '''''smb://hd-hni-fs.cac.cornell.edu/<user name>''''' in the Server Address field. You may need to use '''smb://<username>@hd-hni-fs.cac.cornell.edu/<username>'''. 
# Enter your CAC user name and password to log in.

====Linux Users - how to mount HD-HNI directories====

You cannot mount the HD-HNI directories vis NFS for security reasons. To mount it as a CIFS drive, you need to be root, which often means using the '''sudo''' command. Then execute
  mount -t cifs //hd-hni-fs.cac.cornell.edu/<user name> /mnt/test -o user=<user name>,workgroup=ctc_ith,vers=3.0

where <username> is your username, and '''/mount/point''' is the name of a directory you have already created on your local filesystem. Enter the password for your CAC project when prompted. See '''''man mount.cifs''''' for available options for the mount command

If you see errors, such as "missing codepage or helper program," then you have not installed the mount and umount packages for CIFS on your local machine. If problems persist, send your initial command and the results of '''dmesg | tail'''.

==File Transfers To The HD-HNI file server==
Users can transfer data to the HD-HNI-FS file server without mounting the file systems locally. 

===File Transfers between File Server and your Linux or MAC workstation===
====Secure Copy====
Secure copy is a standard tool to copy files to and from remote hosts.
 This example copies a file named "localfile.dat" on your local workstation to a file named remoteinput.dat on the remote file server:
 localhost$ scp localfile.dat username@hd-hni-fs.cac.cornell.edu:remoteinput.dat
 
 This example copies the file named results.dat on the remote server to a file named localresults.dat on your local workstation.
 localhost$ scp username@hd-hni-fs.cac.cornell.edu:results.dat localresults.dat 
====Secure FTP====
FTP is disabled for security reasons, but sftp's interface is nearly identical.
 localhost$sftp username@hd-hni-fs.cac.cornell.edu
     <enter your username's password when prompted>
 sftp> put localresults.dat results.dat
 sftp> quit
====Samba Client====
Type
 smbclient //hd-hni-fs.cac.cornell.edu/<user name> -U <user name>
Enter the password for your HD-HNI account when prompted
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and your HD-HNI home directory similar to ftp client. Type '''''help''''' for more instructions.
 -sh-3.2$ smbclient //hd-hni-fs.cac.cornell.edu/<user name> -U <user name>
     Password: 
     Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]
     smb: \> help

===File Transfers between the File Server and your Windows Workstation===
====Secure Copy====
The people who make Putty provide a secure copy client called [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. 
<pre>
 This example copies local workstation file to the remote file server:
  cmd> pscp localfile.dat username@hd-hni-fs.cac.cornell.edu:remoteinput.dat
     <enter your username's password when prompted>

 This example copies a file on the remote file server to the local workstation:
  cmd> pscp username@hd-hni-fs.cac.cornell.edu:results.dat localresults.dat
</pre>

====Secure FTP====
FTP is disabled for security reasons, but [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:
 cmd> psftp username@hd-hni-fs.cac.cornell.edu
     <enter your username's password when prompted>
 psftp> put localresults.dat results.dat
 psftp> quit

==HD-HNI Data Processing Environment==
There is one [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]] (compute-1-1) and one interactive node (hd-hni-interactive-1-1) for processing HD-HNI data.
===How to access the data processing nodes?===

Submit jobs to the [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]] from the head node or from the interactive node.
ssh directly to the head node and the interactive node.
ssh to the [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]] from the head node only.

====Maui Scheduler and Job submission/monitoring commands====

Jobs are scheduled by the [http://www.adaptivecomputing.com/resources/docs/maui/mauiusers.php Maui scheduler] with the [http://www.adaptivecomputing.com/products/open-source/torque/ Torque resource manager]. We suggest you use a job submission batch file utilizing  [http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/commands/qsub.htm PBS Directives ('Options' section).] 

===== Common Maui Commands===== 
(If you have any experience with PBS/Torque or SGE, [http://www.adaptivecomputing.com/resources/docs/maui/a.gcommandoverview.php Maui Commands] may be recognizable. Most used:

'''[http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/commands/qsub.php qsub]''' - Job submission (jobid will be displayed for the job submitted)
:* $ qsub jobscript.sh

'''[http://www.adaptivecomputing.com/resources/docs/maui/commands/showq.php showq] - Display queue information.'''  

:* $ showq (dump everything)
:* $ showq -r (show running jobs)
:* $ showq -u foo42 (shows foo42's jobs)

'''[http://www.adaptivecomputing.com/resources/docs/maui/commands/checkjob.php checkjob] - Display job information.'''  (You can only checkjob your own jobs.)

:* $ checkjob -A jobid (get dense key-value pair information on job 42)
:* $ checkjob -v jobid (get verbose information on job 42)

'''[http://www.adaptivecomputing.com/resources/docs/maui/commands/canceljob.php canceljob] - Cancel Job.''' (You can only cancel your own jobs.)
:* $ canceljob jobid


====Batch Processing====

Submit jobs to the batch node via the cluster head node or the interactive node. Jobs are scheduled as resources are available.

:*First ssh to the Head node or the Interactive node
<pre>
ssh username@hd-hni.cac.cornell.edu
or
ssh username@inter1.cac.cornell.edu
</pre>

:*Next, using your favorite editor create a job file named test_default_queue.sh (text file of commands) containing the following lines:
<source lang="bash">
#!/bin/bash
#PBS -l walltime=00:05:00,nodes=1
#PBS -j oe
#PBS -N testdefaultqueue 
#PBS -q default

# Turn on echo of shell commands
set -x
# Because jobs start in the HOME directory, move to submit dir 
cd $PBS_O_WORKDIR
echo 'pwd'
echo "PBS_O_WORKDIR is `pwd`"
echo "env is `env`"
# copy your binary that you want to run and any data files to a local directory on node job is executing on
# this example assumes you have a binary file named helloworld.sh in your local bin directory
cp $HOME/bin/helloworld.sh $TMPDIR
cd $TMPDIR
# run the binary file from the local disk on the node the job was placed on
./helloworld.sh >&hello.stdout
# Copy output files to your output folder       
cp -f $TMPDIR/hello.stdout $HOME/output
</source>

:*Submit the test job: 
<pre>
qsub test_default_queue.sh
</pre>

:*Check the status of the job:
<pre>
showq
</pre>

:*Once the job runs the output file will be created with the name you gave it (-N above) with an appended ".o[jobid]".
:To view the output:
<pre>
cat testdefaultqueue.o[jobid]
</pre>

===Matlab - How to?===
The full MATLAB application is installed on the head node and the interactive node. You should use the Batch node (compute-1-1) for production runs of compiled MATLAB codes. Compiled code need the MATLAB runtimes; these are installed on all nodes. For the full application:

:* '''MATLAB R2013a is the default'''. MATLAB R2012b and R2014a are also available.
:** The head node has releases R2013a and R2012b installed.
:** The interactive node has releases R2013a and R2014a installed.
:* Use the '''module load''' and '''module unload''' commands to switch between MATLAB versions on the head node or the interactive node.
:** To view what modules are available: module avail 
:** To load matlab R2012b and add the executables to your PATH: module load matlab/R2012b    
:** After loading the module, just type to run R2012b: matlab                        
:** To unload/remove R2012b from your PATH: module unload matlab/R2012b   
:** You can do the same for matlab/R2013a.  [R2013a is the default if you login and just run matlab.]
:** To force the addition of R2013a to your PATH: module load matlab/R2013a     

====How to Compile Matlab code?====

Assume we have a matlab file named matlabruntimetest.m:

<source lang="matlab">
# cat matlabruntimetest.m
 for i=1:10,
 fprintf('%d\n',i);
 end
</source>

We use mcc to compile the .m into a binary executable file. You can do this from within the MATLAB application, or from the command line after the correct module is loaded:

<pre>
mcc -m matlabruntimetest.m
</pre>

This will translate your .m file into the C language and compile it using a C compiler, so that you end up with an executable named matlabruntimetest. 
It will produce several other files in the process. In Linux, one of these files should be a shell script named run_matlabruntimetest.sh, which you will need to run the executable.
(There should also be a readme.txt with lots and lots of helpful information&mdash;too much, really, more than you want to know&mdash;which is available as a reference.)

Some useful extra options may be added to mcc:

<pre>
mcc -R -nodisplay -R -singleCompThread -m matlabruntimetest.m
</pre>

The first one says that the executable will not be displaying anything, and the second one says to disable multithreading. It is not clear that the latter option is absolutely necessary in batch, but it may help to keep jobs from interfering with each other when running on the same batch node.

====How to run compiled Matlab code on the Batch node?====

Create a job script similar to the following. Be sure your path-to-runtimes matches the version you compiled with. For R2012b, the path needs to be /opt/matlab/MATLAB_Compiler_Runtime/v80, while for R2014a it should be /opt/matlab/MATLAB_Compiler_Runtime/v83. R2013a is shown in the example below. For your convenience, the runtimes for R2012b, R2013a, and R2014a have been installed on all nodes. (This includes the login and interactive nodes - but please only do very brief runs there, for testing purposes.)

<source lang="bash">
$ cat testmatlab.sh

 #!/bin/bash
 #PBS -l walltime=00:02:00,nodes=1
 #PBS -j oe
 #PBS -N testmatlab
 #PBS -q test

 # Turn on echo of shell commands
 set -x
 # Because jobs start in the HOME directory, move to where we submitted.
 cd $PBS_O_WORKDIR
 echo 'pwd'
 echo "PBS_O_WORKDIR is `pwd`"
 echo "env is `env`"
 # copy the binary and data files to a local directory on node job is executing on
 cp $HOME/bin/run_matlabruntimetest.sh $TMPDIR
 cp $HOME/bin/matlabruntimetest $TMPDIR
 # run your executable from the local disk on the node the job was placed on
 # the command line arguments are: matlab_script path-to-runtimes outputfile
 cd $TMPDIR
 ./run_matlabruntimetest.sh /opt/matlab/MATLAB_Compiler_Runtime/v81 >&matlab.stdout 
 # Copy output files to your output folder       
 cp -f matlab.stdout $HOME
 sleep 10

 # end script; the sleep is only there for the purpose of you confirming 
 # you can see if it is running w/ 'showq' or 'checkjob'.
</source>

Now submit your script to the scheduler, taking note of the id# or job number that is returned:

<pre>
qsub testmatlab.sh
</pre>

To view your job's progress:

<pre>
showq
showq -u myusername
checkjob id#
</pre>

After the job runs, the output file specified by the script, matlab.stdout, will appear in your home directory.

==== Matlab code Tips====
Here a few tips to know for creating a MATLAB code that is suitable for mcc:

:* If your .m file is a function that takes one or more arguments, put these arguments on the command line after the path to the runtimes. But remember that the arguments will be input as strings, just like any arguments (argv) to a C program. Therefore your .m file should use str2num or str2double to convert any input arguments to the expected type.
:* Dump figures into files using "saveas" or "print"; or, write a separate postprocessing script that reads data files written by multiple tasks and produces the final plots.

Some minor gotchas...

:* Don't nest curly braces within comment blocks, e.g., %{ comment1 %{ comment2 %} %} - the braces may be misinterpreted as clause delimiters in C, due to an mcc bug.
:* Commands from the Symbolic Math Toolbox may not be recognized, e.g., heaviside().

It is possible to get much more elaborate with MATLAB's mcc system by, for example, using "mcc -l" to create a function library, or perhaps by implementing parallelism, but the above should be enough to get you started. For more information, see the MATLAB help as well as "mcc -?".

===Interactive Processing===

:*Users can login to the interactive node and run software.  
:*Users can submit jobs to the batch node from the interactive node or the cluster head node.
:*There is no scheduling on the interactive node; users will compete for resources.
:*For help with interactive sessions on Linux severs see instructions at [[Connect to Linux]]

==XNAT==
:* URL: [http://hd-hni-xnat.cac.cornell.edu http://hd-hni-xnat.cac.cornell.edu]
:* Initial Login: 
:# Select Login via '''LDAP'''
:# Enter you '''CAC user name and password'''
:# After authentication is successful, you will receive a separate email that your account has been approved.

==Monitoring the Cluster==
[http://hd-hni.cac.cornell.edu/ganglia HD-HNI Cluster Monitor]

==NEED HELP?==
 Password resets can be requested at: [https://{{SERVERNAME}}/services/myacct.aspx password reset]
 Cluster technical questions should be addressed to: [https://{{SERVERNAME}}/help CAC HelpDesk]
 Human Ecology related questions should be addressed to: [http://www.human.cornell.edu/helpdesk Human Ecology HelpDesk]