Difference between revisions of "HD Human Neuroscience Institute (HD-HNI) Computing"

From CAC Documentation wiki
Jump to navigation Jump to search
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Welcome to the User Documentation for the HD Human Neuroscience Institute Computing Environment ==
+
== Documentation for the HD Human Neuroscience Institute Storage ==
  
== IMPORTANT UPDATE - March 5, 2018 ==
+
==The HD-HNI file server consists of:==
All data has been transitioned to new storageSee instructions below for remote access.   
+
:*  1  file server - hd-hni-fs.cac.cornell.edu - serving 180TB capacity
 +
:* 1 head node - hdhni.cac.cornell.edu (access via ssh)
 +
:*  1 interactive node - interactive1.cac.cornell.edu (access via ssh)
 +
:* 1 batch node - access via the hdhni head node
  
 
+
==Who can access the HD-HNI compute environment?==
==The HD-HNI computing environment consists of:==
+
:*Users in Eve DeRosa and Adam Anderson's labs (ktb1_0005 CAC group).
 
+
:*New users will receive a welcome letter with their username and initial CAC HD-HNI password.
  1 Head Node - '''hd-hni.cac.cornell.edu''' - Dell R620 server
+
:*Users should change, reset, query CAC password at https://www.cac.cornell.edu/services/myacct.aspx
  1 Batch Node - compute-1-1.hd-hni - Dell R820 - 32 cores and 256GB RAM
 
  1 Interactive Node - '''hd-hni-interactive-1-1.cac.cornell.edu''' - Dell R820 - 32 cores and 256GB RAM
 
    alias: '''inter1.cac.cornell.edu'''  (easier to type than the full name!)
 
  1 Cluster file server - hd-hni-fs.cac.cornell.edu - serving 22.5TB of storage capacity
 
  1 Force10 switch -  connects everything together with 10GB ethernet!
 
 
 
==Installed Software==
 
 
 
{| class="sortable wikitable" border="1" cellpadding="4" style="width: auto"
 
|+ (sortable table)<br/>
 
! style="background:#e9e9e9;" | Package and Version
 
! style="background:#e9e9e9;" | Location
 
! style="background:#e9e9e9;" | hd-hni.cac.cornell.edu
 
! style="background:#e9e9e9;" | Compute Node
 
! style="background:#e9e9e9;" | Interactive Node
 
! style="background:#e9e9e9;" | Notes
 
|-
 
| Python 2.6.6
 
| /usr/bin/python
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" |
 
|-
 
| Python 2.7.6
 
| /usr/local/bin/python (points to /opt/python)
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | to load env variables, type: module load python-2.7.6
 
|-
 
| PyMVPA2
 
| /usr/bin
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" |
 
|-
 
 
| fsl
 
| /opt/fsl
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" |
 
|-
 
| freesurfer
 
| /opt/freesurfer
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | setup environment, type: source $FREESURFER_HOME/SetUpFreeSurfer.sh
 
|-
 
| afni
 
| /opt/abin
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" |
 
|-
 
| meica
 
| /opt/meica
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" |
 
|-
 
| R 3.2.2
 
| /usr/bin/R
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" | yes
 
| align="center" |
 
|-
 
| xnat
 
| http://hd-hni-xnat.cac.cornell.edu/
 
| align="center" | 
 
| align="center" | 
 
| align="center" | 
 
| align="center" |
 
|
 
|}
 
 
 
:* It is usually possible to install software in your home directory; you can also request through your helpdesk to have the CAC install software, pending the approval from the PI.
 
:* You can view a list of installed software with: ''''rpm -qa''''. Use the grep option to search for specific software: rpm -qa | grep sw_name [i.e. rpm -qa | grep perl ]
 
 
 
==Who can access the HD-HNI computing environment?==
 
:*Users should request access to the HD-HNI project by contacting [mailto:amh352@cornell.edu Allison Hermann].
 
:*When added, users will receive a welcome letter with their username and initial CAC HD-HNI password.
 
 
 
==What servers can users access and how?==
 
HD-HNI users can '''directly access two nodes''': the cluster head node and the interactive node.
 
 
 
:*How to [[Connect to Linux]] servers?
 
:*HD-HNI users submit "jobs" to the [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]].
 
 
 
==First Login==
 
The first thing to do is login to the cluster head node and change your password.
 
This will require an ssh client.
 
How to [[Connect to Linux]] server (head node)?
 
If you do not have a ssh client, contact your local [http://www.human.cornell.edu/helpdesk helpdesk].
 
 
 
[https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started#Managing_your_password Managing Your Password]
 
 
 
After you change your password, you will be prompted for an ssh pass phrase; leave this blank, simply hit the Enter key.
 
You are now ready to begin using the HD-HNI computing environment.
 
  
 
==HD-HNI File Server - Moving data from your workstation to the shared file server==
 
==HD-HNI File Server - Moving data from your workstation to the shared file server==
Line 201: Line 98:
 
  psftp> put localresults.dat results.dat
 
  psftp> put localresults.dat results.dat
 
  psftp> quit
 
  psftp> quit
 
==HD-HNI Data Processing Environment==
 
There is one [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]] (compute-1-1) and one interactive node (hd-hni-interactive-1-1) for processing HD-HNI data.
 
===How to access the data processing nodes?===
 
 
Submit jobs to the [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]] from the head node or from the interactive node.
 
ssh directly to the head node and the interactive node.
 
ssh to the [[HD_Human_Neuroscience_Institute_(HD-HNI)_Computing#Batch_Processing | batch node]] from the head node only.
 
 
====Maui Scheduler and Job submission/monitoring commands====
 
 
Jobs are scheduled by the [http://www.adaptivecomputing.com/resources/docs/maui/mauiusers.php Maui scheduler] with the [http://www.adaptivecomputing.com/products/open-source/torque/ Torque resource manager]. We suggest you use a job submission batch file utilizing  [http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/commands/qsub.htm PBS Directives ('Options' section).]
 
 
===== Common Maui Commands=====
 
(If you have any experience with PBS/Torque or SGE, [http://www.adaptivecomputing.com/resources/docs/maui/a.gcommandoverview.php Maui Commands] may be recognizable. Most used:
 
 
'''[http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/commands/qsub.php qsub]''' - Job submission (jobid will be displayed for the job submitted)
 
:* $ qsub jobscript.sh
 
 
'''[http://www.adaptivecomputing.com/resources/docs/maui/commands/showq.php showq] - Display queue information.''' 
 
 
:* $ showq (dump everything)
 
:* $ showq -r (show running jobs)
 
:* $ showq -u foo42 (shows foo42's jobs)
 
 
'''[http://www.adaptivecomputing.com/resources/docs/maui/commands/checkjob.php checkjob] - Display job information.'''  (You can only checkjob your own jobs.)
 
 
:* $ checkjob -A jobid (get dense key-value pair information on job 42)
 
:* $ checkjob -v jobid (get verbose information on job 42)
 
 
'''[http://www.adaptivecomputing.com/resources/docs/maui/commands/canceljob.php canceljob] - Cancel Job.''' (You can only cancel your own jobs.)
 
:* $ canceljob jobid
 
 
 
====Batch Processing====
 
 
Submit jobs to the batch node via the cluster head node or the interactive node. Jobs are scheduled as resources are available.
 
 
:*First ssh to the Head node or the Interactive node
 
<pre>
 
ssh username@hd-hni.cac.cornell.edu
 
or
 
ssh username@inter1.cac.cornell.edu
 
</pre>
 
 
:*Next, using your favorite editor create a job file named test_default_queue.sh (text file of commands) containing the following lines:
 
<source lang="bash">
 
#!/bin/bash
 
#PBS -l walltime=00:05:00,nodes=1
 
#PBS -j oe
 
#PBS -N testdefaultqueue
 
#PBS -q default
 
 
# Turn on echo of shell commands
 
set -x
 
# Because jobs start in the HOME directory, move to submit dir
 
cd $PBS_O_WORKDIR
 
echo 'pwd'
 
echo "PBS_O_WORKDIR is `pwd`"
 
echo "env is `env`"
 
# copy your binary that you want to run and any data files to a local directory on node job is executing on
 
# this example assumes you have a binary file named helloworld.sh in your local bin directory
 
cp $HOME/bin/helloworld.sh $TMPDIR
 
cd $TMPDIR
 
# run the binary file from the local disk on the node the job was placed on
 
./helloworld.sh >&hello.stdout
 
# Copy output files to your output folder     
 
cp -f $TMPDIR/hello.stdout $HOME/output
 
</source>
 
 
:*Submit the test job:
 
<pre>
 
qsub test_default_queue.sh
 
</pre>
 
 
:*Check the status of the job:
 
<pre>
 
showq
 
</pre>
 
 
:*Once the job runs the output file will be created with the name you gave it (-N above) with an appended ".o[jobid]".
 
:To view the output:
 
<pre>
 
cat testdefaultqueue.o[jobid]
 
</pre>
 
 
===Interactive Processing===
 
 
:*Users can login to the interactive node and run software. 
 
:*Users can submit jobs to the batch node from the interactive node or the cluster head node.
 
:*There is no scheduling on the interactive node; users will compete for resources.
 
:*For help with interactive sessions on Linux severs see instructions at [[Connect to Linux]]
 
  
 
==XNAT==
 
==XNAT==
Line 300: Line 105:
 
:# Enter you '''CAC user name and password'''
 
:# Enter you '''CAC user name and password'''
 
:# After authentication is successful, you will receive a separate email that your account has been approved.
 
:# After authentication is successful, you will receive a separate email that your account has been approved.
 
==Monitoring the Cluster==
 
[http://hd-hni.cac.cornell.edu/ganglia HD-HNI Cluster Monitor]
 
  
 
==NEED HELP?==
 
==NEED HELP?==

Latest revision as of 10:55, 3 December 2020

Documentation for the HD Human Neuroscience Institute Storage

The HD-HNI file server consists of:

  • 1 file server - hd-hni-fs.cac.cornell.edu - serving 180TB capacity
  • 1 head node - hdhni.cac.cornell.edu (access via ssh)
  • 1 interactive node - interactive1.cac.cornell.edu (access via ssh)
  • 1 batch node - access via the hdhni head node

Who can access the HD-HNI compute environment?

  • Users in Eve DeRosa and Adam Anderson's labs (ktb1_0005 CAC group).
  • New users will receive a welcome letter with their username and initial CAC HD-HNI password.
  • Users should change, reset, query CAC password at https://www.cac.cornell.edu/services/myacct.aspx

HD-HNI File Server - Moving data from your workstation to the shared file server

A central file server, hd-hni-fs.cac.cornell.edu, serves all HD-HNI user home directories. Users can not ssh to this server but can connect and access files in a variety of other ways outlined below.

Note: by default, your home directory and its contents will be readable and executable by all other users of the HD-HNI systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.

Attach your home directory to your local computer

You can "mount" your home directory from computers on the Cornell network or if off campus, computers connected to Cornell VPN. Once your home directory is mounted, you can drag and drop files to it.

Windows Users - how to mount HD-HNI directories

NOTE: WE RECOMMEND THAT YOU DO NOT SAVE YOUR PASSWORD ON PUBLIC COMPUTERS, DOING SO MAY CAUSE CONNECTION TROUBLES

  • Open My Computer
  • Click on Tools -> Map Network Drive
  • Drive H: (if you are already using this drive letter, use another letter)
  • Folder: \\hd-hni-fs.cac.cornell.edu\<userid>
  • or
  • Folder: \\hd-hni-fs.cac.cornell.edu\reyna-fmri
  • Then:
    • Select "Connect using a different user name:". This will allow you to enter the CAC domain and your HD-HNI userid at CAC, rather than those associated with your own machine.
    • User name: CTC_ITH\your_userid
    • Password: your CAC HD-HNI password
  • Troubleshooting: If you have already mapped the drive and subsequently have problems, disconnect the drive and remap it.

MacOS X Users - how to mount HD-HNI directories

NOTE: WE RECOMMEND THAT YOU DO NOT SAVE YOUR PASSWORD ON PUBLIC COMPUTERS, DOING SO MAY CAUSE CONNECTION TROUBLES

  1. In the Finder, select Connect to Server... from the Go menu.

    FileAccess1.jpg
  2. Enter smb://hd-hni-fs.cac.cornell.edu/<user name> in the Server Address field. You may need to use smb://<username>@hd-hni-fs.cac.cornell.edu/<username>.
  3. Enter your CAC user name and password to log in.

Linux Users - how to mount HD-HNI directories

You cannot mount the HD-HNI directories vis NFS for security reasons. To mount it as a CIFS drive, you need to be root, which often means using the sudo command. Then execute

 mount -t cifs //hd-hni-fs.cac.cornell.edu/<user name> /mnt/test -o user=<user name>,workgroup=ctc_ith,vers=3.0

where <username> is your username, and /mount/point is the name of a directory you have already created on your local filesystem. Enter the password for your CAC project when prompted. See man mount.cifs for available options for the mount command

If you see errors, such as "missing codepage or helper program," then you have not installed the mount and umount packages for CIFS on your local machine. If problems persist, send your initial command and the results of dmesg | tail.

File Transfers To The HD-HNI file server

Users can transfer data to the HD-HNI-FS file server without mounting the file systems locally.

File Transfers between File Server and your Linux or MAC workstation

Secure Copy

Secure copy is a standard tool to copy files to and from remote hosts.

This example copies a file named "localfile.dat" on your local workstation to a file named remoteinput.dat on the remote file server:
localhost$ scp localfile.dat username@hd-hni-fs.cac.cornell.edu:remoteinput.dat

This example copies the file named results.dat on the remote server to a file named localresults.dat on your local workstation.
localhost$ scp username@hd-hni-fs.cac.cornell.edu:results.dat localresults.dat 

Secure FTP

FTP is disabled for security reasons, but sftp's interface is nearly identical.

localhost$sftp username@hd-hni-fs.cac.cornell.edu
    <enter your username's password when prompted>
sftp> put localresults.dat results.dat
sftp> quit

Samba Client

Type

smbclient //hd-hni-fs.cac.cornell.edu/<user name> -U <user name> -W ctc_ith -m SMB3

Enter the password for your HD-HNI account when prompted You will see the smb:\> prompt. You can now start transferring files between your local machine and your HD-HNI home directory similar to ftp client. Type help for more instructions.

-sh-3.2$ smbclient //hd-hni-fs.cac.cornell.edu/<user name> -U <user name> -W ctc_ith -m SMB3
    Password: 
    Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]
    smb: \> help

File Transfers between the File Server and your Windows Workstation

Secure Copy

The people who make Putty provide a secure copy client called pscp.

 This example copies local workstation file to the remote file server:
  cmd> pscp localfile.dat username@hd-hni-fs.cac.cornell.edu:remoteinput.dat
     <enter your username's password when prompted>

 This example copies a file on the remote file server to the local workstation:
  cmd> pscp username@hd-hni-fs.cac.cornell.edu:results.dat localresults.dat

Secure FTP

FTP is disabled for security reasons, but psftp's interface is nearly identical. From the command prompt, type:

cmd> psftp username@hd-hni-fs.cac.cornell.edu
    <enter your username's password when prompted>
psftp> put localresults.dat results.dat
psftp> quit

XNAT

  1. Select Login via LDAP
  2. Enter you CAC user name and password
  3. After authentication is successful, you will receive a separate email that your account has been approved.

NEED HELP?

Password resets can be requested at: password reset
Cluster technical questions should be addressed to: CAC HelpDesk
Human Ecology related questions should be addressed to: Human Ecology HelpDesk