# Getting Started

Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.

• Each password must have at least eight characters.
• Each password must contain at least three of the following four elements among its first eight characters:
- uppercase letters (English, A through Z)
- lowercase letters (English, a through z)
- special characters (for example, !, $, #, %) - digits (0 through 9) • Do not use a space in a password. Though technically allowed, it may be a source of confusion. • Do not form a password by appending a digit to a word--this type of password is easily guessed. • Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent. • New passwords should differ from the old by at least three characters. If you need additional ideas for creating a new password, Microsoft has a few tips. ### Change a password at first login When you are issued a CAC user name, you should first log in to a login node. You will be prompted to change your password. You do this just once, and it should change your password for all CAC resources that require a login. Refer to the rules for creating passwords. After you change your password, you will be logged in. Windows users: Log in to the Windows login node, winlogin.cac.cornell.edu, using the Remote Desktop Connection client that comes with Windows. (There is also a free client for macOS that you can download from the App Store.) Here are the steps to follow in the RDC client: • Log in with your CAC user name and your current password (the domain name should be CTC_ITH) • From the desktop, choose "Windows Security" from the Start menu; or, if you are coming from Windows, enter the key combination Ctrl-Alt-End • Choose "Change a password..." and enter your old and new passwords as indicated. Old Password: 0ldpassw0rd!! New Password: newpassw0rd!! Confirm New Password: newpassw0rd!!  Linux and Mac users: You can follow these steps on the head node of a private cluster; if your CAC project doesn't have a private cluster, you can follow them on linuxlogin.cac.cornell.edu. On many Linux clusters, when you first change your password, you will also be asked for an ssh passphrase. You can leave this blank--just hit the Enter key. Assume that you have an old password '0ldpassw0rd!!' and a new password 'newpassw0rd!!'. This what should happen when you do the following in a Terminal client window: $ ssh your_username@linuxlogin.cac.cornell.edu
Retype new UNIX password: (ENTER newpassw0rd!!)
passwd: all authentication tokens updated successfully.
Connection to linuxlogin closed. 

If you get a token error it very likely means that the password is not complex enough. Your password must be a mix of any three of the following: lower case letters, upper case letters, numbers and some sort of punctuation to create an 8 character or longer password (it is slightly more complex; don't use your user name or previous password - more info in Password Policy ).

If you have additional trouble, you can use either the rdesktop client (for linux) or the Remote Desktop client (for Windows or Mac) to log into the Windows login node winlogin.cac.cornell.edu, then follow the instructions for Windows above. This gives you better information about password complexity issues during the password change.

### Change password at any time

You can change your CAC password before it expires. You will want to do so if you feel that your password has been compromised in any way. For example, suppose you think that someone else knows your password or you are concerned that you issued your password in a nonsecure setting that would have led to sending it in clear text. To change it:

• Be sure that you have no other open connections to any CAC resources.
- The only open interactive session should be the one in which you are changing the password.
- Log off all sessions connected to login nodes.
- Log off all remote connections to other CAC machines.
- It is not enough to disconnect the active sessions; you must log off. Failure to do so will lead to the system locking your account.
- Disconnect locally mapped drives to the CAC file server. Again, if this is not done, the system will automatically lock your account.
- Use Remote Desktop Connection to connect to winlogin, or an SSH client to connect to linuxlogin or to a Linux head node (for a private cluster).

• On winlogin, open a command prompt window (cmd), then issue the command
 net user <your CAC user name> /domain

Look for the line "Password expires".
• There is no equivalent command on the Linux login nodes.

Please contact CAC Help by submitting a ticket on our issue tracking system or by calling 607-254-8691.

### Locked Accounts

There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:

Solution: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!
• Trying to login to a Windows login node by using SSH when you have a new or expired password.
Solution: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.
• Failing to log off all other sessions connected to login nodes.
Solution: Log off all remote connections. Disconnecting the sessions is not enough.
• Failing to disconnect locally mapped drives to the CAC file server before changing your password.
Solution: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.

If you can't log on or can't wait you can submit a Password Reset ticket on our issue tracking system

Cornell University users can view their account limits at CAC Account Limits.
Partner Program members should contact Paul Redfern at red@cac.cornell.edu if they need information on their membership limits.

## Using CAC resources

### Connecting to CAC

There are two types of login nodes:

• Linux login nodes: linuxlogin.cac.cornell.edu as well as the head nodes for the various Linux-based private clusters.

#### Connect to Linux

There are three distinct ways to connect to a login node:

1. Use SSH to open a Linux shell on a login node, which provides a text-only interface.
2. Use SSH together with X-Windows, which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.
3. Use VNC to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.

These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.

Whichever method you choose, at your first login, you will be challenged for a new password. Find help at Changing a Password at First Login. You will also be asked for an ssh passphrase. You can just leave this blank; hit the Enter key in response.

##### Using Secure Shell

For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.

• Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.
• If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.
• The non-secure predecessor of SSH, telnet, is disabled for security reasons.

Linux users:

To connect to the second login node with ssh, you simply open a terminal window and type

localhost$ssh username@linuxlogin.cac.cornell.edu  Mac OS X users: OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application. • One option is to use the shortcut cmd-space to open Spotlight and then type "Terminal" to open a Terminal window. Otherwise: • Navigate in the Finder to the Applications folder and Utilities sub-folder. • Then double-click on the Terminal application to see a Bash command-line. • As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window. Windows users: Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free PuTTY client. • The simplest installation is to download the Windows installer, called putty-0.67-installer.msi, and run it. This installs PuTTY into your Start menu. • To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open". ##### Using X-Windows X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH. Appropriate use Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does NOT imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. On linuxlogin, such usage is contrary to CAC policy. On other shared resources, it is disrespectful toward other users because the login node may become unresponsive through your actions. Linux users: The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the -X option, it will automatically set up the necessary tunnel and environment variables. localhost$ ssh -X username@linuxlogin.cac.cornell.edu
linuxlogin$echo$DISPLAY
localhost:11.0
linuxlogin$gs  If all goes well, you should see a valid setting for your DISPLAY environment variable, then have a blank window presented to you by gs (Ghostscript, the PostScript and PDF previewer). Note, if gs is not installed on the machine you're logging into, you can try another X client such as xclock, xlogo, emacs, etc. There is another option to use a trusted version of X-windows forwarding, linuxlogin$ ssh -Y compute-1-37


When you're working on a cluster, the trusted (-Y) version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.

Mac OS X users:

If you start ssh with the -X or -Y option, X-Windows should start up automatically. You can then try the "gs" test, as described above for Linux.

X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the Mac OS X Server Introduction to Command-Line Administration, "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."

Windows users:

Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.

• Xming - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download
• Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.
• Xming-fonts (public domain release)

If you purchase the website release of Xming, remember to install the Xming-fonts, as well.

Here are some other X-server possibilities for Windows:

• VcXsrv - Open Source. Freeware solutions like this one can often work very well, but as always, the installation and use of such packages comes with no guarantees.
• Cygwin/X - Open Source. Cygwin is much more that just an X-Windows server. It actually creates an entire Linux-like environment within Windows.
• OpenText's Exceed and Exceed 3D - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu; choose the one that just says "Exceed" because it starts the program in multi-window mode, which is usually what you want.

Here is how to start a session using PuTTY and Xming.

1. Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.
2. Start PuTTY.
3. In the window that appears, type a host name, linuxlogin.cac.cornell.edu.
4. Use the tree menu on the left to set X11 forwarding. It's in the Connection > SSH branch.
5. For PuTTY 0.61 and above - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck "Attempt GSSAPI authentication". This will prevent an annoying "Access denied" message from appearing in your terminal window.
6. You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.
7. Click Open, and it will connect to a login node.
8. Test your X-Windows setup by typing the command for Ghostscript, which is a PostScript and PDF previewer:
gs


You should see a blank window appear on your screen. You can stop it by typing Ctrl-c in the terminal window.

##### Using VNC

VNC lets you see a whole Linux desktop on a remote computer from your local computer. Connecting to Linux via SSH and X-Windows is efficient in that it uses a lot less of the remote computer's resources, but VNC can be much faster if you are doing visualization on the remote computer from off campus.

For security reasons, CAC requires all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the remote computer using SSH. The firewalls running at CAC for all login nodes (e.g., linuxlogin) commonly block all incoming ports except for ssh, so VNC connections must be made over a ssh tunnel as described below.

Appropriate use on clusters

VNC gives you the ability to establish a remote desktop on the login node(s) for a cluster, but this ability does NOT imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. On linuxlogin, such usage is contrary to CAC policy. On other shared resources, it is disrespectful toward other users because the login node may become unresponsive through your actions.

Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.

1. Open a VNC connection to the login node through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.
2. In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).
3. Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).
4. Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"
5. This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".

Initial setup (You only need to do this once)

• Install a local VNC client if one isn't installed. For Windows, TightVNC works well, but so do others. For Mac, you can use the built-in Screen Sharing app.

Start your VNC server (Do these steps from an ssh shell)

• On the Linux login node, start the VNC server using the "vncserver" command like this:
 vncserver -geometry 1024x768 -localhost


The geometry numbers 1024x768 (or other numbers of your choosing) specify the size of the desktop in pixels.

• You will need to get the display number from the output of the vncserver command:
 New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1
Starting applications specified in /home/fs01/shl1/.vnc/xstartup

• vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.

Set up your ssh tunnel (Do these steps on your local computer)

• Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12.
• From Linux, in order to start ssh port forwarding or tunneling to that port, type into a terminal:
 ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu

• From Mac OS X, open a Terminal and enter the Linux command above.
• From Windows, ssh clients such as PuTTY can do port forwarding (tunneling); see VNC Tunnel Windows.
• Leave this ssh session running on your local client computer. (It can run in the background.)

• Launch your VNC client program. Connect it to localhost:10000. When prompted, type in your VNC server password.
• A nice GNOME desktop should appear!
• See this link for how to prevent the "Authenticate" pop-up from appearing in your future vncserver sessions.

• Close the VNC client program.
• Disconnect the ssh forwarding session (i.e., kill it).

• Restart port forwarding with ssh, using the same remote port number as before.
• Again connect the VNC client to localhost:10000.

When you are all done

• On the Linux computer, type this command to shut down the VNC server:
 vncserver -kill :<display number>

• If you merely log out from the Linux desktop, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)

Create ssh key pair

Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to change your password. This will become your password for connecting to the nodes.

Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.

Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running plink.exe to connect to the linux login node, as shown in this example:

>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu
Rocks 5.0 (V)
Profile built 12:54 06-May-2008

Kickstarted 09:22 06-May-2008
-----------------------------------------------------------
Welcome to the Center for Advanced Computing Cluster!
-----------------------------------------------------------
-----------------------------------------------------------

It doesn't appear that you have set up your ssh key.
This process will make the files:
/home/gfs01/cacshl1/.ssh/id_rsa.pub
/home/gfs01/cacshl1/.ssh/id_rsa
/home/gfs01/cacshl1/.ssh/authorized_keys

Generating public/private rsa key pair.
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default
Created directory '/home/gfs01/cacshl1/.ssh'.
Enter passphrase (empty for no passphrase): Press Enter to accept default
Enter same passphrase again: Press Enter to accept default
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.


After this is done, type "exit" to log out of the linux login node.

Convert ssh Private Key for Putty / Plink

Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:

• Run C:\Program Files (x86)\Putty\puttygen.exe.
• Select Import Key from the Conversions menu and select H:\.ssh\id_rsa in your home directory. And click on the Open button.
• Click on the "Save Private Key" button.
• Click on "Yes" when asked to save the private key without a passphrase.
• Save the private key as private.ppk in the .ssh directory inside your home directory.
• Close (choose File, then Exit)
• To confirm you have converted the ssh private key successfully, do:
"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu

It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."

• You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.

#### Connect to Windows

Using Remote Desktop Connection to connect to winlogin

This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to CTC_ITH, not the local name of the machine to which you are connecting.

Remote Desktop Connect Details:

Remote Desktop sessions do not expire, but they will end when machines are rebooted during down times.

• If you use a Windows machine:
Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, click Start, then All Programs > Accessories > Communications > Remote Desktop Connection. Otherwise you need to download the client before you can use it.
• If you use Mac OS X 10.7 or later:
Use the free download from the Mac App Store. Works just like the Remote Desktop Connection in Windows 7. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.
• If you use Unix or Linux or Mac:
You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from rdesktop. Executables for old versions are available from here

### Home Directory Access

There is one filesystem which is shared by both Linux and Windows systems:

• Your home directory on Linux is: /home/fs01/userid
• Your home directory on Windows is: \\storage01.cac.cornell.edu\userid

localhost$scp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat  ##### Secure FTP FTP is disabled for security reasons, but sftp's interface is nearly identical. ##### Samba Client This technique only works from Cornell campus locations or via a Cornell VPN connection. Type smbclient //storage01.cac.cornell.edu/<user name> -U ctc_ith\\<user name>  (Note, the shell interprets \\ as a single backslash.) Enter the password for your CAC account when prompted. You will see the smb:\> prompt. Now you can start transferring files between your local machine and your CAC home directory, using commands similar to the sftp client. Type help for more instructions.  -bash-4.1$ smbclient //storage01.cac.cornell.edu/<user name> -U ctc_ith\\<user name>
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.6.23-24.el6_7]
smb: \> help


#### Windows users

##### Secure Copy

The individual who created PuTTY provides a secure copy client called pscp. From the command prompt, type:

cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat

##### Secure FTP

FTP is disabled for security reasons, but psftp's interface is nearly identical. From the command prompt, type:

cmd> psftp username@linuxlogin.cac.cornell.edu
psftp> put localresults.dat results.dat
psftp> quit


### Linux Usage Tips

If you have never used Linux before, we recommend exploring the Linux Tutorial.

#### Linux shells

• /bin/sh is the default login shell.
• Edit $HOME/.profile to change interactive variables. • The$HOME/.bashrc file will not be run for non-interactive shells.
• /bin/bash
• Edit $HOME/.profile to change interactive variables. • The$HOME/.bashrc file will be run for non-interactive shells.
• /bin/csh and /bin/tcsh
• Edit $HOME/.login to change interactive variables. • The$HOME/.cshrc file will be run for non-interactive shells.

The change shell command, chsh, will not permanently change your shell. You must send a request instead. Contact Support

The default login shell on v4 interactive and batch nodes is sh. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of bash. Accordingly, you will find that "man sh" brings up the man page (the help document) for bash. In a way, then, you can think of your login shell as being bash, too.

There are slight differences between sh and bash, however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both sh and bash; but any interactive sh shells you start thereafter will not run ~/.bashrc as you might expect from bash. The way to get sh to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).

Let's say you simply prefer to have bash as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be bash. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be bash (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is sh.

The relationship between the csh and tcsh shells is similar to the one between sh and bash. For instance, your csh shells are automatically endowed with the tcsh-style ability to retrieve history through the up- and down-arrow keys. The best way to make tcsh into your everyday working shell is to run it on top of sh after you log in (again, you can do this as part of your .profile).

References

#### Compiling and linking code on Linux

Use /tmp to compile large codes and software packages. This will provide improved performance and greater system stability.

If you want to know what processor features a cluster supports, submit a batch job that does "cat /proc/cpuinfo" in order to find out the CPU type. The v4 cluster is composed mostly of Intel E5420 CPUs (in Nov. 2011). Then you go to Wikipedia's Intel Xeon page or Intel's ARK to find that these are Harpertown cores that support SSE, SSE2, SSE3, SSSE3, SSE4.1 and VMX.

##### C/C++ and Fortran Codes
• GNU compilers gcc, g++, g77, gfortran are in /usr/bin, which is in the default path.
• For compiling OpenMP directives, add the option -fopenmp.
• Intel 12.1 compilers icc, ifort are in the default path on the login nodes.
• For compiling OpenMP directives, add the option -openmp.
• The following Intel libraries and tools are available to you automatically through the default setup on the login nodes:
- MKL, the Math Kernel Library 10.3.6 (additional help below)
- idb, the Intel debugger for Linux
- TBB, the Threading Building Blocks
- IPP, the Integrated Performance Primitives
• If any of the above libraries are linked dynamically, the correct runtimes will be loaded automatically on the compute nodes by default; no additional setup is required.
• Note - if you find that your code segfaults after compiling with Intel 12.1, try disabling optimization or using the older 11.1 version of the compilers instead.
Reason: there is a known bug in the vectorizer of the 12.1 compiler which is due to be fixed in a future release.
• Intel 11.1 compilers icc, ifort are available, also, but these older compilers require special setup files.
• Before compiling in bash: source /opt/intel/intel-11.sh
• Before compiling in tcsh: source /opt/intel/intel-11.csh
• At runtime, in a batch sh-script: source /opt/intel/intel-11.sh
• At runtime, in a batch csh-script: source /opt/intel/intel-11.csh
• The above steps also enable the use of the older Intel performance libraries, e.g., MKL 10.2 (additional information below).
• Help for Intel compilers (if you are using 11.1, be sure to source the setup file first):
• Fortran: man ifort, info ifort, ifort -help
• C/C++: man icc, info icc, icc -help
• Standard compiler options - the clusters have Intel Core2 processors, so standard compiler options are:
• For Intel: -O3 -ipo -mtune=pentium4 -march=pentium4
• Other options of possible interest (consult man pages):
• For Intel: -fno-alias -align -scalar_rep -prefetch
##### Generating Debugging Info
• Intel compilers
• icc -Wall
• ifort -g -debug -warn -C (-CB for bounds checking only)
##### MPI Programs

For compiling MPI codes, we recommend using mpicc and mpif90. If you specifically need a C++ compiler, try mpicxx. Because of these handy wrapper scripts, you may not need to do very much to convert existing makefiles to work with CAC's preferred software stack. Currently the default paths are set up so that the mpicc, mpif90 and mpicxx utilities invoke the Intel 12.1 compilers to compile your codes and link them properly to the Intel MPI 3.1 libraries. However, if you run the Intel 11.1 compiler setup file first, then these utilities will automatically use the older 11.1 compiler version. Documentation for the Intel MPI 3.1 Library, including mpdboot and mpiexec, is in PDF on the Intel Support Site.

To view a sample batch script that will run an MPI job for you, see the section on Running a parallel MPI job.

The ROCKS operating system comes with several alternate MPI implementations (e.g., mpich2, OpenMPI). You have to play with environment variables and paths to get them to work.

##### Intel MKL

Intel's Math Kernel Library (MKL) is a good source of optimized routines for linear algebra, Fast Fourier Transforms, vector math, and other mathematical operations. In particular, it provides a way to incorporate Intel's optimized BLAS and LAPACK routines into your code.

OpenMP multithreading is built into certain MKL libs. When these libs are linked, calls to MKL will detect the same settings that would affect any other OpenMP-enabled code. This means MKL will attempt to use all the cores present on a v4 node during the execution of parallelized sections. Therefore, when you link your code with a "_thread" version of the MKL library, your should realize that all your calls to MKL will generally fork the same number of threads as the number of cores present. This may cause undesired interference with other parallelization strategies you are using, e.g., MPI. If this is not the behavior you want, you can do one of two things:

• At run time, set the OMP_NUM_THREADS environment variable to 1. (Use "export" or "setenv".) A value of 8 recovers the default behavior on v4.
###### Linking Intel MKL 10.3.6 with the Intel 12.1 Compilers

MKL 10.3.6 is the version installed with the 12.1 compilers. The easiest way to link MKL is to compile as follows, where the last two lines pertain to MPI codes:

• icc mycode.c -o mycode -mkl
• ifort mympicode.c -o mycode -mkl
• mpicc mympicode.c -o mympicode -mkl
• mpif90 mympicode.f90 -o mympicode -mkl

Note, -mkl is the same as -mkl=parallel, which enables OpenMPI mulithreading. If you don't want this, use -mkl=sequential.

With just plain -mkl (or -mkl=...), the resulting executable will be dynamically linked. This means that at run time, your program has to know where to find the MKL shared libraries. Since MKL 10.3.6 is the default, the appropriate paths have been predefined for you on the compute nodes, and your batch jobs should have no trouble.

Should you want to link MKL in some different way--e.g., statically--the compile line will start looking messier. Linking to MKL has become rather complicated due to Intel's decision to maximize MKL's flexibility and multi-platform compatibility by splitting out four separate layers of libraries: interface, threading, computational, and runtime (meaning OpenMP, if the _thread lib is requested). To make sure you have all these layers, we recommend appending one of the following snippets to your ifort, icc, mpif90, or mpicc command (after first setting MKLPATH = $MKLROOT/lib/intel64): • static, multithreaded:$MKLPATH/libmkl_solver_lp64.a -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a$MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -openmp -lpthread • static, sequential:$MKLPATH/libmkl_solver_lp64_sequential.a -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a$MKLPATH/libmkl_sequential.a $MKLPATH/libmkl_core.a -Wl,--end-group -lpthread These options will generate a (mostly) statically linked executable. Note, each .a-lib must be identified by its full path in order to prevent the .so-lib (its dynamic equivalent) from being found instead. If you do not need access to the MKL solver routines, simply remove that item from the head of the list. As noted previously, if your main program is itself threaded with OpenMP, or if it is parallelized with MPI, you may want to select libmkl_sequential.a in order to reduce contention and get better performance. To generate a dynamically linked rather than statically linked executable, the above options become: • dynamic, multithreaded:$MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -openmp -lpthread
• dynamic, sequential:
$MKLPATH/libmkl_solver_lp64_sequential.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread These sets of options are pretty much equivalent to -mkl=parallel and -mkl=sequential, respectively. If your batch script needs strict control over LD_LIBRARY_PATH, then one other compiler/linker option may be helpful for a dynamically-linked code: • -Wl,-rpath,$MKLPATH,-rpath,$IOMPPATH The above variables should be set to MKLPATH =$MKLROOT/lib/intel64 and IOMPPATH = $MKLROOT/../compiler/lib/intel64. This option "hardwires" the correct paths into the executable; these paths are valid on both the v4 compute nodes and the v4 login nodes. If you don't wish to restrict your executable in this fashion, the alternative is to add these paths manually to LD_LIBRARY_PATH. Intel has put together a helpful tool for generating the correct linker options to match your specific needs, the Link Line Advisor. This is definitely the place to go if you want to, e.g., use extra-long integers or compile with gcc or gfortran. It's well worth a visit. Much more information on linking MKL 10.3.6 can be found in the "Linking Your Application" section of the User Guide, which you can access from the login node ("firefox /opt/intel/composer_xe_2011_sp1.6.233/Documentation/en_US/mkl/mkl_userguide/index.htm"). ###### Linking Intel MKL 10.2 with the Intel 11.1 Compilers MKL 10.2 is the version installed with the 11.1 compilers. Since 11.1 is not the current default version of the compilers, you must first source the setup file: • In bash (or sh): source /opt/intel/intel-11.sh • In tcsh (or csh): source /opt/intel/intel-11.csh If your program links MKL dynamically, it has to know where to find the correct MKL shared libraries at run time. Bear in mind that MKL 10.2 is not the default on the compute nodes, either. The easiest way to ensure correct behavior at run time is to put the same line into your batch script: • In a batch sh-script: source /opt/intel/intel-11.sh • In a batch csh-script: source /opt/intel/intel-11.csh Otherwise the instructions for linking MKL 10.2 are identical to the instructions for MKL 10.3.6 and Intel 12.1 above. There is one exception: you need to set MKLPATH =$MKLROOT/lib/em64t and IOMPPATH = \$MKLROOT/../lib/intel64.

The Link Line Advisor can be applied to older versions of the Intel compilers and MKL. It's well worth a visit.

Much more information on linking MKL 10.2 can be found in the Sec. 5 of the User Guide, which you can access from the login node ("firefox /opt/intel/Compiler/11.1/072/Documentation/en_US/mkl/userguide.pdf").

#### FAQ

##### How do I determine my program's dependencies on shared library (.so) files?
• ldd - see the man page.

If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.

##### How do I display an image file (such as jpeg or gif)?
• display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.
• firefox mypic.jpg - any decent Web browser can handle it.

Note, the image will show up only if you have X11 forwarding enabled.

##### How do I use revision control?
• Subversion, Git and CVS are examples of revision control (or version control or source control) software, which means they help you collaborate with others on revising your source code by saving versions of the code as you write it. Clients for all three are installed on the login nodes. See the man pages for svn, git and cvs for details. To see the installed versions, type the commands with --version.

CIT runs a free TeamForge server for Subversion users. You can login with Cornell Single Sign-on. There is also a GitHub server that is intended for users in Engineering, CIS, and Cornell Tech.

For more detailed instructions on how to use the Linux nodes, see CAC's tutorial.

### Windows Usage Tips

#### Compiling and Linking Codes on Windows

The Windows login node provides the Microsoft Visual Studio 2005 and 2008 integrated development environments (IDEs) into which the following compilers are integrated: Microsoft C/C++, Intel C/C++ 11.0, and Intel Fortran 11.0, among others. If desired, any one of these compilers can be invoked from the command line in a Windows command shell (cmd) as well. (The command-line compiler names are cl, icl, and ifort.) However, the Visual Studio environment is preferred because it doubles as a debugger.

##### Getting Started in Visual Studio

In Visual Studio, you begin by creating a solution that builds one or more projects. Roughly speaking, the projects within a solution are equivalent to the targets of a Linux makefile. Each separate project corresponds to one executable or one library that is to be built.

To start a new project that (e.g.) builds an executable, given one or more files of Fortran or C code:

• First select "File | New | Project...", then
• either "Intel Fortran | Console Application | Empty Project"
• or "Visual C++ | Win32 | Win32 Console Application"
• Next, add files containing source code to the project, using
• either "Project | Add | New Item..."
• or "Project | Add | Existing Item..."

A C/C++ project can later be converted to Intel C/C++ if desired.

Compilation is controlled through the Properties dialog, in the Project menu. This dialog is nothing more than a GUI for setting the command-line compiler flags. You can define multiple configurations for each project. The active configuration determines the particular set of flags that are passed to the compiler when the project is built. By default, there are two configurations named Debug and Release that are predefined for the Win32 platform, and Debug is the active configuration.

The problem is we don't generally want to use the Win32 platform; we want the x64 platform. Strangely, this choice does not appear anywhere in the Properties dialog until you click the Configuration Manager button (which is also available from "Build | Configuration Manager"). Under "Active solution platform", choose "New...", then "x64", then "OK" the changes you have made.

(Note: Unexpected problems with the Visual Studio interface can sometimes be cleared by starting it up with "devenv /resetsettings".)

##### Additional Steps for MPI Codes

For MPI codes, a few more steps are necessary in Properties. Set the Configuration to All Configurations and add the following:

• For x64 platform -
• C/C++ or Fortran | General | Additional Include Directories:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Include
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64
msmpi.lib

For Fortran codes, in addition to msmpi.lib, one of the msmpif*c.lib static libraries is required:

• msmpifec.lib supports Fortran compilers that put the string length of character variables at the very end of the parameter stack (e = end; most common case).
• msmpifmc.lib supports Fortran compilers that put the string length of character variables immediately after the corresponding string pointers on the parameter stack (m = mixed).

The Intel EM64T processors on the V4 Windows Cluster can run either 64-bit ("amd64") or 32-bit ("i386") applications, so it is possible to compile and link MPI codes for the Win32 platform, as well as x64. To build for Win32, only one minor change must be made to the above Properties:

• For Win32 platform -
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386
##### Compiling at the Command Prompt

There seems to be no convenient "mpicc" wrapper designed for use with the Microsoft C compiler and msmpi.lib. But if you prefer to compile your MPI code at the command prompt instead of within Visual Studio, it's not hard at all to pass a couple of options to cl.exe. First, fire up a shell in which you can run "cl" (the command-line name of the Microsoft C compiler):

Start > All Programs > Microsoft Visual Studio 2008 > Visual Studio Tools > Visual Studio 2008 x64 Cross Tools Command Prompt

In the shell window that appears, compile your code using "cl", simply supplying it with the include path to the MPI header file via the capital-I flag, and, after the /link flag, the /libpath to msmpi.lib, like this:

cl helloworld.c /I "C:\Program Files\Microsoft HPC Pack 2008 SDK\Include" /link
/libpath:"C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64" msmpi.lib


Be sure to pay careful attention to all spaces, colons, quotes, etc. in the above syntax. More information on the various options for cl can be found by typing "cl /?".

##### Linking Intel's Math Kernel Library

Similar steps to the above are involved for linking libraries other than MPI. For example, should one wish to link any of the LAPACK or BLAS routines in Intel's MKL 10 (with a choice of static linking, rather than dll's), here are the additions to make in the Visual Studio dialogs and tabs:

• For x64 platform -
• C/C++ | General | Additional Include Directories (optional for Fortran):
C:\Program Files\Intel\MKL\10.0.3.021\include
C:\Program Files\Intel\MKL\10.0.3.021\em64t\lib

Linking to MKL has become rather complicated due to Intel's decision to maximize MKL's flexibility and multi-platform compatibility by splitting off multithreading into four separate layers of libraries. The four libraries listed above correspond to the four essential layers: interface layer, threading layer, computational layer, and runtime (OpenMP) layer. For more help, including examples of using both static and dynamic linking in various ways, refer to the Intel MKL User Guide, C:\Program Files\Intel\MKL\10.0.3.021\doc\userguide.pdf, Chapter 5.

#### Changing Python version

We have a few versions of Python on the system.

• C:\Python25epd - Enthought Python 2.5.4 is the default. It has a large list of helpful libraries that are already tested.
• C:\Python25 - For anyone who might prefer to use the python.org distribution of Python 2.5.1.
• C:\Python26 - For those who need 2.6 language features. Popular numeric libraries aren't available for this release.
• C:\Python30 - For those who need 3.0 language features. This new release changes the language. Most numeric libraries aren't available.

To choose a particular version of Python over the default, in a batch script or interactively, set the PYTHONHOME variable to the appropriate directory, then start up the command-line interpreter:

set PYTHONHOME=C:\Python30
C:\Python30\python myfile.py


If you fail to set PYTHONHOME, the executable will load its libraries from Python25epd.

In order to start IDLE, which is the integrated development environment (IDE) for all of the above versions of Python, enter:

%PYTHONHOME%\pythonw %PYTHONHOME%\Lib\idlelib\idle.pyw


The CAC Web site is here . There are many useful documents on the Support page at CAC documentation.

### Acknowledging CAC

When you publish a paper, make presentations, or are interviewed by the Cornell Chronicle, national news media, etc., please acknowledge the Center by including:

"This research was conducted with support from the Cornell University Center for Advanced Computing."

Alternatively, the full acknowledgement is:

"This research was conducted with support from the Cornell University Center for Advanced Computing, which receives funding from Cornell University, the National Science Foundation, and members of its Partner Program."

1. Account FAQ