# Getting Started

Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.

• Each password must have at least eight characters.
• Each password must contain at least three of the following four elements among its first eight characters:
- uppercase letters (English, A through Z)
- lowercase letters (English, a through z)
- special characters (for example, !, $, #, %) - digits (0 through 9) • Do not use a space in a password. Though technically allowed, it may be a source of confusion. • Do not form a password by appending a digit to a word--this type of password is easily guessed. • Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent. • New passwords should differ from the old by at least three characters. If you need additional ideas for creating a new password, Cornell IT has a password generation recipe. ### Change a password at first login When you are issued a CAC user name, you should first log in to a login node. You will be prompted to change your password. You do this just once, and it should change your password for all CAC resources that require a login. Refer to the rules for creating passwords. After you change your password, you will be logged in. Windows users: Log in to the Windows login node, winlogin.cac.cornell.edu, using the Remote Desktop Connection client that comes with Windows. (There is also a free client for macOS that you can download from the App Store.) Here are the steps to follow in the RDC client: • Log in with your CAC user name and your current password (the domain name should be CTC_ITH) • From the desktop, choose "Windows Security" from the Start menu; or, if you are coming from Windows, enter the key combination Ctrl-Alt-End • Choose "Change a password..." and enter your old and new passwords as indicated. Old Password: 0ldpassw0rd!! New Password: newpassw0rd!! Confirm New Password: newpassw0rd!!  Linux and Mac users: You can follow these steps on the head node of a private cluster; if your CAC project doesn't have a private cluster, you can follow them on linuxlogin.cac.cornell.edu. On many Linux clusters, when you first change your password, you will also be asked for an ssh passphrase. You can leave this blank--just hit the Enter key. Assume that you have an old password '0ldpassw0rd!!' and a new password 'newpassw0rd!!'. This what should happen when you do the following in a Terminal client window: $ ssh your_username@linuxlogin.cac.cornell.edu
Retype new UNIX password: (ENTER newpassw0rd!!)
passwd: all authentication tokens updated successfully.
Connection to linuxlogin closed. 

If you get a token error it very likely means that the password is not complex enough. Your password must be a mix of any three of the following: lower case letters, upper case letters, numbers and some sort of punctuation to create an 8 character or longer password (it is slightly more complex; don't use your user name or previous password - more info in Password Policy ).

If you have additional trouble, you can use either the rdesktop client (for linux) or the Remote Desktop client (for Windows or Mac) to log into the Windows login node winlogin.cac.cornell.edu, then follow the instructions for Windows above. This gives you better information about password complexity issues during the password change.

### Change password at any time

You can change your CAC password before it expires. You will want to do so if you feel that your password has been compromised in any way. For example, suppose you think that someone else knows your password or you are concerned that you issued your password in a nonsecure setting that would have led to sending it in clear text. To change it:

• Be sure that you have no other open connections to any CAC resources.
- The only open interactive session should be the one in which you are changing the password.
- Log off all sessions connected to login nodes.
- Log off all remote connections to other CAC machines.
- It is not enough to disconnect the active sessions; you must log off. Failure to do so will lead to the system locking your account.
- Disconnect locally mapped drives to the CAC file server. Again, if this is not done, the system will automatically lock your account.
- Use Remote Desktop Connection to connect to winlogin, or an SSH client to connect to linuxlogin or to a Linux head node (for a private cluster).

• On winlogin, open a command prompt window (cmd), then issue the command
 net user <your CAC user name> /domain

Look for the line "Password expires".
• There is no equivalent command on the Linux login nodes.

Please contact CAC Help by submitting a ticket on our issue tracking system or by calling 607-254-8691.

### Locked Accounts

There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:

Solution: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!
• Trying to login to a Windows login node by using SSH when you have a new or expired password.
Solution: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.
• Failing to log off all other sessions connected to login nodes.
Solution: Log off all remote connections. Disconnecting the sessions is not enough.
• Failing to disconnect locally mapped drives to the CAC file server before changing your password.
Solution: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.

If you can't log on or can't wait you can submit a Password Reset ticket on our issue tracking system

Cornell University users can view their account limits at CAC Account Limits.
Partner Program members should contact Paul Redfern at red@cac.cornell.edu if they need information on their membership limits.

## Using CAC resources

### Connecting to CAC

There are two types of login nodes:

• Linux login nodes: linuxlogin.cac.cornell.edu as well as the head nodes for the various Linux-based private clusters.

#### Connect to Linux

These instructions are written primarily for users trying to log in to a CAC login node, taking linuxlogin as an example; however, the same methods should work for nearly any remote Linux machine. Note: If you are trying to connect to a Red Cloud Linux instance, please see the specific connection instructions in the documentation first. There are also troubleshooting steps to help you if you get stuck.

There are three distinct ways to connect to a remote Linux machine:

1. Use SSH to open a Linux shell on a login node, which provides a text-only interface.
2. Use SSH together with X-Windows, which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.
3. Use VNC to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.

These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.

Whichever method you choose, at your first login, you will be challenged for a new password (this does not apply to Red Cloud resources). Find help at Change a Password at First Login.

#### Using Secure Shell

For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.

• Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.
• If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.
• The non-secure predecessor of SSH, telnet, is disabled for security reasons.

Linux users:

To connect to the CAC general login node with ssh, you simply open a terminal window and type

localhost$ssh <your_CAC_username>@linuxlogin.cac.cornell.edu  Mac users: macOS is built on a version of Unix, so ssh is available directly from the Terminal application. • One option is to use the shortcut cmd-space to open Spotlight and then type "Terminal" to open a Terminal window. Otherwise: • Navigate in the Finder to the Applications folder and Utilities sub-folder. • Then double-click on the Terminal application to see a Bash command-line. • As in Linux, simply type "ssh <your_CAC_username>@linuxlogin.cac.cornell.edu" into this window. Windows users: Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free PuTTY client. • The simplest installation is to download the Windows installer (having the file extension .msi) and run it. This installs PuTTY into your Start menu. • To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open". • Tip for advanced users: a slight inconvenience of PuTTY is that in order to use passwordless SSH, your private key must first be converted into PuTTY's special "PPK" format using PuTTYgen. When choosing an SSH client, one consideration is how well other SSH-related tools are integrated, such as SCP (secure copy) and SFTP (secure file transfer protocol). With PuTTY, you can use the separate PSFTP client or PSCP command to transfer files back and forth. But PuTTY is just one choice; other clients for Windows exist. • MobaXterm has a free Home Edition. It gives you not just SSH, but integrated SFTP/SCP, X-Windows, VNC desktop, and quite a few other useful connectivity tools, all within one convenient client. • Token2Shell is a non-free commercial product available from the Microsoft Store. It provides SSH along with SFTP/SCP; however, it does not have X-Windows or VNC. A completely different approach is to create a self-contained Linux environment within Windows and use the usual Linux commands. • Windows Subsystem for Linux (WSL), free from Microsoft, allows you to run Ubuntu or another popular Linux distro within Windows 10 or Windows Server 2019. This allows you to use command-line SSH, SFTP, and SCP plus many other useful Linux tools. WSL does not come with an X server, but if you install one for Windows (see below), you can even install and run a VNC client for Linux in WSL. • Cygwin/X is free and open-source. It includes an xterm within which you can run OpenSSH commands such as SSH, SFTP, and SCP. Unlike WSL, it includes an X server. While it does not provide a VNC client, plenty of native-Windows VNC clients are available (see below). #### Using X-Windows X-Windows (also called X11) is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH. Appropriate use: Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does NOT imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. On linuxlogin, such usage is contrary to CAC policy. On other shared resources, it is disrespectful toward other users because the login node may become unresponsive through your actions. Linux users: The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the -X option, it will automatically set up the necessary tunnel and environment variables. localhost$ ssh -X <your_CAC_username>@linuxlogin.cac.cornell.edu
linuxlogin$echo$DISPLAY
localhost:11.0
linuxlogin$gs # on another system, you want to might try a different X-capable client, like xterm  If all goes well, you should see a valid setting for your DISPLAY environment variable, then have a blank window presented to you by gs (Ghostscript, the PostScript and PDF previewer). If not, check the target system to make sure xorg-x11-xauth has been installed. Note, if gs is not installed on the machine you're logging into, you can try another X client such as xterm, xclock, xlogo, emacs, etc. The -Y option is much like -X, but it uses a trusted version of X-windows forwarding. cluster-login-node$ ssh -Y compute-1-37


When you're working on a cluster, the trusted (-Y) version is necessary for forwarding X11 connections in two steps: from a compute node to the login node, then back to your client machine.

Mac users:

In order to use X-Windows on a Mac, an X11 server needs to be installed on the system. The recommended X11 server for use on a Mac is provided by the XQuartz project. XQuartz used to be included with Mac OS X installations (versions 10.5 through 10.7), but is no longer included and must be downloaded and installed manually. After installing XQuartz, be sure to restart your Mac before using X11.

Once XQuartz is installed you should start ssh with the -X or -Y option, which will cause the X11 server to start automatically on your Mac. You can then try the "gs" test in the shell, as described above for Linux.

Windows users:

A few of the SSH clients mentioned above come with a bundled X server. Otherwise, along with your SSH client (e.g., PuTTY), you will generally need to install an X-Windows server on your Windows machine.

• VcXsrv - Open Source, and still being actively maintained. Works with Windows 10. Note that freeware solutions like this one can often work very well, but as always, the installation and use of such packages comes with no guarantees.
• Xming - Open Source/Proprietary. Even though the public domain release of Xming is quite old, it still works fine with Windows 10. For a donation, you can download a more up-to-date "website release" with improved performance for graphics (GLX) and other enhancements. As described at the author's website, there are two pieces to download:
• Xming or Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.
• Xming-fonts (public domain release)

If you purchase the website release of Xming, remember to install the Xming-fonts, as well.

Here are some other X-server possibilities for Windows:

• Cygwin/X - Open Source. Cygwin is much more that just an X-Windows server. It actually creates an entire Linux-like environment within Windows.
• X410 - Commercial product from the same company that produces Token2Shell, available from the Microsoft Store.

Here is how to start a X-Windows-capable session using PuTTY and either VcXsrv or Xming.

1. Start VcXsrv or Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray. (Note, the first time you start VcXsrv, you will need to do a few configuration steps.)
2. Start PuTTY.
3. In the window that appears, type a host name, e.g., linuxlogin.cac.cornell.edu.
4. Use the tree menu on the left to set X11 forwarding. It's in the Connection > SSH branch.
5. You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.
6. Click Open, and it will connect to the CAC general login node.
7. Note for PuTTY 0.61 and above - If an "Access denied" message appears in your terminal window for no good reason, you can prevent this annoyance in future sessions by going to the "GSSAPI" area in the "Auth" section of the SSH branch, and unchecking the "Attempt GSSAPI authentication" box there.

Whichever Windows client and X server you choose, you should test your X-Windows setup by typing the command for Ghostscript, which is a PostScript and PDF previewer:

linuxlogin$gs # on another system, you want to might try a different X-capable client, like xterm  A blank window should appear on your screen. You can stop it by typing Ctrl-c in the terminal window. If this test fails, check to make sure xorg-x11-xauth has been installed on the target system. Also, if you are using a Linux-like shell in Windows (WSL or Cygwin/X), there are a couple of other things to check: • Make sure you have specified ssh -X or ssh -Y as necessary. • Type echo$DISPLAY in your shell to make sure this environment variable is set locally; if not, enter the following command (which you can add to your .bashrc)
export DISPLAY=localhost:0.0


#### Using VNC

VNC lets you see a whole Linux desktop on a remote computer from your local computer. Connecting to Linux via SSH and X-Windows is efficient in that it uses a lot less of the remote computer's resources, but VNC can be much faster if you are doing visualization on the remote computer from off campus.

TigerVNC server has been installed on linuxlogin so you can try VNC there. If you would like to use VNC on a private cluster managed by CAC, please ask your PI to request the VNC installation. Note, a Linux desktop manager is a required part of a VNC installation; GNOME is often a good choice (as is xfce, for those who prefer a minimal desktop).

For security reasons, CAC requires all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the remote computer using SSH. The firewalls running at CAC for all login nodes (e.g., linuxlogin) commonly block all incoming ports except for ssh, so VNC connections must be made over a ssh tunnel as described below.

Appropriate use

VNC gives you the ability to establish a remote desktop on a login node, but this ability does NOT imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. On linuxlogin, such usage is contrary to CAC policy. On other shared resources, it is disrespectful toward other users because the login node may become unresponsive through your actions.

Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.

1. Open a VNC connection to the login node through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.
2. In one of the terminal windows, submit an interactive job to the queue of your choice.
3. Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).
4. Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"
5. This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".

Local setup for all platforms

• Install a local VNC client on your local machine if one isn't installed already. For Linux, TigerVNC is a popular choice. On Mac, you can use the built-in Screen Sharing app. For Windows, TightVNC works well, but so do others.

Starting your remote VNC server (do these steps from an ssh shell)

• Set the password for your VNC server using the vncpasswd command.
• Start the VNC server using the vncserver command like this:
 vncserver -geometry 1600x900 -localhost


The geometry numbers 1600x900 (or other numbers of your choosing) specify the size of the desktop in pixels.

• You will need to get the display number from the output of the vncserver command:
 New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1
Starting applications specified in /home/fs01/shl1/.vnc/xstartup

• vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.

Set up your ssh tunnel (Do these steps on your local computer)

Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. Here's what to do to start ssh port forwarding, or tunneling, to that port.

• From Linux, if you are still in the same ssh session you used to start vncserver, type ~ then C to get to an ssh> prompt, then enter:
 -L 10000:localhost:5901


Or you can type the following full command into a local terminal, which opens another ssh session including the tunnel:

 ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu

• From macOS, open a Terminal and follow one of the Linux methods shown above.
• From Windows, ssh clients such as PuTTY can do port forwarding (tunneling); see VNC Tunnel Windows.
• Leave this ssh session running on your local client computer. (It can run in the background.)

• Launch your VNC client program. Connect it to localhost:10000. When prompted, type in your VNC server password.
• A nice GNOME desktop should appear!
• If a pop-up asks you to authenticate, just cancel it. See this link for how to prevent the annoying "Authenticate" pop-up from appearing in your future vncserver sessions.

• Close the VNC client program.
• Disconnect the ssh forwarding session (i.e., kill it).

• Restart port forwarding with ssh, using the same remote port number as before.
• Again connect the VNC client to localhost:10000.

When you are all done

• It may not be possible to to log out from the Linux desktop. But even if this appears to work, it will leave the VNC server running.
• When you are finished with your session, shut down all your applications in the desktop, disconnect (close) it, and type this command into a separate ssh session to shut down the VNC server completely:
 vncserver -kill :<display number>


An alternative to password-based authentication is public key authentication (PKA). SSH has a well-established mechanism for making use of a public/private key pair.

Note: setting this up for yourself is completely optional! On CAC private clusters where an SSH key pair is required for intra-cluster communications, one will be created for you when first log in to the head node, and you never need to worry about it.

How it works

• When you connect via SSH, instead of entering a password, you provide the name of an identity file on your computer. This is your private key, part of a public/private key pair.
• The computer you are connecting to must already have the matching public key stored in a special location. On Linux systems, it should be found among the list of public keys in the file ~/.ssh/authorized_keys.
• Upon receiving your initial SSH request, the remote computer encrypts a message using one of the public keys in ~/.ssh/authorized_keys. It sends the encrypted message to your computer.
• Your local SSH client attempts to decrypt this message using the private key file you specified. The decrypted message is then sent back to the remote computer.
• The remote computer checks whether your client succeeded in decrypting the message. If so, you have proven your identity. If not, it tries the next public key until all are exhausted.

Clearly, if you want to make use of this mechanism, you will need to set up a public/private key pair!

Create an SSH key pair

mkdir .ssh
chmod 700 .ssh
cd .ssh
ssh-keygen (and take all default options)
cat id_rsa.pub >> authorized_keys


As a final step, you need to copy the private key, id_rsa, to the computer that you will be logging in from.

Linux and macOS users:

You'll want to put the private key in the ~/.ssh directory. You may wish to call this key id_rsa for convenience, BUT be careful not to overwrite an existing key with that name. Let's say you decide instead to call your new private key cac_id_rsa on your local computer, just to be safe. You MUST change permissions on this key to keep it private:

chmod 600 ~/.ssh/cac_id_rsa


Now you're ready to use passwordless SSH to connect to linuxlogin. From a terminal in macOS or Linux, you do the following:

ssh -i ~/.ssh/cac_id_rsa <your_CAC_username>@linuxlogin.cac.cornell.edu


Windows users:

The way to proceed in Windows depends on the SSH client you are using. Here, we cover PuTTY as an example. The first step is to use PuTTYgen to convert the SSH private key for use with PuTTY and plink.

• Run "C:\Program Files (x86)\Putty\puttygen.exe".
• Select Import Key from the Conversions menu and enter C:\Users\<your_local_username>\.ssh in the address bar, assuming this is where you placed your private key in your home directory. Then, select the id_rsa file and click on the Open button.
• Click on the "Save Private Key" button.
• Click on "Yes" when asked to save the private key without a passphrase.
• Save the private key as private.ppk in the .ssh directory inside your home directory.
• Close (choose File, then Exit)
• To confirm you have converted the ssh private key successfully, do:
"C:\Program Files (x86)\Putty\plink.exe" -i %USERPROFILE%\.ssh\private.ppk <your_CAC_username>@linuxlogin.cac.cornell.edu

It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."

• If everything was done correctly, you should now be logged into linuxlogin without being prompted for a password. Type exit to log out.

In PuTTY, you will want to update your Saved Session for linuxlogin to use the new key. Load the linuxlogin session. Navigate to "Connection > SSH > Auth", browse to %USERPROFILE%\.ssh and open the id_rsa file. Go back to Session and click Save. Now you won't need to enter a password for linuxlogin ever again!

#### Connect to Windows

Using Remote Desktop Connection to connect to winlogin

This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to CTC_ITH, not the local name of the machine to which you are connecting.

Remote Desktop sessions do not expire unless you log out, but they will end when machines are rebooted during down times.

• If you use a Windows machine:
Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, find Windows Accessories in your list of apps, then click Remote Desktop Connection.
• If you use macOS:
Search from Microsoft Remote Desktop at the App Store. it works just like the Remote Desktop Connection in Windows 7 and later. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.
• If you use Unix or Linux (or Mac):
You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from rdesktop.

### Home Directory Access

Users of CAC's storage services have a "storage03" directory which can be accessed from both Linux and Windows systems:

Note, private clusters often have their own file servers, so users of private clusters may find that the linuxlogin path is not the same as their home directory on their private clusters. Also, Red Cloud users do not automatically have storage privileges on storage03, unless such storage is included in their CAC project.

localhost$scp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat  ##### Secure FTP FTP is disabled for security reasons, but sftp's interface is nearly identical. ##### Samba Client This technique only works from Cornell campus locations or via a Cornell VPN connection. Type smbclient //storage03.cac.cornell.edu/<user name> -U ctc_ith\\<user name>  (Note, the shell interprets \\ as a single backslash.) Enter the password for your CAC account when prompted. You will see the smb:\> prompt. Now you can start transferring files between your local machine and your CAC home directory, using commands similar to the sftp client. Type help for more instructions.  -bash-4.1$ smbclient //storage03.cac.cornell.edu/<user name> -U ctc_ith\\<user name>
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.6.23-24.el6_7]
smb: \> help


#### Windows users

##### Secure Copy

The individual who created PuTTY provides a secure copy client called pscp. From the command prompt, type:

cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat

##### Secure FTP

FTP is disabled for security reasons, but psftp's interface is nearly identical. From the command prompt, type:

cmd> psftp username@linuxlogin.cac.cornell.edu
psftp> put localresults.dat results.dat
psftp> quit


### Linux Usage Tips

#### Linux shells

• /bin/sh is the default login shell.
• Edit $HOME/.profile to change interactive variables. • The$HOME/.bashrc file will not be run for non-interactive shells.
• /bin/bash
• Edit $HOME/.profile to change interactive variables. • The$HOME/.bashrc file will be run for non-interactive shells.
• /bin/csh and /bin/tcsh
• Edit $HOME/.login to change interactive variables. • The$HOME/.cshrc file will be run for non-interactive shells.

The change shell command, chsh, will not permanently change your shell. You must send a request instead. Contact Support

The default login shell on linuxlogin is sh. Be aware that in CentOS, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of bash. Accordingly, you will find that "man sh" brings up the man page (the help document) for bash. In a way, then, you can think of your login shell as being bash, too.

There are slight differences between sh and bash, however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both sh and bash; but any interactive sh shells you start thereafter will not run ~/.bashrc as you might expect from bash. The way to get sh to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).

Let's say you simply prefer to have bash as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be bash. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be bash (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is sh.

The relationship between the csh and tcsh shells is similar to the one between sh and bash. For instance, your csh shells are automatically endowed with the tcsh-style ability to retrieve history through the up- and down-arrow keys. The best way to make tcsh into your everyday working shell is to run it on top of sh after you log in (again, you can do this as part of your .profile).

References

#### Compiling and linking code on Linux

Use /tmp to compile large codes and software packages. This will provide improved performance and greater system stability.

If you want to know what processor features a cluster supports, submit a batch job that does "cat /proc/cpuinfo" in order to find out the CPU type. The v4 cluster is composed mostly of Intel E5420 CPUs (in Nov. 2011). Then you go to Wikipedia's Intel Xeon page or Intel's ARK to find that these are Harpertown cores that support SSE, SSE2, SSE3, SSSE3, SSE4.1 and VMX.

##### C/C++ and Fortran Codes
• GNU compilers gcc, g++, g77, gfortran are in /usr/bin, which is in the default path.
• For compiling OpenMP directives, add the option -fopenmp.
• Intel 12.1 compilers icc, ifort are in the default path on the login nodes.
• For compiling OpenMP directives, add the option -openmp.
• The following Intel libraries and tools are available to you automatically through the default setup on the login nodes:
- MKL, the Math Kernel Library 10.3.6 (additional help below)
- idb, the Intel debugger for Linux
- TBB, the Threading Building Blocks
- IPP, the Integrated Performance Primitives
• If any of the above libraries are linked dynamically, the correct runtimes will be loaded automatically on the compute nodes by default; no additional setup is required.
• Note - if you find that your code segfaults after compiling with Intel 12.1, try disabling optimization or using the older 11.1 version of the compilers instead.
Reason: there is a known bug in the vectorizer of the 12.1 compiler which is due to be fixed in a future release.
• Intel 11.1 compilers icc, ifort are available, also, but these older compilers require special setup files.
• Before compiling in bash: source /opt/intel/intel-11.sh
• Before compiling in tcsh: source /opt/intel/intel-11.csh
• At runtime, in a batch sh-script: source /opt/intel/intel-11.sh
• At runtime, in a batch csh-script: source /opt/intel/intel-11.csh
• The above steps also enable the use of the older Intel performance libraries, e.g., MKL 10.2 (additional information below).
• Help for Intel compilers (if you are using 11.1, be sure to source the setup file first):
• Fortran: man ifort, info ifort, ifort -help
• C/C++: man icc, info icc, icc -help
• Standard compiler options - the clusters have Intel Core2 processors, so standard compiler options are:
• For Intel: -O3 -ipo -mtune=pentium4 -march=pentium4
• Other options of possible interest (consult man pages):
• For Intel: -fno-alias -align -scalar_rep -prefetch
##### Generating Debugging Info
• Intel compilers
• icc -Wall
• ifort -g -debug -warn -C (-CB for bounds checking only)
##### MPI Programs

For compiling MPI codes, we recommend using mpicc and mpif90. If you specifically need a C++ compiler, try mpicxx. Because of these handy wrapper scripts, you may not need to do very much to convert existing makefiles to work with CAC's preferred software stack. Currently the default paths are set up so that the mpicc, mpif90 and mpicxx utilities invoke the Intel 12.1 compilers to compile your codes and link them properly to the Intel MPI 3.1 libraries. However, if you run the Intel 11.1 compiler setup file first, then these utilities will automatically use the older 11.1 compiler version. Documentation for the Intel MPI 3.1 Library, including mpdboot and mpiexec, is in PDF on the Intel Support Site.

To view a sample batch script that will run an MPI job for you, see the section on Running a parallel MPI job.

The ROCKS operating system comes with several alternate MPI implementations (e.g., mpich2, OpenMPI). You have to play with environment variables and paths to get them to work.

##### Intel MKL

Intel's Math Kernel Library (MKL) is a good source of optimized routines for linear algebra, Fast Fourier Transforms, vector math, and other mathematical operations. In particular, it provides a way to incorporate Intel's optimized BLAS and LAPACK routines into your code.

OpenMP multithreading is built into certain MKL libs. When these libs are linked, calls to MKL will detect the same settings that would affect any other OpenMP-enabled code. This means MKL will attempt to use all the cores present on a v4 node during the execution of parallelized sections. Therefore, when you link your code with a "_thread" version of the MKL library, your should realize that all your calls to MKL will generally fork the same number of threads as the number of cores present. This may cause undesired interference with other parallelization strategies you are using, e.g., MPI. If this is not the behavior you want, you can do one of two things:

• At run time, set the OMP_NUM_THREADS environment variable to 1. (Use "export" or "setenv".) A value of 8 recovers the default behavior on v4.
###### Linking Intel MKL 10.3.6 with the Intel 12.1 Compilers

MKL 10.3.6 is the version installed with the 12.1 compilers. The easiest way to link MKL is to compile as follows, where the last two lines pertain to MPI codes:

• icc mycode.c -o mycode -mkl
• ifort mympicode.c -o mycode -mkl
• mpicc mympicode.c -o mympicode -mkl
• mpif90 mympicode.f90 -o mympicode -mkl

Note, -mkl is the same as -mkl=parallel, which enables OpenMPI mulithreading. If you don't want this, use -mkl=sequential.

With just plain -mkl (or -mkl=...), the resulting executable will be dynamically linked. This means that at run time, your program has to know where to find the MKL shared libraries. Since MKL 10.3.6 is the default, the appropriate paths have been predefined for you on the compute nodes, and your batch jobs should have no trouble.

Should you want to link MKL in some different way--e.g., statically--the compile line will start looking messier. Linking to MKL has become rather complicated due to Intel's decision to maximize MKL's flexibility and multi-platform compatibility by splitting out four separate layers of libraries: interface, threading, computational, and runtime (meaning OpenMP, if the _thread lib is requested). To make sure you have all these layers, we recommend appending one of the following snippets to your ifort, icc, mpif90, or mpicc command (after first setting MKLPATH = $MKLROOT/lib/intel64): • static, multithreaded:$MKLPATH/libmkl_solver_lp64.a -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a$MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -openmp -lpthread • static, sequential:$MKLPATH/libmkl_solver_lp64_sequential.a -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a$MKLPATH/libmkl_sequential.a $MKLPATH/libmkl_core.a -Wl,--end-group -lpthread These options will generate a (mostly) statically linked executable. Note, each .a-lib must be identified by its full path in order to prevent the .so-lib (its dynamic equivalent) from being found instead. If you do not need access to the MKL solver routines, simply remove that item from the head of the list. As noted previously, if your main program is itself threaded with OpenMP, or if it is parallelized with MPI, you may want to select libmkl_sequential.a in order to reduce contention and get better performance. To generate a dynamically linked rather than statically linked executable, the above options become: • dynamic, multithreaded:$MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -openmp -lpthread
• dynamic, sequential:
$MKLPATH/libmkl_solver_lp64_sequential.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread These sets of options are pretty much equivalent to -mkl=parallel and -mkl=sequential, respectively. If your batch script needs strict control over LD_LIBRARY_PATH, then one other compiler/linker option may be helpful for a dynamically-linked code: • -Wl,-rpath,$MKLPATH,-rpath,$IOMPPATH The above variables should be set to MKLPATH =$MKLROOT/lib/intel64 and IOMPPATH = $MKLROOT/../compiler/lib/intel64. This option "hardwires" the correct paths into the executable; these paths are valid on both the v4 compute nodes and the v4 login nodes. If you don't wish to restrict your executable in this fashion, the alternative is to add these paths manually to LD_LIBRARY_PATH. Intel has put together a helpful tool for generating the correct linker options to match your specific needs, the Link Line Advisor. This is definitely the place to go if you want to, e.g., use extra-long integers or compile with gcc or gfortran. It's well worth a visit. Much more information on linking MKL 10.3.6 can be found in the "Linking Your Application" section of the User Guide, which you can access from the login node ("firefox /opt/intel/composer_xe_2011_sp1.6.233/Documentation/en_US/mkl/mkl_userguide/index.htm"). ###### Linking Intel MKL 10.2 with the Intel 11.1 Compilers MKL 10.2 is the version installed with the 11.1 compilers. Since 11.1 is not the current default version of the compilers, you must first source the setup file: • In bash (or sh): source /opt/intel/intel-11.sh • In tcsh (or csh): source /opt/intel/intel-11.csh If your program links MKL dynamically, it has to know where to find the correct MKL shared libraries at run time. Bear in mind that MKL 10.2 is not the default on the compute nodes, either. The easiest way to ensure correct behavior at run time is to put the same line into your batch script: • In a batch sh-script: source /opt/intel/intel-11.sh • In a batch csh-script: source /opt/intel/intel-11.csh Otherwise the instructions for linking MKL 10.2 are identical to the instructions for MKL 10.3.6 and Intel 12.1 above. There is one exception: you need to set MKLPATH =$MKLROOT/lib/em64t and IOMPPATH = \$MKLROOT/../lib/intel64.

The Link Line Advisor can be applied to older versions of the Intel compilers and MKL. It's well worth a visit.

Much more information on linking MKL 10.2 can be found in the Sec. 5 of the User Guide, which you can access from the login node ("firefox /opt/intel/Compiler/11.1/072/Documentation/en_US/mkl/userguide.pdf").

#### Linux FAQ

##### How do I determine my program's dependencies on shared library (.so) files?
• ldd - see the man page.

If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.

##### How do I display an image file (such as jpeg or gif)?
• display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.
• firefox mypic.jpg - any decent Web browser can handle it.

Note, the image will show up only if you have X11 forwarding enabled.

##### How do I use revision control?
• Git, Subversion, and CVS are examples of revision control (or version control or source control) software. These tools help you collaborate with others by allowing you to save and track successive versions of your source code as you modify it. Git is often used in conjunction with GitHub. Git is installed on linuxlogin: see the git man pages for for details. To check the installed version, type git --version.

For more detailed instructions on how to use the Linux node, see here

### Windows Usage Tips

This page contains information for CAC's former Windows HPC clusters, which are no longer in service.

#### Compiling and Linking Codes on Windows

The Windows login node provides the Microsoft Visual Studio 2005 and 2008 integrated development environments (IDEs) into which the following compilers are integrated: Microsoft C/C++, Intel C/C++ 11.0, and Intel Fortran 11.0, among others. If desired, any one of these compilers can be invoked from the command line in a Windows command shell (cmd) as well. (The command-line compiler names are cl, icl, and ifort.) However, the Visual Studio environment is preferred because it doubles as a debugger.

##### Getting Started in Visual Studio

In Visual Studio, you begin by creating a solution that builds one or more projects. Roughly speaking, the projects within a solution are equivalent to the targets of a Linux makefile. Each separate project corresponds to one executable or one library that is to be built.

To start a new project that (e.g.) builds an executable, given one or more files of Fortran or C code:

• First select "File | New | Project...", then
• either "Intel Fortran | Console Application | Empty Project"
• or "Visual C++ | Win32 | Win32 Console Application"
• Next, add files containing source code to the project, using
• either "Project | Add | New Item..."
• or "Project | Add | Existing Item..."

A C/C++ project can later be converted to Intel C/C++ if desired.

Compilation is controlled through the Properties dialog, in the Project menu. This dialog is nothing more than a GUI for setting the command-line compiler flags. You can define multiple configurations for each project. The active configuration determines the particular set of flags that are passed to the compiler when the project is built. By default, there are two configurations named Debug and Release that are predefined for the Win32 platform, and Debug is the active configuration.

The problem is we don't generally want to use the Win32 platform; we want the x64 platform. Strangely, this choice does not appear anywhere in the Properties dialog until you click the Configuration Manager button (which is also available from "Build | Configuration Manager"). Under "Active solution platform", choose "New...", then "x64", then "OK" the changes you have made.

(Note: Unexpected problems with the Visual Studio interface can sometimes be cleared by starting it up with "devenv /resetsettings".)

##### Additional Steps for MPI Codes

For MPI codes, a few more steps are necessary in Properties. Set the Configuration to All Configurations and add the following:

• For x64 platform -
• C/C++ or Fortran | General | Additional Include Directories:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Include
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64
msmpi.lib

For Fortran codes, in addition to msmpi.lib, one of the msmpif*c.lib static libraries is required:

• msmpifec.lib supports Fortran compilers that put the string length of character variables at the very end of the parameter stack (e = end; most common case).
• msmpifmc.lib supports Fortran compilers that put the string length of character variables immediately after the corresponding string pointers on the parameter stack (m = mixed).

The Intel EM64T processors on the V4 Windows Cluster can run either 64-bit ("amd64") or 32-bit ("i386") applications, so it is possible to compile and link MPI codes for the Win32 platform, as well as x64. To build for Win32, only one minor change must be made to the above Properties:

• For Win32 platform -
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386
##### Compiling at the Command Prompt

There seems to be no convenient "mpicc" wrapper designed for use with the Microsoft C compiler and msmpi.lib. But if you prefer to compile your MPI code at the command prompt instead of within Visual Studio, it's not hard at all to pass a couple of options to cl.exe. First, fire up a shell in which you can run "cl" (the command-line name of the Microsoft C compiler):

Start > All Programs > Microsoft Visual Studio 2008 > Visual Studio Tools > Visual Studio 2008 x64 Cross Tools Command Prompt

In the shell window that appears, compile your code using "cl", simply supplying it with the include path to the MPI header file via the capital-I flag, and, after the /link flag, the /libpath to msmpi.lib, like this:

cl helloworld.c /I "C:\Program Files\Microsoft HPC Pack 2008 SDK\Include" /link
/libpath:"C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64" msmpi.lib


Be sure to pay careful attention to all spaces, colons, quotes, etc. in the above syntax. More information on the various options for cl can be found by typing "cl /?".

##### Linking Intel's Math Kernel Library

Similar steps to the above are involved for linking libraries other than MPI. For example, should one wish to link any of the LAPACK or BLAS routines in Intel's MKL 10 (with a choice of static linking, rather than dll's), here are the additions to make in the Visual Studio dialogs and tabs:

• For x64 platform -
• C/C++ | General | Additional Include Directories (optional for Fortran):
C:\Program Files\Intel\MKL\10.0.3.021\include
C:\Program Files\Intel\MKL\10.0.3.021\em64t\lib

Linking to MKL has become rather complicated due to Intel's decision to maximize MKL's flexibility and multi-platform compatibility by splitting off multithreading into four separate layers of libraries. The four libraries listed above correspond to the four essential layers: interface layer, threading layer, computational layer, and runtime (OpenMP) layer. For more help, including examples of using both static and dynamic linking in various ways, refer to the Intel MKL User Guide, C:\Program Files\Intel\MKL\10.0.3.021\doc\userguide.pdf, Chapter 5.

#### Changing Python version

We have a few versions of Python on the system.

• C:\Python25epd - Enthought Python 2.5.4 is the default. It has a large list of helpful libraries that are already tested.
• C:\Python25 - For anyone who might prefer to use the python.org distribution of Python 2.5.1.
• C:\Python26 - For those who need 2.6 language features. Popular numeric libraries aren't available for this release.
• C:\Python30 - For those who need 3.0 language features. This new release changes the language. Most numeric libraries aren't available.

To choose a particular version of Python over the default, in a batch script or interactively, set the PYTHONHOME variable to the appropriate directory, then start up the command-line interpreter:

set PYTHONHOME=C:\Python30
C:\Python30\python myfile.py


If you fail to set PYTHONHOME, the executable will load its libraries from Python25epd.

In order to start IDLE, which is the integrated development environment (IDE) for all of the above versions of Python, enter:

%PYTHONHOME%\pythonw %PYTHONHOME%\Lib\idlelib\idle.pyw


The CAC Web site is here . There are many useful documents on the Support page at [/wiki CAC documentation].

### Acknowledging CAC

When you publish a paper, make presentations, or are interviewed by the Cornell Chronicle, national news media, etc., please acknowledge the Center by including:

"This research was conducted with support from the Cornell University Center for Advanced Computing."

Alternatively, the full acknowledgement is:

"This research was conducted with support from the Cornell University Center for Advanced Computing, which receives funding from Cornell University, the National Science Foundation, and members of its Partner Program."

1. Account FAQ