Getting Started on Private Clusters
- 1 Managing your password
- 2 Checking your CAC project
- 3 Connecting to CAC resources
- 4 Working with CAC file storage
- 5 Linux usage tips
- 5.1 Linux shells
- 5.2 Compiling and linking code on Linux
- 5.3 Linux FAQ
- 6 More information
- 7 Acknowledging CAC
- 8 FAQ/Troubleshooting
Managing your password
CAC has a password policy in effect. The first time that you login to the cac.cornell.edu domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.
Rules for creating passwords
Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.
When you change your password, the new password must comply with our password complexity policy:
- Each password must have at least eight characters.
- Each password must contain at least three of the following four elements among its first eight characters:
- - uppercase letters (English, A through Z)
- - lowercase letters (English, a through z)
- - special characters (for example, !, $, #, %)
- - digits (0 through 9)
- Do not use a space in a password. Though technically allowed, it may be a source of confusion.
- Do not form a password by appending a digit to a word--this type of password is easily guessed.
- Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent.
- New passwords should differ from the old by at least three characters.
If you need additional ideas for creating a new password, Microsoft has a few tips.
Change a password at first login
When you are issued a CAC user name, you should first log in to a login node. You will be prompted to change your password. You do this just once, and it should change your password for all CAC resources that require a login. Refer to the rules for creating passwords. After you change your password, you will be logged in.
Log in to the Windows login node, winlogin.cac.cornell.edu, using the Remote Desktop Connection client that comes with Windows. (There is also a free client for macOS that you can download from the App Store.) Here are the steps to follow in the RDC client:
- Log in with your CAC user name and your current password (the domain name should be CTC_ITH)
- From the desktop, choose "Windows Security" from the Start menu; or, if you are coming from Windows, enter the key combination Ctrl-Alt-End
- Choose "Change a password..." and enter your old and new passwords as indicated.
Old Password: 0ldpassw0rd!! New Password: newpassw0rd!! Confirm New Password: newpassw0rd!!
Linux and Mac users:
You can follow these steps on the head node of a private cluster; if your CAC project doesn't have a private cluster, you can follow them on linuxlogin.cac.cornell.edu. On many Linux clusters, when you first change your password, you will also be asked for an ssh passphrase. You can leave this blank--just hit the Enter key.
Assume that you have an old password '0ldpassw0rd!!' and a new password 'newpassw0rd!!'.
This what should happen when you do the following in a Terminal client window:
$ ssh email@example.com Password: (ENTER 0ldpassw0rd!!) WARNING: Your password has expired. You must change your password now and login again! Changing password for user your_username. Kerberos 5 Password: (ENTER 0ldpassw0rd!!) New UNIX password: (ENTER newpassw0rd!!) Retype new UNIX password: (ENTER newpassw0rd!!) passwd: all authentication tokens updated successfully. Connection to linuxlogin closed.
If you get a token error it very likely means that the password is not complex enough. Your password must be a mix of any three of the following: lower case letters, upper case letters, numbers and some sort of punctuation to create an 8 character or longer password (it is slightly more complex; don't use your user name or previous password - more info in Password Policy ).
If you have additional trouble, you can use either the rdesktop client (for linux) or the Remote Desktop client (for Windows or Mac) to log into the Windows login node winlogin.cac.cornell.edu, then follow the instructions for Windows above. This gives you better information about password complexity issues during the password change.
Change password at any time
You can change your CAC password before it expires. You will want to do so if you feel that your password has been compromised in any way. For example, suppose you think that someone else knows your password or you are concerned that you issued your password in a nonsecure setting that would have led to sending it in clear text. To change it:
- Be sure that you have no other open connections to any CAC resources.
- - The only open interactive session should be the one in which you are changing the password.
- - Log off all sessions connected to login nodes.
- - Log off all remote connections to other CAC machines.
- - It is not enough to disconnect the active sessions; you must log off. Failure to do so will lead to the system locking your account.
- - Disconnect locally mapped drives to the CAC file server. Again, if this is not done, the system will automatically lock your account.
- Log in to one of the CAC login nodes.
- - Use Remote Desktop Connection to connect to winlogin, or an SSH client to connect to linuxlogin or to a Linux head node (for a private cluster).
- - Follow the same steps as you would to change a password at first login.
After you change your password, you will be logged in.
If your password has expired
Your password will expire after six months or 185 days. About a week before your password expires, you will be asked if you want to change it. You can do it then or wait until it expires. If your password has expired, you will be prompted to change it. Follow the instructions to do so, using the same procedure as in change a password at first login.
After you change your password, you will be logged in.
Password expiration date
To see when your password expires:
- On winlogin, open a command prompt window (cmd), then issue the command
net user <your CAC user name> /domain
- Look for the line "Password expires".
- There is no equivalent command on the Linux login nodes.
If you forget or lose your password
Please contact CAC Help by submitting a ticket on our issue tracking system or by calling 607-254-8691.
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:
- Mistyping your password several times in a row.
- Solution: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!
- Trying to login to a Windows login node by using SSH when you have a new or expired password.
- Solution: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.
- Failing to log off all other sessions connected to login nodes.
- Solution: Log off all remote connections. Disconnecting the sessions is not enough.
- Failing to disconnect locally mapped drives to the CAC file server before changing your password.
- Solution: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.
If you can't log on or can't wait you can submit a Password Reset ticket on our issue tracking system
Checking your CAC project
Cornell University users can view their account limits at CAC Account Limits.
Partner Program members should contact Paul Redfern at firstname.lastname@example.org if they need information on their membership limits.
Connecting to CAC resources
The information on the remainder of this page is primarily for users of CAC-maintained Private Clusters (for particular research groups) and data storage services, especially CAC's online, non-archival data storage.
However, it is potentially of interest to users of Red Cloud and other CAC services.
CAC login nodes
There are three types of login nodes:
- The head nodes for the various Linux-based private clusters
- Linux login node: linuxlogin.cac.cornell.edu
- Windows login node: winlogin.cac.cornell.edu
The general CAC login nodes, linuxlogin and winlogin, are mostly intended for researchers who are have procured CAC storage services, apart from Red Cloud and private clusters (see Working with CAC file storage). These two login nodes are broadly accessible from the Internet, and they provide a convenient way for researchers to gain access to their files. On the general login nodes, you will find a modest number of software tools installed to aid in working with files (but please do not use these tools for doing production computing there).
In what follows, we will often use linuxlogin and winlogin as stand-ins for the particular CAC servers that you are trying to access (e.g., in a private cluster or in Red Cloud).
Connect to Linux
These instructions are written primarily for users trying to log in to a CAC login node, taking linuxlogin as an example; however, the same methods should work for nearly any remote Linux machine. Note: If you are trying to connect to a Red Cloud Linux instance, please see the specific connection instructions in the documentation first. There are also troubleshooting steps to help you if you get stuck.
There are three distinct ways to connect to a remote Linux machine:
- Use SSH to open a Linux shell on a login node, which provides a text-only interface.
- Use SSH together with X-Windows, which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.
- Use VNC to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.
Whichever method you choose, at your first login, you will be challenged for a new password (this does not apply to Red Cloud resources). Find help at Changing a Password at First Login. You will also be asked for an ssh passphrase. You can just leave this blank; hit the Enter key in response.
Using Secure Shell
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.
- Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.
- If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.
- The non-secure predecessor of SSH, telnet, is disabled for security reasons.
To connect to the CAC general login node with ssh, you simply open a terminal window and type
localhost$ ssh <your_CAC_username>@linuxlogin.cac.cornell.edu
macOS is built on a version of Unix, so ssh is available directly from the Terminal application.
- One option is to use the shortcut cmd-space to open Spotlight and then type "Terminal" to open a Terminal window.
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free PuTTY client.
- The simplest installation is to download the Windows installer (having the file extension .msi) and run it. This installs PuTTY into your Start menu.
- To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".
- Tip for advanced users: a slight inconvenience of PuTTY is that in order to use Passwordless SSH, your private key must first be converted into PuTTY's special "PPK" format using PuTTYgen.
When choosing an SSH client, one consideration is how well other SSH-related tools are integrated, such as SCP (secure copy) and SFTP (secure file transfer protocol). With PuTTY, you can use the separate PSFTP client or PSCP command to transfer files back and forth. But PuTTY is just one choice; other clients for Windows exist.
- MobaXterm has a free Home Edition. It gives you not just SSH, but integrated SFTP/SCP, X-Windows, VNC desktop, and quite a few other useful connectivity tools, all within one convenient client.
- Token2Shell is a non-free commercial product available from the Microsoft Store. It provides SSH along with SFTP/SCP; however, it does not have X-Windows or VNC.
A completely different approach is to create a self-contained Linux environment within Windows and use the usual Linux commands.
- Windows Subsystem for Linux (WSL), free from Microsoft, allows you to run Ubuntu or another popular Linux distro within Windows 10 or Windows Server 2019. This allows you to use command-line SSH, SFTP, and SCP plus many other useful Linux tools. WSL does not come with an X server, but if you install one for Windows (see below), you can even install and run a VNC client for Linux in WSL.
- Cygwin/X is free and open-source. It includes an xterm within which you can run OpenSSH commands such as SSH, SFTP, and SCP. Unlike WSL, it includes an X server. While it does not provide a VNC client, plenty of native-Windows VNC clients are available (see below).
X-Windows (also called X11) is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does NOT imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. On linuxlogin, such usage is contrary to CAC policy. On other shared resources, it is disrespectful toward other users because the login node may become unresponsive through your actions.
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the -X option, it will automatically set up the necessary tunnel and environment variables.
localhost$ ssh -X <your_CAC_username>@linuxlogin.cac.cornell.edu linuxlogin$ echo $DISPLAY localhost:11.0 linuxlogin$ gs
If all goes well, you should see a valid setting for your DISPLAY environment variable, then have a blank window presented to you by gs (Ghostscript, the PostScript and PDF previewer). If not, check the target system to make sure xorg-x11-xauth has been installed. Note, if gs is not installed on the machine you're logging into, you can try another X client such as xclock, xlogo, emacs, etc.
The -Y option is much like -X, but it uses a trusted version of X-windows forwarding.
cluster-login-node$ ssh -Y compute-1-37
When you're working on a cluster, the trusted (-Y) version is necessary for forwarding X11 connections in two steps: from a compute node to the login node, then back to your client machine.
In order to use X-Windows on a Mac, an X11 server needs to be installed on the system. The recommended X11 server for use on a Mac is provided by the XQuartz project. XQuartz used to be included with Mac OS X installations (versions 10.5 through 10.7), but is no longer included and must be downloaded and installed manually. After installing XQuartz, be sure to restart your Mac before using X11.
Once XQuartz is installed you should start ssh with the -X or -Y option, which will cause the X11 server to start automatically on your Mac. You can then try the "gs" test in the shell, as described above for Linux.
A few of the SSH clients mentioned above come with a bundled X server. Otherwise, along with your SSH client (e.g., PuTTY), you will generally need to install an X-Windows server on your Windows machine.
- VcXsrv - Open Source, and still being actively maintained. Works with Windows 10. Note that freeware solutions like this one can often work very well, but as always, the installation and use of such packages comes with no guarantees.
- Xming - Open Source/Proprietary. Even though the public domain release of Xming is quite old, it still works fine with Windows 10. For a donation, you can download a more up-to-date "website release" with improved performance for graphics (GLX) and other enhancements. There are two pieces to download:
- Xming or Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.
- Xming-fonts (public domain release)
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.
Here are some other X-server possibilities for Windows:
Here is how to start a X-Windows-capable session using PuTTY and either VcXsrv or Xming.
- Start VcXsrv or Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray. (Note, the first time you start VcXsrv, you will need to do a few configuration steps.)
- Start PuTTY.
- In the window that appears, type a host name, e.g., linuxlogin.cac.cornell.edu.
- Use the tree menu on the left to set X11 forwarding. It's in the Connection > SSH branch.
- You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.
- Click Open, and it will connect to the CAC general login node.
- Note for PuTTY 0.61 and above - If an "Access denied" message appears in your terminal window for no good reason, you can prevent this annoyance in future sessions by going to the "GSSAPI" area in the "Auth" section of the SSH branch, and unchecking the "Attempt GSSAPI authentication" box there.
Whichever Windows client and X server you choose, you should test your X-Windows setup by typing the command for Ghostscript, which is a PostScript and PDF previewer:
A blank window should appear on your screen. You can stop it by typing Ctrl-c in the terminal window.
If this test fails, check to make sure xorg-x11-xauth has been installed on the target system. Also, if you are using a Linux-like shell in Windows (WSL or Cygwin/X), there are a couple of other things to check:
- Make sure you have specified ssh -X or ssh -Y as necessary.
- Type echo $DISPLAY in your shell to make sure this environment variable is set locally; if not, enter the following command (which you can add to your .bashrc)
VNC lets you see a whole Linux desktop on a remote computer from your local computer. Connecting to Linux via SSH and X-Windows is efficient in that it uses a lot less of the remote computer's resources, but VNC can be much faster if you are doing visualization on the remote computer from off campus.
TigerVNC server has been installed on linuxlogin so you can try VNC there. If you would like to use VNC on a private cluster managed by CAC, please ask your PI to request the VNC installation. Note, a Linux desktop manager is a required part of a VNC installation; GNOME is often a good choice (as is xfce, for those who prefer a minimal desktop).
For security reasons, CAC requires all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the remote computer using SSH. The firewalls running at CAC for all login nodes (e.g., linuxlogin) commonly block all incoming ports except for ssh, so VNC connections must be made over a ssh tunnel as described below.
VNC gives you the ability to establish a remote desktop on a login node, but this ability does NOT imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. On linuxlogin, such usage is contrary to CAC policy. On other shared resources, it is disrespectful toward other users because the login node may become unresponsive through your actions.
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.
- Open a VNC connection to the login node through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.
- In one of the terminal windows, submit an interactive job to the queue of your choice.
- Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).
- Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"
- This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".
Local setup for all platforms
Starting your remote VNC server (do these steps from an ssh shell)
- Use ssh to log in to the remote Linux computer.
- Set the password for your VNC server using the vncpasswd command.
- Start the VNC server using the vncserver command like this:
vncserver -geometry 1024x768 -localhost
The geometry numbers 1024x768 (or other numbers of your choosing) specify the size of the desktop in pixels.
- You will need to get the display number from the output of the vncserver command:
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1 Starting applications specified in /home/fs01/shl1/.vnc/xstartup Log file is /home/fs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log
- vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.
Set up your ssh tunnel (Do these steps on your local computer)
- Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12.
- From Linux, in order to start ssh port forwarding or tunneling to that port, type into a terminal:
ssh -L 10000:localhost:5901 email@example.com
- From macOS, open a Terminal and enter the Linux command above.
- From Windows, ssh clients such as PuTTY can do port forwarding (tunneling); see VNC Tunnel Windows.
- Leave this ssh session running on your local client computer. (It can run in the background.)
Connect your VNC client
- Launch your VNC client program. Connect it to localhost:10000. When prompted, type in your VNC server password.
- A nice GNOME desktop should appear!
- If a pop-up asks you to authenticate, just cancel it. See this link for how to prevent the annoying "Authenticate" pop-up from appearing in your future vncserver sessions.
To disconnect your client
- Close the VNC client program.
- Disconnect the ssh forwarding session (i.e., kill it).
To reconnect your client
- Restart port forwarding with ssh, using the same remote port number as before.
- Again connect the VNC client to localhost:10000.
When you are all done
- It may not be possible to to log out from the Linux desktop. But even if this appears to work, it will leave the VNC server running.
- When you are finished with your session, shut down all your applications in the desktop, disconnect (close) it, and type this command into a separate ssh session to shut down the VNC server completely:
vncserver -kill :<display number>
An alternative to password-based authentication is public key authentication (PKA). SSH has a well-established mechanism for making use of a public/private key pair.
Note: setting this up for yourself is completely optional! On CAC private clusters where an SSH key pair is required for intra-cluster communications, one will be created for you when first log in to the head node, and you never need to worry about it.
How it works
- When you connect via SSH, instead of entering a password, you provide the name of an identity file on your computer. This is your private key, part of a public/private key pair.
- The computer you are connecting to must already have the matching public key stored in a special location. On Linux systems, it should be found among the list of public keys in the file ~/.ssh/authorized_keys.
- Upon receiving your initial SSH request, the remote computer encrypts a message using one of the public keys in ~/.ssh/authorized_keys. It sends the encrypted message to your computer.
- Your local SSH client attempts to decrypt this message using the private key file you specified. The decrypted message is then sent back to the remote computer.
- The remote computer checks whether your client succeeded in decrypting the message. If so, you have proven your identity. If not, it tries the next public key until all are exhausted.
Clearly, if you want to make use of this mechanism, you will need to set up a public/private key pair!
Create an SSH key pair
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from linuxlogin.cac.cornell.edu, the general Linux login node, or from the login node of a private cluster. (If this is your first login to a CAC login node, it will ask you to change your password; this will become your password for connecting to CAC login nodes in the future.) The steps to create your key pair are as follows:
mkdir .ssh chmod 700 .ssh cd .ssh ssh-keygen (and take all default options) cat id_rsa.pub >> authorized_keys
As a final step, you need to copy the private key, id_rsa, to the computer that you will be logging in from.
Linux and macOS users:
You'll want to put the private key in the ~/.ssh directory. You may wish to call this key id_rsa for convenience, BUT be careful not to overwrite an existing key with that name. Let's say you decide instead to call your new private key cac_id_rsa on your local computer, just to be safe. You MUST change permissions on this key to keep it private:
chmod 600 ~/.ssh/cac_id_rsa
Now you're ready to use passwordless SSH to connect to linuxlogin. From a terminal in macOS or Linux, you do the following:
ssh -i ~/.ssh/cac_id_rsa <your_CAC_username>@linuxlogin.cac.cornell.edu
The way to proceed in Windows depends on the SSH client you are using. Here, we cover PuTTY as an example. The first step is to use PuTTYgen to convert the SSH private key for use with PuTTY and plink.
- Run "C:\Program Files (x86)\Putty\puttygen.exe".
- Select Import Key from the Conversions menu and enter C:\Users\<your_local_username>\.ssh in the address bar, assuming this is where you placed your private key in your home directory. Then, select the id_rsa file and click on the Open button.
- Click on the "Save Private Key" button.
- Click on "Yes" when asked to save the private key without a passphrase.
- Save the private key as private.ppk in the .ssh directory inside your home directory.
- Close (choose File, then Exit)
- To confirm you have converted the ssh private key successfully, do:
"C:\Program Files (x86)\Putty\plink.exe" -i %USERPROFILE%\.ssh\private.ppk <your_CAC_username>@linuxlogin.cac.cornell.edu
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."
- If everything was done correctly, you should now be logged into linuxlogin without being prompted for a password. Type exit to log out.
In PuTTY, you will want to update your Saved Session for linuxlogin to use the new key. Load the linuxlogin session. Navigate to "Connection > SSH > Auth", browse to %USERPROFILE%\.ssh and open the id_rsa file. Go back to Session and click Save. Now you won't need to enter a password for linuxlogin ever again!
Connect to Windows
Using Remote Desktop Connection to connect to winlogin
This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to CTC_ITH, not the local name of the machine to which you are connecting.
Remote Desktop sessions do not expire unless you log out, but they will end when machines are rebooted during down times.
- If you use a Windows machine:
Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, find Windows Accessories in your list of apps, then click Remote Desktop Connection.
- If you use macOS:
Search from Microsoft Remote Desktop at the App Store. it works just like the Remote Desktop Connection in Windows 7 and later. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.
- If you use Unix or Linux (or Mac):
You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from rdesktop.
- If you use a Windows machine:
Working with CAC file storage
A dedicated file server named storage03.cac.cornell.edu provides access to much of CAC's file storage, including the home directories for many of the private clusters. To work with your files, you can access this server in a variety of ways from any operating system. The first two of the following methods are covered in detail in this section:
- Home directory access - Mount/map your portion of the storage03 filesystem as network share or network drive. Once the filesystem is mounted, your files on storage03 appear in a folder that you can access just like other folders on your computer. On Linux, use the mount command; on Mac, use "Go > Connect to server"; on Windows (including winlogin), enter the UNC address into the address bar of a File Explorer window, or do "Map a network drive".
- File transfer - Use a file transfer utility like scp or sftp to copy your files to or from storage03. Connect to linuxlogin to do this, because storage03 is not directly accessible. On linuxlogin, your main CAC folder on storage03 is your home folder when connect via ssh, scp, or sftp.
- - Use Globus to transfer files to or from storage03. The source or destination of the files must also be a Globus endpoint (and note, you can set up any computer to be a personal endpoint). Endpoints at CAC are described on the File Transfer using Globus page.
Note: by default, your home directory on linuxlogin and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.
Home directory access
Users of CAC's storage services have a "storage03" directory which can be accessed from both Linux and Windows systems:
- On linuxlogin, it is generally your home directory: /home/fs01/<username>
- On winlogin, you generally access it at this address: \\storage03.cac.cornell.edu\<username>
Note, private clusters often have their own file servers, so users of private clusters may find that the linuxlogin path is not the same as their home directory on their private clusters. Also, Red Cloud users do not automatically have storage privileges on storage03, unless such storage is included in their CAC project.
In Linux, it is generally safe to refer to your home directory as either ~, ~<username>, or $HOME, so you never need to specify the exact mount point. On CAC's Windows systems, you can map your home directory to a letter drive (such as H:) using the "Map Network Drive" feature; however, it is often preferable to use the full UNC path to the network share, as given above.
You can mount your CAC home directory on your local machine, as long as your machine is connected to either the Cornell campus network or CU VPN.
You mount your storage03 directory via SMB/CIFS like this:
sudo mount -t cifs //storage03.cac.cornell.edu/<username> /mnt/pt -o user=<username>,domain=CTC_ITH,uid=<localid>,vers=2.1
where <username> is your CAC user name, <localid> is your local user name, and /mnt/pt is the name of a directory you have created ahead of time to be the mount point on your local filesystem. Enter the password for CAC account when prompted. See man mount.cifs for available options for the mount command.
If you see errors, such as "missing codepage or helper program," then you have not installed the mount and umount packages for CIFS on your local machine. If problems persist, send your initial command and the results of dmesg | tail to CAC Help.
- In the Finder, either select Connect to Server... from the Go menu or use the shortcut cmd-K.
- Enter smb://storage03.cac.cornell.edu/<username> in the Server Address field as shown below. You may need to use smb://<username>@storage03.cac.cornell.edu/<username>.
- Enter your CAC user name and password to log in. You may need to use <username>@tc.cornell.edu in place of your username.
- In the Finder, either select Connect to Server... from the Go menu or use the shortcut cmd-K.
- In a File Explorer window, right-click on "This PC"
- Choose "Map Network Drive..." from the menu that appears
- Select "H:" from the drop-down menu (if you are already using this drive letter, select another letter)
- Folder: \\storage03.cac.cornell.edu\<username>
- -Check "Connect using different credentials". This will allow you to enter the domain associated with CAC and your username at CAC, rather than those associated with your own machine.
- -User name: CTC_ITH\<username>
- -Password: your CAC password
- Troubleshooting: If you have already mapped the drive and subsequently have problems, disconnect the drive and remap it.
A single, central file server, storage03.cac.cornell.edu, provides access to much of CAC's file storage for individual users. It serves the bulk of the home directories on linuxlogin as well as many private clusters. Here we look at various clients that can be used to transfer (i.e., copy) files to and from this server, mainly via linuxlogin.
Linux and macOS users
Secure copy is a standard tool to copy files to and from remote hosts.
localhost$ scp localfile.dat firstname.lastname@example.org:remoteinput.dat localhost$ scp email@example.com:results.dat localresults.dat
FTP is disabled for security reasons, but sftp's interface is nearly identical.
This technique only works from Cornell campus locations or via a Cornell VPN connection. Type
smbclient //storage03.cac.cornell.edu/<user name> -U ctc_ith\\<user name>
(Note, the shell interprets \\ as a single backslash.) Enter the password for your CAC account when prompted. You will see the smb:\> prompt. Now you can start transferring files between your local machine and your CAC home directory, using commands similar to the sftp client. Type help for more instructions.
-bash-4.1$ smbclient //storage03.cac.cornell.edu/<user name> -U ctc_ith\\<user name> Enter ctc_ith\<user name>'s password: Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.6.23-24.el6_7] smb: \> help
The individual who created PuTTY provides a secure copy client called pscp. From the command prompt, type:
cmd> pscp localfile.dat firstname.lastname@example.org:remoteinput.dat <enter your username's password when prompted> cmd> pscp email@example.com:results.dat localresults.dat
FTP is disabled for security reasons, but psftp's interface is nearly identical. From the command prompt, type:
cmd> psftp firstname.lastname@example.org <enter your username's password when prompted> psftp> put localresults.dat results.dat psftp> quit
Linux usage tips
If you have never used Linux before, we recommend exploring the Linux Tutorial.
- /bin/sh is the default login shell.
- Edit $HOME/.profile to change interactive variables.
- The $HOME/.bashrc file will not be run for non-interactive shells.
- Edit $HOME/.profile to change interactive variables.
- The $HOME/.bashrc file will be run for non-interactive shells.
- /bin/csh and /bin/tcsh
- Edit $HOME/.login to change interactive variables.
- The $HOME/.cshrc file will be run for non-interactive shells.
- /bin/sh is the default login shell.
The change shell command, chsh, will not permanently change your shell. You must send a request instead. Contact Support
The default login shell on linuxlogin is sh. Be aware that in CentOS, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of bash. Accordingly, you will find that "man sh" brings up the man page (the help document) for bash. In a way, then, you can think of your login shell as being bash, too.
There are slight differences between sh and bash, however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both sh and bash; but any interactive sh shells you start thereafter will not run ~/.bashrc as you might expect from bash. The way to get sh to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).
Let's say you simply prefer to have bash as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be bash. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be bash (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is sh.
The relationship between the csh and tcsh shells is similar to the one between sh and bash. For instance, your csh shells are automatically endowed with the tcsh-style ability to retrieve history through the up- and down-arrow keys. The best way to make tcsh into your everyday working shell is to run it on top of sh after you log in (again, you can do this as part of your .profile).
Use /tmp to compile large codes and software packages. This will provide improved performance and greater system stability.
If you want to know what processor features a cluster supports, submit a batch job that does "cat /proc/cpuinfo" in order to find out the CPU type. The v4 cluster is composed mostly of Intel E5420 CPUs (in Nov. 2011). Then you go to Wikipedia's Intel Xeon page or Intel's ARK to find that these are Harpertown cores that support SSE, SSE2, SSE3, SSSE3, SSE4.1 and VMX.
C/C++ and Fortran Codes
- GNU compilers gcc, g++, g77, gfortran are in /usr/bin, which is in the default path.
- For compiling OpenMP directives, add the option -fopenmp.
- Intel 12.1 compilers icc, ifort are in the default path on the login nodes.
- For compiling OpenMP directives, add the option -openmp.
- The following Intel libraries and tools are available to you automatically through the default setup on the login nodes:
- MKL, the Math Kernel Library 10.3.6 (additional help below)
- idb, the Intel debugger for Linux
- TBB, the Threading Building Blocks
- IPP, the Integrated Performance Primitives
- If any of the above libraries are linked dynamically, the correct runtimes will be loaded automatically on the compute nodes by default; no additional setup is required.
- Note - if you find that your code segfaults after compiling with Intel 12.1, try disabling optimization or using the older 11.1 version of the compilers instead.
Reason: there is a known bug in the vectorizer of the 12.1 compiler which is due to be fixed in a future release.
- Intel 11.1 compilers icc, ifort are available, also, but these older compilers require special setup files.
- Before compiling in bash: source /opt/intel/intel-11.sh
- Before compiling in tcsh: source /opt/intel/intel-11.csh
- At runtime, in a batch sh-script: source /opt/intel/intel-11.sh
- At runtime, in a batch csh-script: source /opt/intel/intel-11.csh
- The above steps also enable the use of the older Intel performance libraries, e.g., MKL 10.2 (additional information below).
- Help for Intel compilers (if you are using 11.1, be sure to source the setup file first):
- Fortran: man ifort, info ifort, ifort -help
- C/C++: man icc, info icc, icc -help
- Standard compiler options - the clusters have Intel Core2 processors, so standard compiler options are:
- For Intel: -O3 -ipo -mtune=pentium4 -march=pentium4
- Other options of possible interest (consult man pages):
- For Intel: -fno-alias -align -scalar_rep -prefetch
- GNU compilers gcc, g++, g77, gfortran are in /usr/bin, which is in the default path.
Generating Debugging Info
- Intel compilers
- icc -Wall
- ifort -g -debug -warn -C (-CB for bounds checking only)
- Intel compilers
For compiling MPI codes, we recommend using mpicc and mpif90. If you specifically need a C++ compiler, try mpicxx. Because of these handy wrapper scripts, you may not need to do very much to convert existing makefiles to work with CAC's preferred software stack. Currently the default paths are set up so that the mpicc, mpif90 and mpicxx utilities invoke the Intel 12.1 compilers to compile your codes and link them properly to the Intel MPI 3.1 libraries. However, if you run the Intel 11.1 compiler setup file first, then these utilities will automatically use the older 11.1 compiler version. Documentation for the Intel MPI 3.1 Library, including mpdboot and mpiexec, is in PDF on the Intel Support Site.
To view a sample batch script that will run an MPI job for you, see the section on Running a parallel MPI job.
The ROCKS operating system comes with several alternate MPI implementations (e.g., mpich2, OpenMPI). You have to play with environment variables and paths to get them to work.
Intel's Math Kernel Library (MKL) is a good source of optimized routines for linear algebra, Fast Fourier Transforms, vector math, and other mathematical operations. In particular, it provides a way to incorporate Intel's optimized BLAS and LAPACK routines into your code.
OpenMP multithreading is built into certain MKL libs. When these libs are linked, calls to MKL will detect the same settings that would affect any other OpenMP-enabled code. This means MKL will attempt to use all the cores present on a v4 node during the execution of parallelized sections. Therefore, when you link your code with a "_thread" version of the MKL library, your should realize that all your calls to MKL will generally fork the same number of threads as the number of cores present. This may cause undesired interference with other parallelization strategies you are using, e.g., MPI. If this is not the behavior you want, you can do one of two things:
- Link with "mkl_sequential" (or -mkl=sequential in 12.1) instead of, e.g., "mkl_intel_thread" (or -mkl=parallel in 12.1).
- At run time, set the OMP_NUM_THREADS environment variable to 1. (Use "export" or "setenv".) A value of 8 recovers the default behavior on v4.
Linking Intel MKL 10.3.6 with the Intel 12.1 Compilers
MKL 10.3.6 is the version installed with the 12.1 compilers. The easiest way to link MKL is to compile as follows, where the last two lines pertain to MPI codes:
- icc mycode.c -o mycode -mkl
- ifort mympicode.c -o mycode -mkl
- mpicc mympicode.c -o mympicode -mkl
- mpif90 mympicode.f90 -o mympicode -mkl
Note, -mkl is the same as -mkl=parallel, which enables OpenMPI mulithreading. If you don't want this, use -mkl=sequential.
With just plain -mkl (or -mkl=...), the resulting executable will be dynamically linked. This means that at run time, your program has to know where to find the MKL shared libraries. Since MKL 10.3.6 is the default, the appropriate paths have been predefined for you on the compute nodes, and your batch jobs should have no trouble.
Should you want to link MKL in some different way--e.g., statically--the compile line will start looking messier. Linking to MKL has become rather complicated due to Intel's decision to maximize MKL's flexibility and multi-platform compatibility by splitting out four separate layers of libraries: interface, threading, computational, and runtime (meaning OpenMP, if the _thread lib is requested). To make sure you have all these layers, we recommend appending one of the following snippets to your ifort, icc, mpif90, or mpicc command (after first setting MKLPATH = $MKLROOT/lib/intel64):
- static, multithreaded:
$MKLPATH/libmkl_solver_lp64.a -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -openmp -lpthread
- static, sequential:
$MKLPATH/libmkl_solver_lp64_sequential.a -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a $MKLPATH/libmkl_sequential.a $MKLPATH/libmkl_core.a -Wl,--end-group -lpthread
- static, multithreaded:
These options will generate a (mostly) statically linked executable. Note, each .a-lib must be identified by its full path in order to prevent the .so-lib (its dynamic equivalent) from being found instead. If you do not need access to the MKL solver routines, simply remove that item from the head of the list. As noted previously, if your main program is itself threaded with OpenMP, or if it is parallelized with MPI, you may want to select libmkl_sequential.a in order to reduce contention and get better performance.
To generate a dynamically linked rather than statically linked executable, the above options become:
- dynamic, multithreaded:
$MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -openmp -lpthread
- dynamic, sequential:
$MKLPATH/libmkl_solver_lp64_sequential.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread
- dynamic, multithreaded:
These sets of options are pretty much equivalent to -mkl=parallel and -mkl=sequential, respectively.
If your batch script needs strict control over LD_LIBRARY_PATH, then one other compiler/linker option may be helpful for a dynamically-linked code:
The above variables should be set to MKLPATH = $MKLROOT/lib/intel64 and IOMPPATH = $MKLROOT/../compiler/lib/intel64. This option "hardwires" the correct paths into the executable; these paths are valid on both the v4 compute nodes and the v4 login nodes. If you don't wish to restrict your executable in this fashion, the alternative is to add these paths manually to LD_LIBRARY_PATH.
Intel has put together a helpful tool for generating the correct linker options to match your specific needs, the Link Line Advisor. This is definitely the place to go if you want to, e.g., use extra-long integers or compile with gcc or gfortran. It's well worth a visit.
Much more information on linking MKL 10.3.6 can be found in the "Linking Your Application" section of the User Guide, which you can access from the login node ("firefox /opt/intel/composer_xe_2011_sp1.6.233/Documentation/en_US/mkl/mkl_userguide/index.htm").
Linking Intel MKL 10.2 with the Intel 11.1 Compilers
MKL 10.2 is the version installed with the 11.1 compilers. Since 11.1 is not the current default version of the compilers, you must first source the setup file:
- In bash (or sh): source /opt/intel/intel-11.sh
- In tcsh (or csh): source /opt/intel/intel-11.csh
If your program links MKL dynamically, it has to know where to find the correct MKL shared libraries at run time. Bear in mind that MKL 10.2 is not the default on the compute nodes, either. The easiest way to ensure correct behavior at run time is to put the same line into your batch script:
- In a batch sh-script: source /opt/intel/intel-11.sh
- In a batch csh-script: source /opt/intel/intel-11.csh
Otherwise the instructions for linking MKL 10.2 are identical to the instructions for MKL 10.3.6 and Intel 12.1 above. There is one exception: you need to set MKLPATH = $MKLROOT/lib/em64t and IOMPPATH = $MKLROOT/../lib/intel64.
The Link Line Advisor can be applied to older versions of the Intel compilers and MKL. It's well worth a visit.
Much more information on linking MKL 10.2 can be found in the Sec. 5 of the User Guide, which you can access from the login node ("firefox /opt/intel/Compiler/11.1/072/Documentation/en_US/mkl/userguide.pdf").
- ldd - see the man page.
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.
How do I display an image file (such as jpeg or gif)?
- display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.
- firefox mypic.jpg - any decent Web browser can handle it.
Note, the image will show up only if you have X11 forwarding enabled.
How do I use revision control?
- Git, Subversion, and CVS are examples of revision control (or version control or source control) software. These tools help you collaborate with others by allowing you to save and track successive versions of your source code as you modify it. Git is often used in conjunction with GitHub. Git is installed on linuxlogin: see the git man pages for for details. To check the installed version, type git --version.
When you publish a paper, make presentations, or are interviewed by the Cornell Chronicle, national news media, etc., please acknowledge the Center by including:
"This research was conducted with support from the Cornell University Center for Advanced Computing."
Alternatively, the full acknowledgement is:
"This research was conducted with support from the Cornell University Center for Advanced Computing, which receives funding from Cornell University, the National Science Foundation, and members of its Partner Program."