Perseus used Hermes' flying sandals to gain victories - They can attack big problems!
Perseus, Slayer of Medusa, by jovial_cynic, 2008.
Image freely copyable under a Creative Commons license. Some rights reserved.
 

Tips and Tricks for Running PERSEUS on
Hermes, the 36-Core Supermicro Server

Logging In
      -  Connecting to Linux Systems from Windows
      -  Shell Navigation and the Screen Command
      -  Shell Customization
      -  Editors
Storing and Accessing Files
      -  Backups
Compiling and Linking Codes
      -  Using the Intel Fortran Compiler
      -  Linking an MPI Library
      -  Linking Other Libraries
Visualization and Analysis
      -  Opening a Remote Desktop with X2Go
      -  Running ParaView
      -  Running Mathematica
      -  Setting Up a Python 3 Environment
Monitoring the Load


Perseus used Hermes' flying sandals to gain victories - They can attack big problems!

Hermes is a high-performance workstation at the Laboratory of Plasma Studies at Cornell University. It is primarily intended for running the simulation code PERSEUS. For technical details on this resource, see the system configuration page for the 36-core Hermes server. (You can view a similar page for the now-defunct 64-core Perseus server. It was a nice resource in its time.) This page is meant to be a basic user guide to the machine.

Hermes runs CentOS 7.6 for its operating system.

Logging In

Once you have been assigned a username, the easiest way to access Hermes is to use ssh. Start ssh from the command line on Linux systems, as well as on Mac (use the Terminal application in macOS).

For security reasons, Hermes is on an ECE subnet that restricts incoming connections to those that appear to originate from a Cornell subdomain. This means you must use VPN if you want your ssh client to connect from a network location outside of Cornell. See CU VPN to learn how to install and use Cornell's VPN software.

Connecting to Linux Systems from Windows

PuTTY is a free, secure ssh client for connecting to Linux or Unix systems. To enable graphical display from your remote systems, PuTTY must be coupled with an X window server. VcXsrv and Xming (both free and open-source) are good choices for this purpose. Here is the sequence for starting an ssh session with X11 forwarding enabled:

After logging out from PuTTY, you can choose to quit Xming by right-clicking its icon in the System Tray and selecting "Exit".

Shell Navigation and the Screen Command

On Hermes, the default Linux shell (=OS command processor) is bash. Becoming competent in bash is a whole learning experience in itself. Here are some links to get you started:

The last two links give you advice on (1) how to get a long-running code like PERSEUS out of your way while it is running, and (2) how to log out without killing your PERSEUS run. All by itself, screen will be adequate for both (1) and (2). Just do this:

screen
mpiexec -n 16 ./perseus
<Ctrl>-a d

The initial screen command places you into a new, detachable session. The final key sequence is what detaches the session and drops you back to your original shell. You can safely log out at this point. After you log back in, you can restore the previous session with "screen -R". To terminate the screen, first restore it, then kill all its processes and exit the screen. If you want to be totally safe when invoking screen (perhaps to the point of overkill), add nohup:

screen nohup mpiexec -n 16 ./perseus > outputfile &

Screen is per-session; it allows you to disconnect from a terminal session without killing it. Nohup (no-hangup) is per-process; it keeps the process running even if the parent shell goes away.

Shell Customization

Why would you want to customize your shell environment in CentOS 7? The main reasons would be to put Open MPI tools such as mpif90 and mpiexec (mpirun) into your PATH, and to set up your environment to favor the Intel compilers. However, Hermes is already set up to do these things for you automatically every time you log in or start up a new shell. Therefore, the following information is presented just for your general enlightenment.

In bash, login shells are initialized by the file ~/.bash_profile, while non-login interactive shells are initialized by the file ~/.bashrc. (Note, ~ and $HOME both refer to your home directory.) To ensure consistency between both types of interactive shells, the default ~/.bash_profile (at login) merely calls the ~/.bashrc before adding a few locations to the PATH. So why not just put tweaks to the PATH or other environment variables in ~/.bashrc? The problem is, a parent shell will export its environment variables to its child shells. So if one interactive shell starts another, which starts another... and ~/.bashrc is executed repeatedly... then PATH (e.g.) grows and grows as a consequence. That is why your default .bash_profile and .bashrc look like the following in CentOS 7, prior to any customization:

.bash_profile .bashrc
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
-- PUT YOUR ADDITIONS HERE --

You still have to be careful with customizations because in Linux, shells you open on the desktop or in a screen session can have unexpected behavior. As a result, you may want to put if-then constructs into your .bashrc; testing is definitely a good idea. For example, on the Mac, OS X behaves a bit differently from Linux; interactive shells started by the Terminal app are all login shells.

Editors

The usual Linux command-line editors are available: vi (really vim), emacs, and nano. If you prefer working with a basic GUI editor, gedit is available as well.

If you prefer emacs—meaning you have already learned emacs well enough not to be intimidated by it—then you don't necessarily have to start it in its own window via the "emacs" command. Instead, you might want to start it with "emacs -nw" so it occupies your current terminal window. Why? Generally it starts up faster that way in an "ssh -X" session. However, if you prefer to run emacs in a separate X window, that should be fine too (though the emacs window used to be unstable due to a bug in GTK.)

Storing and Accessing Files

The main locations to store your files are (1) in your $HOME, i.e., in the directory /home/$USER, and (2) in your personal data directory, /dataN/$USER, on one of the two 4TB hard drives. Handy commands to see how much disk space is used/available at your current location are "du -s ." and "df ."

Backups

Data on Hermes are not backed up. To reduce the risk of data loss, keep copies of your important files on more than one of the above devices (or on external storage media). Here is a command that is useful for doing such a backup:

rsync -avz /dataP/$USER/ /dataQ/$USER

The trailing slash on the name of the first directory is significant—it means "get the contents of this directory". The interpretation of the flags, -avz, is as follows: archive the files with original properties, do it verbosely, and use compression while copying.

Compiling and Linking Codes

For everyone's convenience, certain commands are AUTOMATICALLY run in each new shell that is started In CentOS 7 on Hermes. These commands are the ones that must be run in your shell before you can compile and link your MPI codes using Open MPI and ifort, the Intel Fortran compiler. For the record, here are the commands that run automatically on Hermes:

. /opt/intel/bin/ifortvars.sh intel64
module load mpi
export OMPI_FC=ifort

On less-friendly systems, you might have to place commands like these in your .bashrc as noted above (inside the suggested if-clause).

Using the Intel Fortran Compiler

Here is a recommended Makefile that you can use for compiling PERSEUS in any shell. (It assumes that the above commands were auto-run for you when you opened the shell.)

Linking an MPI Library

Just make sure you use the "mpifort" (or mpif90) command, and the MPI linking should be done correctly. Remember, every MPI implementation comes with its own "mpif.h" file; therefore, that file is not part of your code, and you should never copy it from system to system!

Linking Other Libraries

The Unix convention is that the option "-lmylib" refers to libmylib.so or libmylib.a. If both libraries are present in the same directory, the .so file (shared-object or dynamic library) will be chosen over the .a file (static library). If the library cannot be found at link time, you can specify the path to it via the -L option to the compiler. (The compiler just passes this option to the linker.)

Visualization and Analysis

Windowing in Linux is built around X11, a system for displaying individual windows locally or remotely. But the performance of X11 can be draggy if your remote window is running a rich, graphics-heavy application like ParaView or Mathematica. For doing interactive visualization, it is preferable to set up an entire remote desktop. Several solutions exist for this, notably VNC and OpenNX. The solution chosen for Hermes is X2Go.

Opening a Remote Desktop with X2Go

  1. First, download and install the appropriate X2Go client on your laptop or workstation.
  2. Once that's done, set up a VPN as you would for an ssh connection.
  3. Start the X2Go client. From the Session menu, choose "New Session...", then enter the following:
Session tab Connection tab Input/Output tab Media tab Shared folders tab
Session name: hermes
Host: hermes.lps.cornell.edu
Login: <your username>
Session type - MATE (in drop-down)
Connection speed -
WAN (move slider)
Display - Custom -
Width: 1200, Height: 750
Enable sound support -
  (clear check box)
Client side printing support -
  (clear check box)
(optional)
Path: <choose local path>
  (click "Add")
  (put checks in all 3 boxes)
*Obviously these settings are just the minimal recommendations; you are welcome to customize them further to suit your own preferences.
  1. OK the settings. A new "hermes" tile appears in the client window. Click it. Enter your password, then "Ok".
  2. After a short wait, a new window should appear with a remote desktop on Hermes! Resize it manually if you like.
  3. If you designated a shared folder, a file icon may appear on your desktop with a name ending in (sshfs-disk-mate); ignore it.
    Visit the X2Go website for complete information on how to access your shared folders.
  4. If you click the Pause button, or close the local window, your remote session continues to run! You have merely disconnected it.
  5. To log back in, enter your password and hit "Ok" again. Your desktop should appear exactly as you left it (even from a different machine!).
  6. To terminate the session completely, don't log out. Go back to the initial X2Go window and click the 0/1 button.

If you need to terminate an X2Go session on Hermes that has become unreachable, the easiest way to do so is to log in with ssh and enter the following, which will kill every single process that you are running (including the shell you just started!):

pkill -9 -u $USER

The X2Go client for Mac does not always have the correct key mappings initially. If you see the wrong characters showing up while you are typing, disconnect your session and reconnect to it. Often this is enough to correct the problem.

Running ParaView

From the MATE desktop, the ParaView 4.3.1 application can be started from the "Applications > Graphics >" menu. An even handier shortcut is available on the top menu bar. If the ParaView icon doesn't appear there, try entering the following command. (The second command is needed only if the ParaView icon doesn't appear after you enter the first.)

mate-panel --reset
mate-panel --reset --layout rhel-pv

ParaView 5.8.1 or 4.3.1 can also be started from the command line; note, your PATH can be adjusted to include the ParaView directory if you like.

/opt/ParaView-5.8.1-MPI-Linux-Python2.7-64bit/bin/paraview-mesa paraview &
/opt/ParaView-4.3.1-Linux-64bit/bin/paraview &

Note that with ParaView 5.8.1, paraview must be started through the paraview-mesa wrapper in order to get OpenGL to work properly.

Running Mathematica

If you are logged in with ssh -X and have an X server running locally, just type "mathematica" in your shell to start the GUI. If you want to run it an X2Go remote desktop, the following startup command is necessary to avoid a Qt5 bug (or quirk):

QT_XCB_GL_INTEGRATION=none mathematica &

You will need an activation key if want to use Mathematica on Hermes. To get one, send an email to the author of this page.

Setting Up a Python 3 Environment

Python 3.6 is installed via Red Hat Software Collections (RHSCL). It allows Python 3 to coexist peacefully with the default python (2.7) in CentOS. RHSCL enables you to configure your shell so that the "python" command, as well as several relevant environment variables, all point to Python 3. NumPy, SciPy, and IDLE are included in the RHSCL installation on Hermes.

To start a subshell that is appropriately configured for Python 3, enter this command:

scl enable rh-python36 bash

To change all your shells, including your login shell, so that they will use Python 3 by default, add this line to your ~/.bashrc file:

source scl_source enable rh-python36

An extensive Red Hat blog post has plenty of good advice on how to use Python 3 (scl enable), install Python packages, set up virtual environments, etc.

Monitoring the Load

Here are some helpful commands to see what's happening on the machine. You may want to run them if you're about to start a new MPI job, or if your current MPI job seems to be running more slowly than you expected.

w
uptime
top
ps aux | grep -v root

A fully loaded 36-core machine will have a load average of 36 in "w" and "uptime". Anything over that is likely to be using the cores inefficiently. The "top" command lets you monitor the machine's activity continuously and in much more detail. It lets you see which processes are the heavy CPU users (type "q" to quit). The one-shot "ps" command allows you to filter out things you don't want to see in "top".



Last updated on 8/20/20 by Steve Lantz (slantz ~at~ cac.cornell.edu)