Hermes is a high-performance workstation at the
Laboratory of Plasma Studies at Cornell University. It is
primarily intended for running the simulation code
PERSEUS.
For technical details on this resource, see the system configuration page for the
36-core Hermes server.
(You can view a similar page for the now-defunct
64-core Perseus server, which was a nice resource in its time.
There is also a page for the 56-core Ubuntu server
reserved for Prof. Seyler's use.) This page is meant to be a basic user guide to Hermes.
Hermes runs CentOS 7.9 for its operating system.
Logging In
Once you have been assigned a username, the easiest way to access Hermes is to use ssh. Start
ssh from the command line on Linux systems, as well as on Mac (use the Terminal application in macOS).
For security reasons, Hermes is on an ECE subnet that restricts incoming connections to those that
appear to originate from a Cornell subdomain. This means you must use VPN if you want your ssh client to
connect from a network location outside of Cornell. See CU VPN
to learn how to install and use Cornell's VPN software.
Connecting to Linux Systems from Windows
PuTTY is a free, secure ssh client for connecting to Linux or Unix systems. To enable graphical
display from your remote systems, PuTTY must be coupled with an X window server. VcXsrv and
Xming (both free and open-source) are good choices for this purpose. Here is the sequence for
starting an ssh session with X11 forwarding enabled:
- Start Xming by clicking its Quick Launch icon or by selecting it from the Start menu
- Start PuTTY via the shortcut on the desktop or via the Start menu
- Enter the address of your destination (or just click a name in your saved list and "Load" it)
- Include your username in the address line if you like, e.g., my_username@hermes.lps.cornell.edu
- Verify that SSH | X11 | Enable X11 forwarding is selected
- Select SSH | Enable compression, too, for better X11 performance
- Return to Session, give it a name if needed, and click "Save" so you can re-use it later
- Click "Open" to log in
After logging out from PuTTY, you can choose to quit Xming by right-clicking its icon in the System
Tray and selecting "Exit".
Shell Navigation and the Screen Command
On Hermes, the default Linux shell (=OS command processor) is bash. Becoming competent in
bash is a whole learning experience in itself. Here are some links to get you started:
The last two links give you advice on (1) how to get a long-running code like PERSEUS out of your way
while it is running, and (2) how to log out without killing your PERSEUS run. All by itself,
screen
will be adequate for both (1) and (2). Just do this:
screen
mpiexec -n 16 ./perseus
<Ctrl>-a d
The initial screen command places you into a new, detachable session. The final key sequence is what
detaches the session and drops you back to your original shell. You can safely log out at this point.
After you log back in, you can restore the previous session with "screen -R". To terminate the screen,
first restore it, then kill all its processes and exit the screen. If you want to be totally safe when
invoking screen (perhaps to the point of overkill), add nohup:
screen nohup mpiexec -n 16 ./perseus > outputfile &
Screen is per-session; it allows you to disconnect from a terminal session without killing it. Nohup
(no-hangup) is per-process; it keeps the process running even if the parent shell goes away.
Shell Customization
Why would you want to customize your shell environment in CentOS 7? The main reasons would be to put
Open MPI tools such as mpif90 and mpiexec (mpirun) into your PATH, and to set up your environment to
favor the Intel compilers. However, Hermes is already set up to do these things for you automatically
every time you log in or start up a new shell. Therefore, the following information is presented just
for your general enlightenment.
In bash, login shells are initialized by the file ~/.bash_profile, while non-login
interactive shells are initialized by the file ~/.bashrc. (Note, ~ and $HOME both refer to your home
directory.) To ensure consistency between both types of interactive shells, the default ~/.bash_profile
(at login) merely calls the ~/.bashrc before adding a few locations to the PATH. So why not just put
tweaks to the PATH or other environment variables in ~/.bashrc? The problem is, a parent shell will
export its environment variables to its child shells. So if one interactive shell starts another, which
starts another... and ~/.bashrc is executed repeatedly... then PATH (e.g.) grows and grows as a
consequence. That is why your default .bash_profile and .bashrc look like the following in CentOS 7,
prior to any customization:
| .bash_profile |
.bashrc |
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH |
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
-- PUT YOUR ADDITIONS HERE -- |
You still have to be careful with customizations because in Linux, shells you open on the
desktop or in a screen session can have unexpected behavior. As a result, you may want
to put if-then constructs into your .bashrc; testing is definitely a good idea. For example, on the
Mac, OS X behaves a bit differently from Linux; interactive shells started by the Terminal app are
all login shells.
Editors
The usual Linux command-line editors are available: vi (really vim), emacs, and nano. If you prefer
working with a basic GUI editor, gedit is available as well.
If you prefer emacs—meaning you have already learned emacs well enough not to be intimidated
by it—then you don't necessarily have to start it in its own window via the "emacs" command.
Instead, you might want to start it with "emacs -nw" so it occupies your current terminal window. Why?
Generally it starts up faster that way in an "ssh -X" session. However, if you prefer to run emacs in
a separate X window, that should be fine too (though the emacs window used to be unstable due to a bug
in GTK.)
Storing and Accessing Files
The main locations to store your files are (1) in your $HOME, i.e., in the directory /home/$USER,
and (2) in your personal data directory, /dataN/$USER, on one of the two 4TB hard drives. Handy
commands to see how much disk space is used/available at your current location are "du -s ." and
"df ."
Backups
Data on Hermes are not backed up. To reduce the risk of data loss, keep copies of your
important files on more than one of the above devices (or on external storage media). Here is a
command that is useful for doing such a backup:
rsync -avz /dataP/$USER/ /dataQ/$USER
The trailing slash on the name of the first directory is significant—it means "get the
contents of this directory". The interpretation of the flags, -avz, is as follows: archive the
files with original properties, do it verbosely, and use compression while copying.
Compiling and Linking Codes
For everyone's convenience, certain commands are AUTOMATICALLY run in each new shell that is
started In CentOS 7 on Hermes. These commands are the ones that must be run in your shell before
you can compile and link your MPI codes using Open MPI and ifort, the Intel Fortran compiler. For
the record, here are the commands that run automatically on Hermes:
. /opt/intel/bin/ifortvars.sh intel64
module load mpi
export OMPI_FC=ifort
On less-friendly systems, you might have to place commands like these in your .bashrc as noted
above (inside the suggested if-clause).
Using the Intel Fortran Compiler
Here is a recommended Makefile that you can use for compiling PERSEUS
in any shell. (It assumes that the above commands were auto-run for you when you opened the
shell.)
Linking an MPI Library
Just make sure you use the "mpifort" (or mpif90) command, and the MPI linking should be done
correctly. Remember, every MPI implementation comes with its own "mpif.h" file; therefore, that file
is not part of your code, and you should never copy it from system to system!
Linking Other Libraries
The Unix convention is that the option "-lmylib" refers to libmylib.so or libmylib.a. If both
libraries are present in the same directory, the .so file (shared-object or dynamic library) will
be chosen over the .a file (static library). If the library cannot be found at link time, you can
specify the path to it via the -L option to the compiler. (The compiler just passes this option to
the linker.)
FLASH
The FLASH code should be set up and built
separately for each particular user application. Therefore, to use FLASH, you should follow the
instructions in /data3/scratch/FLASH/README to produce your own custom build. The compiling and linking
commands, along with all the appropriate options, are encapsulated in /data3/scratch/FLASH/Makefile.h.
Note that FLASH is an MPI application, so it should be run with mpirun as described in the README.
Visualization and Analysis
Windowing in Linux is built around X11, a system for displaying individual windows locally or
remotely. But the performance of X11 can be draggy if your remote window is running a rich,
graphics-heavy application like ParaView or Mathematica. For doing interactive visualization, it is
preferable to set up an entire remote desktop. Several solutions exist for this, notably VNC and
OpenNX. The solution chosen for Hermes is X2Go.
Opening a Remote Desktop with X2Go
- First, download and install the appropriate
X2Go client on your laptop or
workstation.
- Once that's done, set up a VPN as you would for an ssh connection.
- Start the X2Go client. From the Session menu, choose "New Session...", then enter the
following:
| Session tab |
Connection tab |
Input/Output tab |
Media tab |
Shared folders tab |
Session name: hermes
Host: hermes.lps.cornell.edu
Login: <your username>
Session type - MATE (in drop-down) |
Connection speed -
WAN (move slider) |
Display - Custom -
Width: 1200, Height: 750 |
Enable sound support -
(clear check box)
Client side printing support -
(clear check box) |
(optional)
Path: <choose local path>
(click "Add")
(put checks in all 3 boxes) |
*Obviously these settings are just the minimal recommendations; you are welcome
to customize them further to suit your own preferences.
|
- OK the settings. A new "hermes" tile appears in the client window. Click it. Enter your password,
then "Ok".
- After a short wait, a new window should appear with a remote desktop on Hermes! Resize it
manually if you like.
- If you designated a shared folder, a file icon may appear on your desktop with a name ending in
(sshfs-disk-mate); ignore it.
Visit the X2Go website for complete information on
how to access your
shared folders.
- If you click the Pause button, or close the local window, your remote session continues to run!
You have merely disconnected it.
- To log back in, enter your password and hit "Ok" again. Your desktop should appear exactly as you
left it (even from a different machine!).
- To terminate the session completely, don't log out. Go back to the initial X2Go window and click
the 0/1 button.
If you need to terminate an X2Go session on Hermes that has become unreachable, the
easiest way to do so is to log in with ssh and enter the following, which will kill every
single process that you are running (including the shell you just started!):
pkill -9 -u $USER
In macOS, the X2Go client does not always have the correct key mappings. The familiar "Cmd+V" shortcut
for pasting the Clipboard may not work in xterm and other remote windows. Try the following
modification to .Xdefaults on your Mac to
restore the Cmd+V shortcut (a restart of XQuartz will be required):
echo "*VT100.translations: #override Meta V: insert-selection(PRIMARY, CUT_BUFFER0) \n" >> ~/.Xdefaults
If that doesn't work, simply right-click and choose "Paste" from the contextual menu, or use the
X11 mouse shortcut
for paste instead. (The latter trick may require you to enable 3-button-mouse emulation in XQuartz
Preferences.)
Finally, if you see the wrong characters showing up while you are typing on your Mac, disconnect your
session and reconnect to it. Often this is enough to correct the problem.
Running ParaView
From the MATE desktop, the ParaView 5.8.1 application can be started from the "Applications >
Graphics >" menu. An even handier shortcut is available on the top menu bar. If the ParaView icon
doesn't appear there, try entering the following command. (The second command is needed only if the
ParaView icon doesn't appear after you enter the first.)
mate-panel --reset
mate-panel --reset --layout rhel-pv
ParaView 5.8.1 or 4.3.1 can also be started from the command line; note, your PATH can be adjusted to
include the ParaView directory if you like.
VTK_DEBUG_SKIP_VISRTX_CHECK=1 /opt/ParaView-5.8.1-MPI-Linux-Python2.7-64bit/bin/paraview-mesa paraview &
/opt/ParaView-4.3.1-Linux-64bit/bin/paraview &
Note that with ParaView 5.8.1, paraview must be started through the paraview-mesa wrapper in order to
get OpenGL to work properly. If the application crashes, setting the indicated variable may prevent
trouble.
Running Mathematica
If you are logged in with ssh -X and have an X server running locally, just type "mathematica" in
your shell to start the GUI. If you want to run it an X2Go remote desktop, the following startup command
is necessary to avoid a Qt5 bug (or quirk):
QT_XCB_GL_INTEGRATION=none mathematica &
You will need an activation key if want to use Mathematica on Hermes. To get one, send an email to
the author of this page.
Setting Up Python 3 and Conda Environments
It turns out that Python 3.6 was introduced into standard CentOS between versions 7.6 and 7.8. After
this upgrade, it became unnecessary to use Python 3 from the Red Hat Software Collections (RHSCL).
Instead, the command "python3" is now sufficient to start a Python 3.6 interpreter. To make this your
default Python interpreter, simply add this to your .bashrc:
alias python=python3
The pip package manager is also installed and available through the command "pip3".
As an alternative, the Anaconda Python distribution is available in /usr/local/miniconda3. The
installation includes Python 3.9, as well as the handy conda and pip package managers. The latter are
very useful for customizing your preferred set of Python libraries and environments. To make Anaconda
your (permanent) default Python environment, run the following command:
/usr/local/miniconda3/bin/conda init
The above command modifies your .bashrc so Anaconda Python and its tools will be found ahead of
others. This in turn enables you to use conda environments to install one or more collections of Python
libraries to suit your own needs:
conda create -n <name-of-your-environment>
conda activate <name-of-your-environment>
You may want to edit your .bashrc to add the above "conda activate" command, following the lines
with conda's other modifications. That way, your custom environment will be automatically activated
every time you start a new shell. Conversely, you may want to deactivate all conda environments,
including the base environment, by default when you log in. To do that, run the following command:
conda config --set auto_activate_base false
This produces a ~/.condarc file that enforces the desired configuration change. It prevents conda's
environments from altering your $PATH and other environment variables in in ways that might interfere
with your non-Python applications.
Note that the NumPy and SciPy libraries have been preinstalled for you and are already included in
the Anaconda base environment.
Monitoring the Load
Here are some helpful commands to see what's happening on the machine. You may want to run them if
you're about to start a new MPI job, or if your current MPI job seems to be running more slowly than
you expected.
w
uptime
top
ps aux | grep -v root
A fully loaded 36-core machine will have a load average of 36 in "w" and "uptime". Anything over that
is likely to be using the cores inefficiently. The "top" command lets you monitor the machine's activity
continuously and in much more detail. It lets you see which processes are the heavy CPU users (type "q"
to quit). The one-shot "ps" command allows you to filter out things you don't want to see in "top".
Last updated on 2/10/22 by Steve Lantz (slantz ~at~ cac.cornell.edu)
