Perseus, Slayer of Medusa, by jovial_cynic, 2008. Image freely copyable under a Creative Commons license. Some rights reserved.

Tips and Tricks for Running PERSEUS on Hermes and Athena, the Supermicro Servers Logging In       -  Connecting from Older Versions of Windows       -  Shell Navigation and the Screen Command       -  Shell Customization       -  Editors Storing and Accessing Files       -  Backups Compiling and Linking Codes       -  Using the Intel Fortran Compiler       -  Linking an MPI Library       -  Linking Other Libraries       -  FLASH Visualization and Analysis       -  Opening a Remote Desktop with X2Go       -  Running ParaView and VisIt       -  Running Mathematica       -  Setting Up Python 3 and Conda Environments Monitoring the Load

Hermes and Athena are high-performance workstations at the Laboratory of Plasma Studies at Cornell University. They are primarily intended for running the simulation code PERSEUS. For technical details on these resources, see the system configuration pages for the 36-core Hermes server and the 56-core Athena server. (You can also view a similar page for the now-defunct 64-core Perseus server, which was a nice resource in its time.) Athena is reserved for Prof. Seyler's use. This page is meant to be a basic user guide to Hermes and Athena.

These are Linux-based machines: Hermes runs CentOS 7.9 for its operating system, while Athena runs Ubuntu 20.04 LTS.

Logging In

Once you have been assigned a username, the easiest way to access Hermes or Athena from your local machine is to use ssh. The well-known Unix/Linux client for making network connections is included with macOS and Windows, too, through the Terminal application in these OSes. The LPS servers have the network names hermes.lps.cornell.edu and athena.lps.cornell.edu. To open a connection, enter:

ssh <your_username>@hermes.lps.cornell.edu

For security reasons, Hermes and Athena are on an ECE subnet that restricts incoming connections to those that appear to originate from an allowed Cornell subdomain. This means you must use VPN if you want your ssh client to connect from a network location outside of Cornell, or perhaps even from the public Wi-Fi on campus. See CU VPN to learn how to install and use Cornell's VPN software.

For viewing graphical displays or using GUI applications from Hermes and Athena, you must also have an X11 "X server" installed locally. X11 is again native to Unix/Linux, but implementations exist for macOS and Windows. (Additionally, there is a full remote desktop solution called X2Go; this is covered later.)

• macOS: Install XQuartz.
• Windows: Install WSL 2, the Windows Subsystem for Linux, which brings in a whole suite of Linux tools.
  (This Microsoft software is supported in up-to-date versions of Windows 10, 11, and Server 2022.)
  

One additional ssh option is needed before any remotely generated "X windows" can appear on your local desktop, e.g.,

ssh -X <your_username>@hermes.lps.cornell.edu

Note, if text labels are badly proportioned in ParaView's GUI, try checking the Override Font box in its Settings dialog. On Windows, if the text looks too big in any GUI, go to Start Menu > Preferences > Look and Feel > Appearance > Fonts > Details > Resolution, and change 192 to 96, e.g.

Connecting from Older Versions of Windows

PuTTY is a free, secure ssh client for connecting to Linux or Unix systems. To enable graphical display from your remote systems, PuTTY must be coupled with an X window server. VcXsrv and Xming (both free and open-source) are good choices for this purpose. Here is the sequence for starting an ssh session with Xming and X11 forwarding enabled:

• Start Xming by clicking its Quick Launch icon or by selecting it from the Start menu
• Start PuTTY via the shortcut on the desktop or via the Start menu
• Enter the address of your destination (or just click a name in your saved list and "Load" it)
• Include your username in the address line if you like, e.g., my_username@hermes.lps.cornell.edu
• Verify that SSH | X11 | Enable X11 forwarding is selected
• Select SSH | Enable compression, too, for better X11 performance
• Return to Session, give it a name if needed, and click "Save" so you can re-use it later
• Click "Open" to log in

After logging out from PuTTY, you can choose to quit Xming by right-clicking its icon in the System Tray and selecting "Exit".

Shell Navigation and the Screen Command

On Hermes and Athena, the default Linux shell (=OS command processor) is bash. Becoming competent in bash is a whole learning experience in itself. Here are some links to get you started:

• UNIX Tutorial for Beginners
• Shell stuff: job control and screen
• Introduction to GNU Screen

The last two links give you advice on (1) how to get a long-running code like PERSEUS out of your way while it is running, and (2) how to log out without killing your PERSEUS run. All by itself, screen will be adequate for both (1) and (2). Just do this:

screen
mpiexec -n 16 ./perseus
<Ctrl>-a d

The initial screen command places you into a new, detachable session. The final key sequence is what detaches the session and drops you back to your original shell. You can safely log out at this point. After you log back in, you can restore the previous session with "screen -R". To terminate the screen, first restore it, then kill all its processes and exit the screen. If you want to be totally safe when invoking screen (perhaps to the point of overkill), add nohup:

screen nohup mpiexec -n 16 ./perseus > outputfile &

Screen is per-session; it allows you to disconnect from a terminal session without killing it. Nohup (no-hangup) is per-process; it keeps the process running even if the parent shell goes away.

Shell Customization

Why would you want to customize your shell environment in Linux? The main reasons would be to put Open MPI tools such as mpif90 and mpiexec (mpirun) into your PATH, and to set up your environment to favor the Intel compilers. However, Hermes and Athena are already set up to do these things for you automatically every time you log in or start up a new shell. Therefore, the following information is presented just for your general enlightenment.

In bash, login shells are initialized by the file ~/.bash_profile (in Ubuntu, ~/.profile), while non-login interactive shells are initialized by the file ~/.bashrc. (Note, ~ and $HOME both refer to your home directory.) To ensure consistency between both types of interactive shells, the default ~/.bash_profile (or ~/.profile) merely calls the ~/.bashrc at login, before adding a few locations to the PATH. So why not just put tweaks to the PATH or other environment variables in ~/.bashrc? The problem is, a parent shell will export its environment variables to its child shells. So if one interactive shell starts another, which starts another... and ~/.bashrc is executed repeatedly... then PATH (e.g.) grows and grows as a consequence. That is why your default .bash_profile and .bashrc look like the following in CentOS 7, prior to any customization:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH
-- PUT YOUR ENVIRONMENT ADDITIONS HERE --<

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
-- PUT YOUR ALIAS ADDITIONS HERE --

You still have to be careful with such customizations because in Linux, shells you open on a remote desktop or in a screen session can have unexpected behavior. As a result, you may want to put if-then constructs into your .bashrc; testing is definitely a good idea. Note too that macOS behaves a bit differently from Linux; interactive shells started by the Terminal app are all login shells.

Editors

The usual Linux command-line editors are available: vi (really vim), emacs, and nano. If you prefer working with a basic GUI editor, gedit is available as well.

If you prefer emacs—meaning you have already learned emacs well enough not to be intimidated by it—then you don't necessarily have to start it in its own window via the "emacs" command. Instead, you might want to start it with "emacs -nw" so it occupies your current terminal window. Why? Generally it starts up faster that way in an "ssh -X" session. However, if you prefer to run emacs in a separate X window, that should be fine too (since the emacs window is no longer unstable due to a bug in GTK).

Storing and Accessing Files

The main locations to store your files are (1) in your $HOME, i.e., in the directory /home/$USER, and (2) on Hermes only, in a personal data directory, /dataN/$USER, which is on one of the two large-capacity hard drives on that system. Handy commands to see how much disk space is used/available at your current location are "du -s ." and "df ."

Backups

Data on Hermes and Athena are not backed up. To reduce the risk of data loss, keep copies of your important files on more than one of the above devices (or on external storage media). Here is a command that is useful for doing such a backup:

rsync -avz /dataP/$USER/ /dataQ/$USER

The trailing slash on the name of the first directory is significant—it means "get the contents of this directory". The interpretation of the flags, -avz, is as follows: archive the files with original properties, do it verbosely, and use compression while copying.

Compiling and Linking Codes

For everyone's convenience, certain commands are AUTOMATICALLY run in each new shell that is started on Hermes and Athena. These commands are the ones that must be run in your shell before you can compile and link your MPI codes using Open MPI and ifort, the Intel Fortran compiler. For the record, here are the commands that run automatically on Hermes:

if [ -z "`echo $PATH | grep intel`" ]; then
  . /opt/intel/bin/ifortvars.sh intel64
fi
PATH=/usr/bin:$PATH
LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH 
export OMPI_FC=ifort

And here are the ones that run automatically on Athena:

if [ -z "`echo $PATH | grep intel`" ]; then
   source /opt/intel/oneapi/compiler/latest/env/vars.sh
fi
export PATH=$PATH:/opt/ParaView-5.10/bin/
export HWLOC_COMPONENTS=-gl
export OMPI_FC=ifort

On less-tailored systems, you might have to place similar commands in your .bash_profile or .profile as noted above.

Using the Intel Fortran Compiler

Here is a recommended Makefile that you can use for compiling PERSEUS in any shell. (It assumes that the above commands were auto-run for you when you opened the shell.)

Linking an MPI Library

Just make sure you use the "mpifort" (or mpif90) command, and the MPI linking should be done correctly. Remember, every MPI implementation comes with its own "mpif.h" file; therefore, that file is not part of your code, and you should never copy it from system to system!

Linking Other Libraries

The Unix convention is that the option "-lmylib" refers to libmylib.so or libmylib.a. If both libraries are present in the same directory, the .so file (shared-object or dynamic library) will be chosen over the .a file (static library). If the library cannot be found at link time, you can specify the path to it via the -L option to the compiler. (The compiler just passes this option to the linker.)

FLASH

The FLASH code should be set up and built separately for each particular user application. Therefore, to use FLASH, you should follow the instructions in /data3/scratch/FLASH/README to produce your own custom build. The compiling and linking commands, along with all the appropriate options, are encapsulated in /data3/scratch/FLASH/Makefile.h. Note that FLASH is an MPI application, so it should be run with mpirun as described in the README.

Visualization and Analysis

Windowing in Linux is built around X11, a system for displaying individual windows locally or remotely. But the performance of X11 can be draggy if your remote window is running a rich, graphics-heavy application like ParaView or Mathematica. For doing interactive visualization, it is preferable to set up an entire remote desktop. Several solutions exist for this, notably VNC and OpenNX. The solution chosen for Hermes and Athena is X2Go.

Opening a Remote Desktop with X2Go

1. First, download and install the appropriate X2Go client on your laptop or workstation.
2. Once that's done, set up a VPN as you would for an ssh connection.
3. Start the X2Go client. From the Session menu, choose "New Session...", then enter the following (for Athena, modify this in obvious ways):

Session name: hermes
Host: hermes.lps.cornell.edu
Login: <your username>
Session type - MATE (in drop-down)

Connection speed -
WAN (move slider)

Display - Custom -
Width: 1600, Height: 900

Enable sound support -
  (clear check box)
Client side printing support -
  (clear check box)

(optional)
Path: <choose local path>
  (click "Add")
  (put checks in all 3 boxes)

4. OK the settings. A new "hermes" tile appears in the client window. Click it. Enter your password, then "Ok".
5. After a short wait, a new window should appear with a remote desktop on Hermes! Resize it manually if you like.
6. If you designated a shared folder, a file icon may appear on your desktop with a name ending in (sshfs-disk-mate); ignore it.
   Visit the X2Go website for complete information on how to access your shared folders.
7. If you click the Pause button, or close the local window, your remote session continues to run! You have merely disconnected it.
8. To log back in, enter your password and hit "Ok" again. Your desktop should appear exactly as you left it (even from a different machine!).
9. To terminate the session completely, don't log out. Go back to the initial X2Go window and click the 0/1 button.

If you need to terminate an X2Go session on Hermes that has become unreachable, the easiest way to do so is to log in with ssh and enter the following, which will kill every single process that you are running (including the shell you just started!):

pkill -9 -u $USER

In macOS, the X2Go client does not always have the correct key mappings. The familiar "Cmd+V" shortcut for pasting the Clipboard may not work in xterm and other remote windows. Try the following modification to .Xdefaults on your Mac to restore the Cmd+V shortcut (a restart of XQuartz will be required):

echo "*VT100.translations: #override Meta <KeyPress> V: insert-selection(PRIMARY, CUT_BUFFER0) \n" >> ~/.Xdefaults

If that doesn't work, simply right-click and choose "Paste" from the contextual menu, or use the X11 mouse shortcut for paste instead. (The latter trick may require you to enable 3-button-mouse emulation in XQuartz Preferences.)

Finally, if you see the wrong characters showing up while you are typing on your Mac, disconnect your session and reconnect to it. Often this is enough to correct the problem.

Running ParaView and VisIt

As mentioned previously, visualization applications like ParaView and Visit are likely to be more responsive if they are started up within an X2Go desktop.

From the MATE desktop on Hermes, the "ParaView 5.8.1" application can be started from the "Applications > Graphics >" menu; on Athena, look for "ParaView 5.10 (Mesa)" under "Menu > Other". An even handier shortcut is available on the top menu bar.

On Hermes, ParaView 5.8.1 or 4.3.1 can also be started from the command line. If you like, the command can be shortened by adjusting your PATH to include the ParaView directory.

VTK_DEBUG_SKIP_VISRTX_CHECK=1 /opt/ParaView-5.8.1-MPI-Linux-Python2.7-64bit/bin/paraview-mesa paraview &
/opt/ParaView-4.3.1-Linux-64bit/bin/paraview &

Note that with ParaView 5.8.1, paraview must be started through the paraview-mesa wrapper in order to get OpenGL to work properly. If the application crashes, setting the indicated variable may prevent trouble.

On Athena, ParaView 5.10 can be easily started from the command line, as the application should already be in your path:

paraview --mesa

mate-panel --reset
mate-panel --reset --layout rhel-pv   # replace rhel-pv with familiar-pv on Athena
rm -rf ~/.config   # undoes all your desktop customizations and reverts to system defaults

Only Hermes has VisIt, and you can start it with the following command:

/opt/VisIt-3.3.3/bin/visit -gui

Running Mathematica

If you are logged into Hermes with ssh -X and have an X server running locally, just type "mathematica" in your shell, and this will start up the GUI for version 13.3.

This also works in an X2Go remote desktop session, though your cursor may appear annoying offset from its actual position when you are typing commands into the GUI for version 13.3.

Version 13.1, which does not have the issue with the cursor, is also available as "mathematica.old". If you want to run the older Mathematica in an X2Go remote desktop, the following startup command is necessary to avoid a Qt5 bug (or quirk):

QT_XCB_GL_INTEGRATION=none mathematica &

You might need an activation key if want to use Mathematica on Hermes. To get one, send an email to the author of this page.

Setting Up Python 3 and Conda Environments

It turns out that Python 3.6 was introduced into standard CentOS between versions 7.6 and 7.8. After this upgrade, it became unnecessary to use Python 3 from the Red Hat Software Collections (RHSCL). Instead, the command "python3" is now sufficient to start a Python 3.6 interpreter. To make this your default Python interpreter, simply add this to your .bashrc:

alias python=python3

The pip package manager is also installed and available through the command "pip3".

As an alternative, the Anaconda Python distribution is available in /usr/local/miniconda3. The installation includes Python 3.9, as well as the handy conda and pip package managers. The latter are very useful for customizing your preferred set of Python libraries and environments. To make Anaconda your (permanent) default Python environment, run the following command:

/usr/local/miniconda3/bin/conda init

The above command modifies your .bashrc so Anaconda Python and its tools will be found ahead of others. This in turn enables you to use conda environments to install one or more collections of Python libraries to suit your own needs:

conda create -n <name-of-your-environment>
conda activate <name-of-your-environment>

You may want to edit your .bashrc to add the above "conda activate" command, following the lines with conda's other modifications. That way, your custom environment will be automatically activated every time you start a new shell. Conversely, you may want to deactivate all conda environments, including the base environment, by default when you log in. To do that, run the following command:

conda config --set auto_activate_base false

This produces a ~/.condarc file that enforces the desired configuration change. It prevents conda's environments from altering your $PATH and other environment variables in in ways that might interfere with your non-Python applications.

Note that the NumPy and SciPy libraries have been preinstalled for you and are already included in the Anaconda base environment.

Monitoring the Load

Here are some helpful commands to see what's happening on the machine. You may want to run them if you're about to start a new MPI job, or if your current MPI job seems to be running more slowly than you expected.

w
uptime
top
ps aux | grep -v root

A fully loaded 36-core machine will have a load average of 36 in "w" and "uptime". Anything over that is likely to be using the cores inefficiently. The "top" command lets you monitor the machine's activity continuously and in much more detail. It lets you see which processes are the heavy CPU users (type "q" to quit). The one-shot "ps" command allows you to filter out things you don't want to see in "top".