Links below were helpful guides in what was done. It should be assumed that the indicated steps were executed by root.
A batch of software requests came in April-May 2023... generally, these were needed to support the FLASH code.
A newer version of FLASH, 4.7, was placed into /data3/scratch/FLASH. The README and Makefile.h were updated accordingly.
VisIt is a nice complement to FLASH, so it too was installed. This had been attempted in 2021, but the process was different then.
Fortunately, VisIt is now downloadable from GitHub,
and there is even a readthedocs
document that describes how to install it.
The basic instructions outlined at readthedocs.io worked out in the end, though somewhat modified.
wget https://github.com/visit-dav/visit/releases/download/v3.3.3/visit3_3_3.linux-x86_64-rhel7-wmesa.tar.gz wget https://github.com/visit-dav/visit/releases/download/v3.3.3/visit-install3_3_3 chmod +x visit-install3_3_3 ./visit-install3_3_3 3.3.3 linux-x86_64-rhel7-wmesa /opt/VisIt-3.3.3
FLASH is meant to be built by users in various ways depending on the case to be simulated. This can bring in new dependencies.
For the current request, a few extra dependencies had to be installed to support the desired HEDP setup with an impinging laser field:
yum install hypre-devel # also installs SuperLU and openblas yum install hypre-openmpi # also installs mpich-3.2 (?!), opensm-libs-3.3, compat-openmpi16-1.6 yum install petsc # turned out to be unnecessary, but was installed anyway
The final requirement was hdf5-mpi, but this turned out to be a problem, because the Open MPI on Hermes has always been 2.1.1.
CentOS packages do exist for hdf5-openmpi and hdf5-openmpi3, but these packages depend on Open MPI 1.10 and 3.0 respectively.
Doing the following did not resolve the matter:
sudo yum install compat-openmpi16-devel
The crux of the issue is that the hdf5-openmpi package specifically requires libmpi_mpifh.so.12 and libmpi_usempi.so.5.
Unfortunately, it will not honor the existing, newer equivalents on Hermes, which are libmpi_mpifh.so.20 and libmpi_usempi.so.20.
The irony here is that HDF5 needs MPI only for doing parallel I/O with MPI-IO, which relies on a parallel file system like Lustre.
MPI-IO is utterly useless on a system like Hermes that has just plain HDDs. Yet this issue turned out to be a showstopper.
A further option, which was not pursued, was to try switching completely to MPICH, which in principle can be done easily on Hermes:
module load mpi/mpich-3.2-x86_64
But this switch would necessitate installing the available hdf5-mpich package, with unknown effects on FLASH as currently installed.
Around this time, Athena was set up for broader access, which revealed a few weaknesses with the default setup on Hermes.
The file /etc/profile.d/zzintel-openmpi.sh was improved by exporting PATH and LD_LIBRARY_PATH (only set locally, previously).
The script was also clarified by substituting "source" for ".". After editing, the file looked like this:
if [ -z "`echo $PATH | grep intel`" ]; then source /opt/intel/bin/ifortvars.sh intel64 fi #. /usr/share/Modules/init/sh #module unload mpi #module load mpi export PATH=/usr/bin:$PATH export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH export OMPI_FC=ifort