Hermes Software Configuration Changes in 2019

Links below were helpful guides in what was done. It should be assumed that the indicated steps were executed by root.

The next general refresh came in 2019... and it basically advanced Hermes from CentOS 7.4 to 7.6.

This update was motivated by an upgrade of Mathematica from 11.3 to 12.0; the newer version had trouble with remote displays.
In particular, upon starting Mathematica 12.0 in an X2Go desktop, the following error would arise:

Qt issued a fatal error: Could not initialize GLX

Qt tried to load plugins from:
/usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64/Qt/plugins
/usr/local/Wolfram/Mathematica/12.0/SystemFiles/FrontEnd/Binaries/Linux-x86-64 ...
UPDATE: This issue and some earlier ones with ParaView were all resolved during the 2020 system refresh.

The "-d" option to Mathematica starts it within gdb. The stack backtrace at the crash point looks like this, in part:

#0  0x00007ffff29db2c7 in raise () from /lib64/libc.so.6
#1  0x00007ffff29dc9b8 in abort () from /lib64/libc.so.6
#2  0x00007ffff3b738d8 in QMessageLogger::fatal(char const*, ...) const ()
   from /usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64/Qt/lib/libQt5Core.so.5
#3  0x00007ffff7e4489f in ?? ()
   from /usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64/Qt/plugins/xcbglintegrations/
   libqxcb-glx-integration.so
#4  0x00007ffff7e41f27 in ?? ()
   from /usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64/Qt/plugins/xcbglintegrations/
   libqxcb-glx-integration.so
#5  0x00007fffcd52ab01 in QXcbIntegration::createPlatformOpenGLContext(QOpenGLContext*) const ()
   from /usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64/Qt/lib/libQt5XcbQpa.so.5
#6  0x00007ffff43f59ad in QOpenGLContext::create() () ...

Note that the error does not occur when X11 is forwarded through an SSH connection to a Mac running macOS 10.14 Mojave.
The X server is XQuartz 2.7.11. At first only "ssh -Y" would work; a change to .ssh/config was required to get "ssh -X" working.
SSH -X "Warning: untrusted X11 forwarding setup failed: xauth key data not generated"

echo "XAuthLocation /opt/X11/bin/xauth" > ~/.ssh/config

The X2Go issue seems to involve Qt5 and GLX. These GUI libs were implicated in prior issues with ParaView 5.4 that persisted in 5.5.
What to try next? The following command indicated that several Mesa libraries were out of date:

yum --showduplicates list mesa*

Hermes was overdue for a total system refresh, so a general "yum update" was the obvious thing to do at this point.
Based on past experience, the recommended follow-up to an OS refresh would be to update the NVIDIA drivers as well.
In order to download the correct drivers from the NVIDIA website, it is necessary to know the model of the video card:

lshw -numeric -C display
lspci | grep VGA

The above show that the card is a Quadro P600. Drivers for this card and for CUDA were downloaded from the NVIDIA website.
Before running "yum update", the CentOS website was consulted to see if the release notes mentioned any avoidable issues.
Seeing none, the update was run in terminal mode on the console, accessed by Ctrl-Alt-F6 (where Ctrl-Alt-F1 returns to GUI mode).
Access the Command Line in CentOS 7

yum update
shutdown -r +0

DKMS was already installed on Hermes, so it was assumed that NVIDIA kernel modules would be rebuilt correctly for the new kernel.
The update did in fact succeed, and CentOS 7.6 came up cleanly after the reboot.

New NVIDIA drivers can be installed by running scripts, but first, the X server must be shut down by a sudoer, again in terminal mode.

# "systemctl isolate multi-user.target" might also work for shutting down X11; use "init 5" to resume
init 3
chmod 775 *.run
./cuda_10.1.168_418.67_linux.run
# run this script after CUDA to install the most recent graphics drivers
./NVIDIA-Linux-x86_64-430.40.run
shutdown -r +0

DKMS again did the right thing, and the NVIDIA kernel modules were apparently rebuilt correctly based on the new drivers.
One anomaly was noted; it had cropped up previously. During reboots, the following message appears (see "dmesg | grep DEAD"):

[Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version 0xb000020 (or later)

However, "yum --showduplicates list microcode_ctl" shows that the installed microcode is the most recent version available.
This issue does not seem to prevent Hermes from working properly, so its resolution is apparently not urgent.
intel-ucode not updating?... (the "intel-ucode" package is a solution for Arch Linux, but not for CentOS)
[Firmware Bug]: TSC_DEADLINE disabled... (the "intel-microcode" package is a solution for Ubuntu, but not for CentOS)

Unfortunately, this whole system refresh still did not resolve the issues with remote access to Mathematica 12.0 or ParaView 5.4/5.5.
One speculation is that X2Go assumes the server has loaded the generic Nouveau driver rather than a vendor-supplied one.
The X2Go website has some possible workarounds, but not for CentOS 7; once again, could VirtualGL offer a potential solution?
Workaround for getting GLX 1.4 working
VirtualGL on CentOS 7 using NVIDIA Tesla GPUs
What does LIBGL_ALWAYS_INDIRECT=1 actually do?
Re-enabling INdirect GLX on your X server

Maybe conflicts exist between OpenGL libs (libGLX?) installed by NVIDIA vs. Mesa vs. - ? These commands may give relevant info:

cat /var/log/nvidia-installer.log
repoquery --list libglvnd
repoquery --list libglvnd-glx
rpm -qa | grep mesa
repoquery --list mesa-libGL
# can repeat for other mesa packages...
# check what Mathematica is picking up
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Wolfram/Mathematica/12.0/SystemFiles/Libraries/Linux-x86-64/Qt/lib
ldd /usr/local/Wolfram/Mathematica/12.0/SystemFiles/FrontEnd/Binaries/Linux-x86-64/Mathematica | grep libGL

Note, a follow-up "yum update" was done on 8/9/19, after a power outage which did no apparent damage to the disks or file systems.

And Python 3 was installed near the end of the year... via the Red Hat Software Collections, on 12/12/19.

The relevant guidance came from a Red Hat blog; however, it needed a slight CentOS twist.
How to install Python 3 on Red Hat Enterprise Linux
The Software Collections (SCL) Repository

yum install centos-release-scl
yum install rh-python36
yum install rh-python36-numpy
yum install rh-python36-scipy
yum install rh-python36-python-tools

The following information was added to the main Perseus page:

Python 3.6 is installed via Red Hat Software Collections (RHSCL). It allows Python 3 to coexist peacefully with the default python
(2.7) in CentOS. RHSCL enables you to configure your shell so that the "python" command, as well as several relevant environment
variables, all point to Python 3. NumPy, SciPy, and IDLE are included in the RHSCL installation on Hermes.

To start a subshell that is appropriately configured for Python 3, enter this command:

scl enable rh-python36 bash

To change all your shells, including your login shell, so that they will use Python 3 by default, add this line to your ~/.bashrc file:

source scl_source enable rh-python36

An extensive Red Hat blog post has plenty of good advice on how to use Python 3 (scl enable), install Python packages,
set up virtual environments, etc.



Last updated on 10/21/21 by Steve Lantz (steve.lantz ~at~ cornell.edu)