Cornell-SYS-740A-T System Configuration

Theoretical peak speed: (56 cores) * (2.6 Gcycles/s/core) * (8 DP flop/cycle) = 2.33 Tflop/s, double precision

Assumes one AVX-512 FMA (fused multiply-add with 8 DP additions and 8 DP multiplications) every cycle

Does not account for memory bandwidth limitations, or dynamic frequency scaling due to AVX-512 instructions

Timeline

Early Oct. 2021:	Purchased from Red Barn Computers with Ubuntu 20.04 and selected applications preinstalled
Late Oct. 2021:	Did additional installations of Open MPI and Intel oneAPI Fortran
Jan. 2022:	Installed OpenSSH, new NVIDIA drivers, DKMS, X2Go server, and MATE desktop environment

Hardware Configuration

Server Type:	Supermicro Super Workstation 740A-T - 4U tower - serial no. C7450FJ45MQ0180
	(Appears be a custom build, as Red Barn's Chimera series is typically ASUS- or Tyan-based)

Processors:	56 cores, 2x 28-Core Intel Xeon Gold "Ice Lake" Processors*, 6348, 42MB Cache, 2.60GHz
	*Sockets for this processor are LGA 4189 or "Socket P4"
Memory:	128GB, 8x 16GB ECC Registered DDR4 SDRAM DIMMs, 3200MHz*
	*Highest speed possible
Bootable Hard Drive:	1TB, Samsung 980 PRO SSD, PCIe 4.0, M.2
Additional Hard Drives:	16TB, 2x Seagate Exos 7E8 512e SATA ST8000NM000A (each 8TB), 7200RPM SATA3 6.0Gbps, 3.5"
Chipset:	Intel C621A ("Lewisburg"), 14 SATA3 ports
Network Interface Controller:	Dual Intel Corporation I210 Gigabit Network Connections

Motherboard and BIOS:	Supermicro X12DAi-N6; American Megatrends, Inc. (AMI) BIOS

Graphics Accelerator:	GIGABYTE GeForce RTX 3080 TURBO 10G, 10GB GDDR6X, 8704 CUDA cores, 1710 MHz core clock
	Interfaces: 2x HDMI 2.1, 2x DisplayPort 1.4a, PCI Express 4.0 x16

Disk Structure

Property	Drive Maker and Size
Property	Samsung SSD, 1TB			Seagate HDD, 8TB	Seagate HDD, 8TB
Partition Names	/dev/nvme0n1p1	/dev/nvme0n1p2	/dev/nvme0n1p3	/dev/sda	/dev/sdb
Mount Points in Ubuntu	/boot/efi	(swap)	/ (root)	/data1	/data2

All user partitions are formatted with ext4 file systems.

Hardware Interrogation Commands

Processors:	`more /proc/cpuinfo`
Memory:	`more /proc/meminfo; top; sudo dmidecode --type memory #grep for Size or Speed`
Bootable Hard Drives:	`sudo fdisk -l \| grep dev #bootable partitions are marked *`
Additional Hard Drives:	`lsblk; sudo blkid; sudo gparted (Device Information panel in GParted GUI)`
RAID-0 Hard Drives (N/A):	`lspci \| grep -i raid; dmraid -r; cat /proc/mdstat #the last one detects software RAID`
I/O Controller Hub (N/A):	`lspci \| grep SATA; sudo dmidecode --type 8 \| grep SATA`
Network Interface Controller:	`lspci \| grep -i ether`
Devices and Mount Points:	`sudo parted -l; df -h; mount`

Software Configuration Changes

Late Oct. 2021: Did additional installations of Open MPI and Intel oneAPI Fortran

Steps to install Open MPI and the Intel oneAPI Fortran compiler in Ubuntu 20.04:

$ sudo apt install openmpi-bin libopenmpi-dev
$ wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
$ sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
$ rm GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
$ sudo add-apt-repository "deb https://apt.repos.intel.com/oneapi all main"
$ sudo apt install intel-oneapi-compiler-fortran

Ref. 1 - Install Open MPI from repository with GCC instead of Intel
Ref. 2 - Intel releases oneAPI Toolkit, free, Fortran 2018
Here are the required entries in .bashrc to make the above work together correctly:

source /opt/intel/oneapi/compiler/latest/env/vars.sh
export OMPI_FC=ifort

Don't run "source /opt/intel/oneapi/setvars.sh", as this activates Intel MPI along with the compiler.

Jan. 2022: Installed OpenSSH, new NVIDIA drivers, DKMS, X2Go server, and MATE desktop environment

The OS had "kept back" an unattended upgrade of libnvidia-common-470, leaving the machine without a graphics display.
This issue was resolved by manually installing one of the package's X11 dependencies (see https://askubuntu.com/a/602).
But before that, even, SSH access to the workstation had to be established by installing the OpenSSH server.

$ sudo add-apt-repository ppa:graphics-drivers/ppa   #this turned out to be unnecessary
$ sudo apt update
$ sudo apt install net-tools
$ sudo apt install openssh-server
$ sudo apt install dkms   #ensuring that the NVIDIA kernel modules will be rebuilt, for initramfs
$ sudo apt upgrade   #(among other things, this step upgraded ifort from 2021.4 to 2022.0)
$ sudo apt install xserver-xorg-video-nvidia-470
$ sudo reboot    #this is what triggers DKMS

The system was now ready to accept the latest NVIDIA drivers:

$ sudo apt install nvidia-driver-495
$ sudo apt autoremove   #to clean up old packages
$ sudo reboot   #to trigger DKMS again

The output of "nvidia-smi" and "lsmod | grep nvidia" gave sensible results after this, and the display started working again.
To improve remote access, X2Go server and MATE desktop environment were then installed:

$ sudo add-apt-repository ppa:x2go/stable   #output of this gave info for subsequent steps
$ sudo apt-get install x2goserver x2goserver-xsession
$ sudo apt install mate-core mate-desktop-environment mate-notification-daemon
$ sudo apt-get install x2gomatebindings
$ sudo apt install mesa-utils   #for glxinfo diagnostics

Ref. - Connect to Ubuntu 20.04 Remote Desktop using X2Go
The following command was found to be necessary to get ParaView 5.10 to work properly in X2Go:

$ paraview --mesa

Last updated on 2/3/22 by Steve Lantz (steve.lantz ~at~ cornell.edu)