Perseus System Configuration

Theoretical peak speed: (64 cores) * (2.67 Gcycles/s/core) * (4 DP flop/cycle) = 680 Gflop/s, double precision

Assumes a pipeline with 1 SSE addition (2 DP additions) and 1 SSE multiplication (2 DP multiplications) every cycle

Timeline

Sept. 2011:	Purchased from Red Barn Computers
Oct. 2011:	Fedora 15 installed on the 500GB HDD
	*At some point an Intel Xeon Phi (MIC card) was installed, but the required BIOS update failed, so the original state was restored
Sept. 2014:	CentOS 7 installed on an added (old) 500GB HDD; 2 (newer) 1TB HDDs added
Nov. 2014:	2 more newly purchased 1TB HDDs added and 2 RAID-0 pairs created, temporarily
	*Eventually RAID-0 pairs were dismantled due to CentOS boot problems, and the 4 1TB HDDs were reformatted as separate devices
Mar. 2015:	CentOS 7 became the default OS
Mar. 2016:	CentOS 7.2 became the default OS
Apr. 2017:	CentOS 7.3 became the default OS
Oct. 2017:	*Went offline due to multiple boot-time issues*

Hardware Configuration

Server Type:	Supermicro SuperServer 5086B-TRF - 5U rack-mountable - serial no. C75800A 25A00040

Processors:	64 cores, 8x Eight-Core Intel Xeon "Westmere-EX" Processors*, E7-8837, 24MB Cache, 2.67GHz
	*No real upgrade path; these are probably the best processors for this platform's LGA1567 or "Socket LS"
Memory:	256GB, 32x 8GB ECC Registered DDR3 SDRAM DIMMs*, 1066MHz
	*Potentially expandable to 2TB (64x32GB); 1066MHz is the highest speed possible on this platform
Bootable Hard Drives:	500GB, SATA Port 0, Seagate Momentus ST9500420AS, 7200RPM SATA2 3.0Gbps 2.5"
	500GB SATA Port 1, Samsung Spinpoint M8 ST500LM012 (HN-M500MBB), 5400RPM SATA2 3.0Gbps 2.5"
Additional Hard Drives:	4TB, SATA Ports 2, 3, 4 and 5, 4x HGST Travelstar HTS721010 (each 1TB), 7200RPM SATA3 6.0Gbps 2.5"
I/O Controller Hub:	Intel 82801JI (ICH10R "Southbridge" family), 4-port and 2-port SATA IDE controllers
	4 ports (slots 0-3) support "fake RAID" in 0, 1, and 10 configurations (no RAID 5 in Linux), 3.0 Gbps
Network Interface Controller:	Dual Intel Corporation 82576 Gigabit Network Connections

Motherboard and BIOS:	Supermicro X8OBN-F Platform; American Megatrends 4.6.3.7 for X8OBN 0.13 x64, 03/29/2011

All user partitions are formatted with ext4 file systems.
Each OS occupies a distinct hard drive with a distinct /home directory.
Fedora has just one partition to hold everything, including /boot and /home, while CentOS has 3 (at least).
For convenience, the CentOS partitions have been assigned mount points in Fedora in vice versa.
Cross-mounted $HOME is either /centoshome/$USER or /fedora/home/$USER (note the extra slash between fedora and home).
User storage locations are in (1) $HOME for each OS, as well as (2) scratch/$USER directories on the four 1TB hard drives.
In principle, each user may have a personal directory in the scratch locations, e.g., /userN/scratch/$USER.
In the table above, drives are listed in the same order in which they appear physically as one looks at the front of the server.

Dual-Boot Instructions

Possible Choices:	CentOS 7.3 by default, with the option to boot into the previous OS, Fedora 15 "Lovelock".
From GRUB Menu:	After the setup phase, when the GRUB menu appears, use the up/down arrows to select the desired OS, then hit <CR>.
	This selection is not remembered on the next boot. To make the change permanent, you have to alter the grub.cfg file.
From BIOS Boot Menu:	Scroll past the end of the page and select the option "Hard Drive BBS Priorities" that appears. (Scrolling up works too.)
	The pop-up will show all available "legacy" drives, not just one. Scroll to the desired drive and move it to the top with "+".
	(This is NOT documented in the manual, pp. 5-29 to 5-31!!) Use Esc and arrows to reach Save Configuration and Exit.

Hardware Interrogation Commands

Processors:	`more /proc/cpuinfo`
Memory:	`more /proc/meminfo; top; sudo dmidecode --type memory #grep for Size or Speed`
Bootable Hard Drives:	`sudo fdisk -l \| grep dev #bootable partitions are marked *`
Additional Hard Drives:	`blkid; gparted (Device Information panel in GParted GUI)`
RAID-0 Hard Drives:	`lspci \| grep -i raid; dmraid -r; cat /proc/mdstat #the last one detects software RAID`
I/O Controller Hub:	`lspci \| grep SATA; sudo dmidecode --type 8 \| grep SATA`
Network Interface Controller:	`lspci \| grep -i ether`

The links below lead to separate pages that give precise, step-by-step descriptions of what was done in each year.

Opted for the full suite of developer tools and GNOME desktop
Installed Intel Composer XE 2013 (license file was copied from the Fedora 15 boot disk)
Fixed issues with system time and Ethernet
Created two software RAID-0 sets from two pairs of 1TB HDDs
Undid these RAID-0 pairs after they broke due to changing the hostname (What happened?)

Synced all UIDs and GIDs with the previous Fedora OS
Installed OpenMPI and the screen utility
Installed X2Go and MATE desktop, since the latter was reported to be the best desktop environment for X2Go on CentOS 7
Downloaded and installed ParaView binary files, and built a few dependencies for ParaView
Tweaked the default MATE desktop to make it easier to launch ParaView
Made sure all shells were initializing Intel Composer XE and OpenMPI properly by default

Did "yum update" to try to fix an issue where the machine would become unresponsive for no good reason
Cleaned up the huge mess of partly installed software that resulted when the machine became unresponsive during the update (!)
Installed SciPy, SymPy, and python-matplotlib (during April)
Reseated all DIMMs as a potential cure for sporadic memory controller errors (during May—it didn't help)

Checked and repaired all file systems due to fresh reports of the server becoming unresponsive (in Feb.)
Detected possible conditions of overheating and sent the server to Red Barn for reconditioning (in Mar.)
Updated to CentOS 7.3 after the machine returned from Red Barn; this also triggered an update of the X2Go package
Noted an MPI error due to libfabric-1.3.0 that won't be fixed until the next CentOS version is released

Here's what people should do if the fans kick into high speed and won't settle down to a reasonable rate.

First, check whether someone is running a heavy-duty simulation. When that's happening, loud fan noise is normal.
But sometimes there may be no good reason; it may even occur immediately following a reboot.
In that case, try this: do a clean shutdown, unplug the machine from the wall, wait 10 seconds, plug it back in, restart.

Last updated on 9/10/18 by Steve Lantz (steve.lantz ~at~ cornell.edu)