Theoretical peak speed: (56 cores) * (2.6 Gcycles/s/core) * (8 DP flop/cycle) = 2.33 Tflop/s, double precision |
Assumes one AVX-512 FMA (fused multiply-add with 8 DP additions and 8 DP multiplications) every cycle |
Does not account for memory bandwidth limitations, or dynamic frequency scaling due to AVX-512 instructions |
Timeline | |
Early Oct. 2021: | Purchased from Red Barn Computers with Ubuntu 20.04 and selected applications preinstalled |
Late Oct. 2021: | Did additional installations of Open MPI and Intel oneAPI Fortran |
Jan. 2022: | Installed OpenSSH, new NVIDIA drivers, DKMS, X2Go server, and MATE desktop environment |
May 2023: | Did a system refresh, made preparations for adding remote users |
Nov. 2023: | Updated Paraview to 5.11.2 |
Hardware Configuration | |
Server Type: | Supermicro Super Workstation 740A-T - 4U tower - serial no. C7450FJ45MQ0180 |
(Appears be a custom build, as Red Barn's Chimera series is typically ASUS- or Tyan-based) | |
Processors: | 56 cores, 2x 28-Core Intel Xeon Gold "Ice Lake" Processors*, 6348, 42MB Cache, 2.60GHz |
*Sockets for this processor are LGA 4189 or "Socket P4" | |
Memory: | 128GB, 8x 16GB ECC Registered DDR4 SDRAM DIMMs, 3200MHz* |
*Highest speed possible | |
Bootable Hard Drive: | 1TB, Samsung 980 PRO SSD, PCIe 4.0, M.2 |
Additional Hard Drives: | 16TB, 2x Seagate Exos 7E8 512e SATA ST8000NM000A (each 8TB), 7200RPM SATA3 6.0Gbps, 3.5" |
Chipset: | Intel C621A ("Lewisburg"), 14 SATA3 ports |
Network Interface Controller: | Dual Intel Corporation I210 Gigabit Network Connections |
Motherboard and BIOS: | Supermicro X12DAi-N6; American Megatrends, Inc. (AMI) BIOS |
Graphics Accelerator: | GIGABYTE GeForce RTX 3080 TURBO 10G, 10GB GDDR6X, 8704 CUDA cores, 1710 MHz boost clock |
Interfaces: 2x HDMI 2.1, 2x DisplayPort 1.4a, PCI Express 4.0 x16 | |
Property | Drive Maker and Size | |||||||
Samsung SSD, 1TB | Seagate HDD, 8TB | Seagate HDD, 8TB | ||||||
Partition Names | /dev/nvme0n1p1 | /dev/nvme0n1p2 | /dev/nvme0n1p3 | /dev/sda | /dev/sdb | |||
Mount Points in Ubuntu | /boot/efi | (swap) | / (root) | /data1 | /data2 |
Hardware Interrogation Commands | |
Processors: | more /proc/cpuinfo |
Memory: | more /proc/meminfo; top; sudo dmidecode --type memory #grep for Size or Speed |
Bootable Hard Drives: | sudo fdisk -l | grep dev #bootable partitions are marked * |
Additional Hard Drives: | lsblk; sudo blkid; sudo gparted (Device Information panel in GParted GUI) |
RAID-0 Hard Drives (N/A): | lspci | grep -i raid; dmraid -r; cat /proc/mdstat #the last one detects software RAID |
I/O Controller Hub (N/A): | lspci | grep SATA; sudo dmidecode --type 8 | grep SATA |
Network Interface Controller: | lspci | grep -i ether |
Motherboard and BIOS: | sudo dmidecode --type baseboard; sudo dmidecode --type bios |
Graphics Accelerator: | lspci | grep VGA; nvidia-smi |
Devices and Mount Points: | sudo parted -l; df -h; mount |
Late Oct. 2021: Did additional installations of Open MPI and Intel oneAPI Fortran
$ sudo apt install openmpi-bin libopenmpi-dev $ wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB $ sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB $ rm GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB $ sudo add-apt-repository "deb https://apt.repos.intel.com/oneapi all main" $ sudo apt install intel-oneapi-compiler-fortran
source /opt/intel/oneapi/compiler/latest/env/vars.sh export OMPI_FC=ifort
Jan. 2022: Installed OpenSSH, new NVIDIA drivers, DKMS, X2Go server, and MATE desktop environment
$ sudo add-apt-repository ppa:graphics-drivers/ppa #this turned out to be unnecessary $ sudo apt update $ sudo apt install net-tools $ sudo apt install openssh-server $ sudo apt install dkms #ensuring that the NVIDIA kernel modules will be rebuilt, for initramfs $ sudo apt upgrade #(among other things, this step upgraded ifort from 2021.4 to 2022.0) $ sudo apt install xserver-xorg-video-nvidia-470 $ sudo reboot #this is what triggers DKMS
$ sudo apt install nvidia-driver-495 $ sudo apt autoremove #to clean up old packages $ sudo reboot #to trigger DKMS again
$ sudo add-apt-repository ppa:x2go/stable #output of this gave info for subsequent steps $ sudo apt-get install x2goserver x2goserver-xsession $ sudo apt install mate-core mate-desktop-environment mate-notification-daemon $ sudo apt-get install x2gomatebindings $ sudo apt install mesa-utils #for glxinfo diagnostics
$ paraview --mesa
May 2023: Did a system refresh, made preparations for adding remote users
$ sudo apt update $ sudo apt upgrade $ sudo reboot
$ sudo hostnamectl set-hostname athena.lps.cornell.edu
$ sudo apt install emacs $ emacs /etc/profile.d/zzintel-openmpi.sh #insert the desired commands... $ cat /etc/profile.d/zzintel-openmpi.sh if [ -z "`echo $PATH | grep intel`" ]; then source /opt/intel/oneapi/compiler/latest/env/vars.sh fi export HWLOC_COMPONENTS=-gl export OMPI_FC=ifort
$ sudo apt install ubuntu-mate-themes #this also installs ubuntu-mate-icon-themes $ sudo emacs /usr/share/glib-2.0/schemas/35_mate-ubuntu.gschema.override #a new file, with overrides... $ cat /usr/share/glib-2.0/schemas/35_mate-ubuntu.gschema.override [org.mate.desktop.background] picture-filename='/usr/share/backgrounds/mate/abstract/Waves.png' [org.mate.Marco.general] compositing-manager=false [org.mate.screensaver] idle-activation-enabled=false [org.mate.panel] default-layout='familiar-pv' $ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/ $ rm -rf ~/.config #log out, log back in, changes were in effect
$ sudo apt install mate-tweak $ mate-tweak $ dconf dump /org/mate/ | less #note positions, use them later for editing the layout $ gsettings get org.mate.panel default-layout 'familiar' $ sudo cp /usr/share/mate-panel/layouts/familiar.layout /usr/share/mate-panel/layouts/familiar-pv.layout $ sudo emacs !$ #edited the familiar-pv.layout file to add the MATE Terminal and ParaView launchers... $ diff /usr/share/mate-panel/layouts/familiar*.layout 25a26,39 > [Object mate-terminal] > locked=true > position=122 > launcher-location=/usr/share/applications/mate-terminal.desktop > toplevel-id=top > object-type=launcher > > [Object launcher] > locked=true > position=158 > launcher-location=/usr/share/applications/paraview-mesa.desktop > toplevel-id=top > object-type=launcher > $ sudo cp /usr/share/applications/paraview.desktop /usr/share/applications/paraview-mesa.desktop $ sudo emacs !$ #edited the paraview-mesa file so it would start properly in MATE desktop... $ gsettings set org.mate.panel default-layout 'familiar-pv' $ gsettings get org.mate.panel default-layout 'familiar-pv' $ mate-panel --reset $ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/ $ rm -rf ~/.config #log out, log back in, changes were in effect
$ cat /usr/share/applications/paraview-mesa.desktop [Desktop Entry] Version=1.0 Name=ParaView 5.10 (Mesa) Comment=Create 3d visualizations of data Exec=/opt/ParaView-5.10/bin/paraview --mesa Terminal=false Icon=/opt/ParaView-5.10/share/icons/hicolor/96x96/apps/paraview.png Type=Application x-Ayatana-Desktop-Shortcuts=NewWindows [NewWindow Shortcut Group] Name=New Window TargetEnvironment=Unity
Nov. 2023: Updated ParaView to 5.11.2
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
$ sudo apt-get install libxcb-xinerama0 $ sudo apt-get install '^libxcb.*-dev' libx11-xcb-dev libglu1-mesa-dev libxrender-dev libxi-dev \ libxkbcommon-dev libxkbcommon-x11-dev
$ sudo rm /usr/bin/paraview $ sudo ln -s /opt/ParaView-5.11.2/bin/paraview /usr/bin/paraview