High Performance Computing Systems

Onyx Onyx
Onyx is a Cray XC40/50 system. It has 4,810 standard compute nodes, 4 large-memory compute nodes, 32 GPU compute nodes, 32 Knights Landing (Phi) compute nodes, and 64 Machine Learning Accelerator (MLA) multi-GPGPU nodes (a total of 4,942 compute nodes or 217,128 compute cores). It is rated at 6.06 peak PFLOPS.

Node Configuration
Login Standard Large-Memory KNL GPU 2-MLA 10-MLA
Total Nodes 12 4,810 4 32 32 60 4
Processor Intel E5-2699v4 Broadwell Intel E5-2699v4 Broadwell Intel E5-2699v4 Broadwell Intel 7230 Knights Landing Intel E5-2699v4 Broadwell Intel 6148 Skylake Intel 6148 Skylake
Processor Speed 2.8 GHz 2.8 GHz 2.8 GHz 1.3 GHz 2.8 GHz 2.4 GHz 2.4 GHz
Sockets / Node 2 2 2 1 1 2 2
Cores / Node 44 44 44 64 22 40 40
Total CPU Cores 528 211,640 176 2,048 704 2,400 160
Usable Memory / Node 247 GB 121 GB 1 TB 90 GB 247 GB 172 GB 735 GB
Accelerators / Node None None None None 1 2 10
Accelerator n/a n/a n/a n/a NVIDIA P100 PCIe3 NVIDIA V100 SXM2 NVIDIA V100 PCIe3
Memory / Accelerator n/a n/a n/a n/a 16 GB 16 GB 32 GB
Storage on Node None None None None None None None
Interconnect Ethernet Cray Aries Cray Aries Cray Aries Cray Aries InfiniBand InfiniBand
Operating System SLES CLE CLE CLE CLE CentOS CentOS

File Systems on Onyx
Path Formatted Capacity File System Type Storage Type User Quota Minimum File Retention
/p/home ($HOME) 897 TB Lustre HDD 100 GB None
/p/work ($WORKDIR) 13 PB Lustre HDD None 30 Days
/p/cwfs ($CENTER) 3.3 PB GPFS HDD 100 TB 120 Days
/p/app ($PROJECTS_HOME) 897 TB Lustre HDD None None