Hardware

The HPC2 cluster is set up as an in-house high performance computing (HPC) facility at HKUST in Sep 2016.  It has 92 compute nodes which are infiniBand (IB) connected at 56 Gbit/s and 960 TB raw disk storage. The total number of CPU and GPU cores is 2248 and 69888 respectively and the theoretical peak performance is 120 TFLOPS.

Standard compute node (64GB RAM)

  • Number of nodes: 30
  • Hostname: hnode-ib-1 to hnode-ib-30
  • Server model: Lenovo NeXtScale nx360 M5
  • 2 x Intel Xeon E5-2670 v3 (12-core, 30M cache, 2.3 GHz, 9.6 GT/s Intel QPI)
  • 8 x 8G DDR4-2133 RAM
  • 1 x 1.2TB 10K rpm 12Gbps SAS HDD
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) Infiniband adapter

High end compute node (128GB RAM)

  • Number of nodes: 36
  • Hostname: hnode-ib-31 to hnode-ib-66
  • Server model: Lenovo NeXtScale nx360 M5
  • 2 x Intel Xeon E5-2670 v3 (12-core, 30M cache, 2.3 GHz, 9.6 GT/s Intel QPI)
  • 8 x 16G DDR4-2133 RAM
  • 1 x 1.2TB 10K rpm 12Gbps SAS HDD
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) Infiniband adapter

High end compute node (128GB RAM)

  • Number of nodes: 8
  • Hostname: hnode-ib-80 to hnode-ib-81, hnode-ib-83 to hnode-ib-88
  • Server model: Lenovo NeXtScale nx360 M5
  • 2 x Intel Xeon E5-2650 v4 (12-core, 30M cache, 2.2 GHz, 9.6 GT/s Intel QPI)
  • 8 x 16G DDR4-2400 RAM
  • 1 x 1.2TB 10K rpm 12Gbps SAS HDD
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) Infiniband adapter

High end compute node (128GB RAM)

  • Number of nodes: 5
  • Hostname: hnode-ib-82, hnode-ib-89 to hnode-ib-92
  • Server model: Lenovo NeXtScale nx360 M5
  • 2 x Intel Xeon E5-2683 v4 (16-core, 40M cache, 2.1 GHz, 9.6 GT/s Intel QPI)
  • 8 x 16G DDR4-2400 RAM
  • 1 x 1.2TB 10K rpm 12Gbps SAS HDD
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) Infiniband adapter

High end compute node (256GB RAM)

  • Number of nodes: 6
  • Hostname: hnode-ib-67 to hnode-ib-72
  • Server model: Lenovo NeXtScale nx360 M5
  • 2 x Intel Xeon E5-2670 v3 (12-core, 30M cache, 2.3 GHz, 9.6 GT/s Intel QPI)
  • 16 x 16G DDR4-2133 RAM
  • 1 x 1.2TB 10K rpm 12Gbps SAS HDD
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) Infiniband adapter

GPU compute node (128GB RAM)

  • Number of nodes: 7
  • Hostname: hnode-ib-73 to hnode-ib-79
  • Server model: Lenovo NeXtScale nx360 M5
  • 2 x Intel Xeon E5-2670 v3 (12-core, 30M cache, 2.3 GHz, 9.6 GT/s Intel QPI)
  • 8 x 16G DDR4-2133 RAM
  • 1 x 1.2TB 10K rpm 12Gbps SAS HDD
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) Infiniband adapter
  • 2 x Nvidia Tesla K80

Login-node

  • Number of node: 1
  • Hostname: hpc2
  • Server model: HP Proliant XL170r Gen9
  • 2 x Intel Xeon E5-2603 v3 (6-core, 15M cache, 1.6 GHz, 6.4 GT/s Intel QPI)
  • 8 x 4G DDR4-1600 RAM
  • 2 x 1TB 6Gb/s SAS 7.2K rpm HDD in RAID 1 configuration
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) infiniband adapter

Master node

  • Number of node: 1
  • Hostname: hpc2svr
  • Server model: HP Proliant XL170r Gen9
  • 2 x Intel Xeon E5-2603 v3 (6-core, 15M cache, 1.6 GHz, 6.4 GT/s Intel QPI)
  • 8 x 4G DDR4-1600 RAM
  • 2 x 500GB 6Gb/s SAS 7.2K rpm HDD in RAID 1 configuration
  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) infiniband adapter

Lustre Metadata Server (MDS)

  • Number of node: 1
  • Server model: HP Proliant XL190r Gen9
  • 2 x Intel Xeon E5-2640 v3 (8-core, 20M cache, 2.6 GHz, 8 GT/s Intel QPI)
  • 8 x 8G DDR4-1866 RAM
  • 2 x 500GB 6Gb/s SAS 7.2K rpm HDD in RAID 1 configuration as the system disk

  • 8 x 1.8TB 6Gb/s SAS 10K rpm HDD in RAID 60 configuration as the data disk

  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) infiniband adapter

Lustre Object Storage Server (OSS)

  • Number of node: 4
  • Server model: HP Apollo 4510 Gen9
  • 2 x Intel Xeon E5-2640 v3 (8-core, 20M cache, 2.6 GHz, 8 GT/s Intel QPI) / 2 x Intel Xeon E5-2640 v4 (10-core, 25M cache, 2.4 GHz, 8 GT/s Intel QPI)
  • 8 x 16G DDR4-2133 RAM / 8 x 16G DDR4-2400 RAM
  • 2 x 500GB/1TB 6Gb/s SAS 7.2K rpm HDD in RAID 1 configuration

  • 60 x 4TB 6Gb/s SAS 7.2K rpm HDD for internal storage

  • 2 x 1Gb Ethernet adapter
  • 1 x Mellanox dual port FDR (56Gb/s) infiniband adapter

Mellanox SX6036 Infiniband Switch with 36-port FDR port

  • Number of switch: 4

Lenovo RackSwitch G7052 with 48 x 1GbE port and 4 x 10GbE port

  • Number of switches: 4

Cluster Schematic Diagram

The simplified schematic diagram is as below.