Skip to content

New Nodes (2018)

As of Dec 27, 2018, there are 4 new nodes in Proteus.

The nodes are:

  • ic23n01
  • ic23n02
  • ic23n03
  • ic23n04

Queue

  • All are in a new queue: new.q
  • Request jobs in new.q:

#$ -q new.q

  • To get an interactive session, use the qlogin command:

[juser@proteusi01 ~]$ qlogin -q new.q -l h_rt=48:00:00 -pe shm 40

Hardware Configuration

  • CPU: 2x Intel Xeon Gold 6148 (Skylake) 20-core 2.40 GHz (i.e. 40 cores total per node)
  • RAM: 192 GiB
  • Storage: 200 GiB SSD + 240 GiB RAID1 SSD for boot
  • Network: EDR InfiniBand (However, we are still limited to the QDR speeds and latency of our switches.)

System Configuration

  • Red Hat 6.8 -- newer than standard Proteus nodes
  • Mellanox OFED 3.4.1.0.0.1
  • The "base" directory where software is installed: /mnt/HA/opt_rh68/
  • Modulefiles to be used should preferably be from the directory (this should be set automatically on login): /mnt/HA/opt_rh68/modulefiles/
    • In your job scripts, add this on to the end of the base list of modules (i.e. before specific modules for software): module load proteus-rh68 (see below)
    • As usual, do module avail to see what is available
  • Intel Compilers
    • The latest (2019u1) Intel Composer XE compiler suite (with Python) will are available as a modulefile under the usual Proteus module directory (rather than the opt_rh68 one shown above):

intel/composerxe/2019u1

Job Scripts

Queue and Resource Requests

  • The nodes have 40 cores and 192 GiB of memory, which means at most 4915.2 MiB of memory per core for a 40-core job

Example:

#$ -q new.q
#$ -pe shm 40
#$ -l h_vmem=4915M
#$ -l m_mem_free=4000M

Base Modulefiles

The base modulefiles to be loaded should be in this order -- the ones loaded later are searched first:

module load shared module load proteus module load proteus-rh68 module load sge/univa module load gcc/7.4.0

Mellanox Infiniband Modulefiles

Open MPI makes use of Mellanox HPC-X® ScalableHPC Software Toolkit[1] software for performance.

In particular, the Open MPI modulefiles require HPC-X.

To use HPC-X, the HPC-X modulefile directory needs to be used:

$ module use /opt/mellanox/hpcx/modulefiles $ module load hpcx-stack $ module load proteus-openmpi/gcc/3.1.4

Compilation Options

GCC

  • For GCC, the architecture string is "skylake-avx512"

ICC

  • For ICC, compile with an interactive job, and specify the Intel compiler option "-xHost"

Benchmarks - High Performance Linpack (HPL)

Informal benchmark on 160 slots (4 nodes, 40 slots per node):

  • GCC + OpenBLAS: 148.5 GFLOPS
  • GCC + MKL 2019: 198.9 GFLOPS

See also: Compiling High Performance Linpack (HPL)

References

[1] Mellanox HPC-X® ScalableHPC Software Toolkit