R6525 7H12 acceptance test

Mikica Kocic, Fysikum, SU, v4 (2021-04-12)

A performance anomaly was observed while testing a dual socket AMD 7H12 in Dell R6525.

Executive summary

The anomaly appears when more than a half of the 7H12-cores are employed; in particular, there is a significant drop in the CPU frequency & HPL performance when using >64 cores of a dual socket R6525 7H12. As a result, the efficiency of one R6525 7H12 node appears to be 20-25% lower than the efficiency of two R6525 7502 nodes connected via InfiniBand (or equally, two R6525 7H12 nodes with half of the cores used).

Note that the performance degradation is not thermal. Forcing all the fans to operate at 100% PWM cools down the CPUs from 75 degC to 65 degC during the tests. This CPU cooling, however, only slightly increases the observed performance (~1%).

Table 1 contains an overview of the observed HPL performance & CPU frequency for typical tests. The performance anomaly for 7H12 is visible for >64 cores (highlighted in red). Note that the Rpeak performance of an EPYC Rome/Zen2 CPU is calculated as:

     Rpeak/flops = number of cores * 16 * frequency/Hz

Here, the nominal CPU frequency of 7H12 is 2.6 GHz, while the nominal CPU frequency of 7502 is 2.5 GHz. This gives Rpeak(2x7H12) = 5.32 Tflops and Rpeak(2x7502) = 2.56 Tflops.

Table 1. Major test results

The performance drop and the kink point at 64 cores can be clearly seen in the following figures.

HPL tests, 4 threads, varying number of ranks

HPL tests, 8 threads, varying number of ranks

Test environment

Software

The Linpack benchmarks were done using HPL v2.3.
The problem size varied from N = 82816 to N = 351680.
The blocking sizes were NB = 128 and NB = 224.
The number of threads was fixed to either 1, 4 or 8, varying the number of MPI ranks.
The used OS were: CentOS 7.7 (kernel 3.10.0-1062.18.1.el7.x86_64), RHEL 8.3 (kernel 4.18.0-240.el8.x86_64) and Ubuntu 20.04 (kernel 5.4.0-70-generic).
The compilers were gnu 8.3, gnu 8.4, intel 2019, and AOCC 3.0.0.
The blas libraries were from MKL 2019.5.281 and AOCL 3.0-6.
The MPI was OpenMPI 3.1.4, OpenMPI 4.0.2 with UCX 1.7, MPICH 3.4.1, MVAPICH2 2.3.2 and intel MPI/2019.
The results are quoted for:
gnu 8.3, MKL 2019.5.281, OpenMPI 4.0.2, UCX 1.7, CentOS 7.7 (kernel 3.10.0-1062.18.1.el7.x86_64).

Hardware

11 servers R6525 with AMD 7502: c03n[01-11]
6 servers R6525 with AMD 7H12: c04n[01-06]
1 Dell EMC lab server with AMD 7502: node202
1 Dell EMC lab server with AMD 7H12: node206

Nodes `c03n[01-11]`

Dell PowerEdge R6525
2 x AMD EPYC 7502 2.5GHz, 32C/64T, 128M, 180W
16 x 32GB RDIMM, 3200MT/s, Dual Rank
Mellanox ConnectX-6 Single Port HDR100 QSFP56 Infiniband Adapter
Performance BIOS Settings

Nodes `c04n[01-06]`

Dell PowerEdge R6525
2 x AMD EPYC 7H12 2.6GHz, 64C/128T, 256M, 280W
16 x 64GB RDIMM, 3200MT/s, Dual Rank
Mellanox ConnectX-6 Single Port HDR100 QSFP56 Infiniband Adapter
Performance BIOS Settings

Fysikum's compute nodes have the following options modified when compared to the default R6525 BIOS settings: (1) hyperthreading disabled, (2) NUMA nodes per socket set to 4, and (3) InfiniBand card configured as a PCI preferred device.

CPU thermals

During the tests, the CPU temperature was ~75 degC. After setting System.ThermalSettings.FanSpeedOffset = Max (which forced the PWM of the fans to 100%), the average CPU temperature dropped down to 65 degC, as shown in the following figure. Nevertheless, cooling down of CPUs only slightly increased the observed performance (~1%).

The following figures show the thermal, power, frequency and performance profiles of some typical tests withoug max cooling (i.e., at 75 degC) when using 128, 96, 48 and 16 cores.