3rd generation Xeon scalable processor is much slower than 2nd generation in some MPI application

[Kuni]

Now we are studying network traffic of HPC use. For this, we are using Intel MPI Library (latest - Intel HPC kit) and Nas Parallel Benchmark (3.4.2). Before measuring network traffic, I measured the performance without using network traffic.  We used following platform:

machine1. Xeon Silver 4310 server 8ch 64GB RAM, Hyper thread off, Alma Linux, Turbo ON

machine 2. Xeon Silver 4214 server 6ch 96GB RAM Hyper thread on CentOS 7.9

machine 3.  4 core 8GB RAM virtual machine on machine 1. CentOS 7.9, Turbo OFF

machine 4. 4 core 8GB RAM vitual machine on machine 2. CentOS 7.9

Results: 

1.  mpirun -n 4 ./bin/bt.B.x (4 process smaller array)

machine 1.  44.64 sec

machine 2. 62.02 sec

machine 3. 43.92 sec

machine 4. 63.11 sec

2. mpirun -n 4 ./bin/bt.C.x  (4 process larger array)

machine 1. 347.85 sec

machine 2. 253.40 sec

machine 3. 184.26 sec

machine 4. 256.78 sec

In case of the above test 1, the result was understandable and performance diffrence was not strange and expected results were shown.

However, 2nd test. I saw very strange results. There is two unexped things.

1. Newer (3rd) generation of Xeon showed much slower result than older (2nd) generation of Xeon on real machine.

2. Newer (3rd) generation of Xeon showed big improvement , if the benchmark was executed on the virtual machine. 

I want to know why such a strange thing happen and want to know the way to improve real machine performance of bt.C.x benchmark (large array mpi program). 

Please help!.

K. Kunita

[JoseH_Intel]

Hello Kuni,

Thank you for joining the community

Looking at this processor comparison https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=215277,197100 we can see the 2nd gen Xeon 4214 is slightly a higher specs in some areas than the 3rd gen 4310. For example, their base and max turbo frequency is higher in the 4214 (2.4 MHz and 3.5 MHz) than the 4310 (2.1 MHz and 3.3 MHz). Besides that, the 4314 is a bit more power efficient even when it is a 14 nm based.

Been said that I can see your benchmark comparison is not really equal to equal as your 4214 machine is running on 96GB of RAM which is about 1/3rd more RAM than your 4310 system. Also, the OS distro is different on both systems.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[Kuni]

Thank you for looking at my issue.

In my explanation of machines, there is one thing I should mention. In case of machine 3, it was showed as "Turbo off" from lscpu output but it seems that virtual machine case, lscpu might not report correct CPU frequency. basically the cpu in the virtual machine is running at the same frequency as the host. Then machne 3 should treated as turbo on.

However the most big issue is that virtual machine showed much better performance than the result on the host machine.  (normally, virtual machine is little bit slower than its host machine. )  And the vitual machine's result  is the expected one from considering about the improvement between 3rd Gen. Xeon and 2nd gen Xeon.  And I also tried same OS destro (CentOS 7.9) the result is almost same. 

I also executed this benchmark with open MPI library instead of intel MPI library , at the case, the result is almost same between the host machine and the vitual machine.  Then it seems that something wrong thing might be happen with Intel MPI library. 

Then I think that I should ask this question to Intel one API HPC community.   Then I want to stop discussion here and move it to other thread related to Intel one API. 

[JoseH_Intel]

Hello Kuni,

Thank you for the notification about moving to the software thread. We will then proceed to mark this thread as closed. If you have further issues or questions just go ahead and submit a new topic.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios