possible perfomance regression on 2019 MKL (solved in 2021.3?)

[RWTHAachen]

We observe significant (4x..5x), reproducible raise of run time of one of our projects, when the binary of interest is (dynamically) linked to Intel MKL shipped with Intel compiler 19.1.3.304.

When the same binady is used and is linked against another version of MKL (by loading just another compiler), the run time is OK. Not affected are MKLs shipped with Intel Compilers 18.0.3.222, 19.0.1.144, 19.0.5.281, 2021.3.0 (so 19.1.3.304 is the only known version with that performance regression).

The application is programmed in Fortran, run w/48 threads on Intel(R) Xeon(R) Platinum 8160 w/48 cores (HT disabled) and known to call zgetrf, zgetrs, zcopy (and maybe more).

Is such an performance regression (likely fixed in 2021?) known?

Are intel developers interested in a reproducer?

Best

Paul Kapinos

===== Processor composition =====
Processor name : Intel(R) Xeon(R) Platinum 8160
Packages(sockets) : 2
Cores : 48
Processors(CPUs) : 48
Cores per package : 24
Threads per core : 1

$ for i in 18.0 19.0 19.0.5.281 19.1 2021.3.0 2021.3.0ng; do echo $i "\t\t"
$R_DELIMITER ; module switch intel intel/$i 2>/dev/null; time env
OMP_NUM_THREADS=48 ./test_kloop.x | grep test_kloop; echo; done
18.0 ------------------------------------------------------------------------------
env OMP_NUM_THREADS=48 ./test_kloop.x 42.66s user 2.66s system 4463% cpu 1.015
total
grep test_kloop 0.00s user 0.00s system 0% cpu 1.015 total

19.0 ------------------------------------------------------------------------------
env OMP_NUM_THREADS=48 ./test_kloop.x 44.60s user 1.45s system 4488% cpu 1.026
total
grep test_kloop 0.00s user 0.00s system 0% cpu 1.026 total

19.0.5.281 ------------------------------------------------------------------------------
env OMP_NUM_THREADS=48 ./test_kloop.x 42.91s user 2.43s system 4344% cpu 1.043
total
grep test_kloop 0.00s user 0.00s system 0% cpu 1.043 total

19.1 ------------------------------------------------------------------------------
env OMP_NUM_THREADS=48 ./test_kloop.x 233.22s user 2.92s system 4698% cpu 5.026
total
grep test_kloop 0.00s user 0.00s system 0% cpu 5.026 total

2021.3.0 ------------------------------------------------------------------------------
env OMP_NUM_THREADS=48 ./test_kloop.x 42.15s user 2.34s system 4380% cpu 1.016
total
grep test_kloop 0.00s user 0.00s system 0% cpu 1.015 total

2021.3.0ng ------------------------------------------------------------------------------
env OMP_NUM_THREADS=48 ./test_kloop.x 42.89s user 2.81s system 4376% cpu 1.044
total
grep test_kloop 0.00s user 0.00s system 0% cpu 1.044 total

[ShanmukhS_Intel]

Hi,

Thanks for reaching out to us.

We would like to request you to share the reproducer, so that we could look into the issue further.

Best Regards,

Shanmukh.SS

[RWTHAachen]

Dear Shanmukh.SS,

I've added a reproducer to Request 05188832

The software is a modified version of

(hier: https://co-design.pop-coe.eu/programs/jukkr-kloop/index.html).

[ShanmukhS_Intel]

Hi,

It seems there is no reproducer as mentioned, being attached to the Request ID 05188832.

Could you please reconfirm the reproducer, if not kindly attach the same so that we could work on your issue.

Best Regards,

Shanmukh.SS

[RWTHAachen]

crazy. I am able to see the file attached to the ticket:

$ md5sum jukkr-kloop.tar.bz2
7088e0191275695bf15707d12d429676 jukkr-kloop.tar.bz2

take a look at the screenshot!

[ShanmukhS_Intel]

Hi,

We have received the attachment. Apologies for inconvenience caused.

We are working on your issue. We will get back to you soon.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi,

Thanks for sharing the reproducer. 

We have tried compiling the source as per the steps shared. However, we faced some issues while running "compile_kloop_openmp.sh" in both 2019.1.3 and 2021.3.0 versions. Attached the error log with this thread. Could you please let us know if anything is being missed.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi,

Reminder:

Could you please help us with exact steps to compile the earlier shared code, as we are facing errors(shared the error log in previous reply) while compiling the code with the steps shared.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi,

Thanks for letting us know the details regarding performance of MKL with various versions. It seems the issue is with 2019 (19.1.3) version of MKL and it got resolved in version (2021.3). Since there is no issue in 2021.3, kindly let us know if we could close this issue.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi,

We assume that your query has been answered. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Best Regards,

Shanmukh.SS