Memory leak in dpotrf and dpotri

[TJ_AGI]

We are seeing a significant (> 800 MB) memory leak when calling dpotrf and dpotri to calculate the inverse of a symmetric matrix. We are using "MKL version Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64", building using MS Visual Studio 2019 and running on various Windows 10 PCs with different Intel processor architectures. The problem has also been seen when running the 2020 MKL release.

Whether the leak occurs or not depends on the matrix size, specific Intel processor we are running on, how many threads are being used, and whether we are configuring the number of threads using omp_set_num_threads() or mkl_set_num_threads(). 

I have attached test code that reproduces the problem, a batch file to run the code using various input parameters, and a spreadsheet that summarizes our observed cases when the leak occurs.

Any advice on how to avoid this issue would be appreciated as it's affecting our customers ability to run our software successfully. Thanks!

[TJ_AGI]

Updated the attachment to include the batch file with test cases.

[Gennady_F_Intel]

Thanks for the report. This is an unknown issue with this version of MKL

[Gennady_F_Intel]

Linking dependencies: Psapi.lib;libiomp5md.lib;mkl_intel_lp64.lib;mkl_intel_thread.lib; mkl_core.lib;mkl_blas95_lp64.lib;

Do we need to have this lib to build the case?

[TJ_AGI]

Psapi.lib is used to support the test routine calls to GetProcessMemoryInfo() which is a Windows specific call to get the process size at the end of the test. You could rip that out and monitor the memory in some other fashion. Just watching it in the task manager is enough, just manually intensive when running lots of test cases.

[Gennady_F_Intel]

check the latest mkl 2020.1

>MatrixTest.exe 64 24 OMP MKL

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R)

64 architecture applications

Maximum number of OMP threads: 24

Maximum number of MKL threads: 24

Matrix size 64, Number of threads 24, Use MKL threads? 0

  MKL peak mem usage [KBytes]: 12456

Working set mem usage [Kbytes]: 5664

Working < 10000 K, no leak found!

Matrix test completed

[Gennady_F_Intel]

increasing the size of the problem, we could see this example reported the mem.leakage:

>MatrixTest.exe 1024 24 OMP MKL

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R)

64 architecture applications

Maximum number of OMP threads: 24

Maximum number of MKL threads: 24

Matrix size 1024, Number of threads 24, Use MKL threads? 0

  MKL peak mem usage [KBytes]: 267963

Working set mem usage [Kbytes]: 32848

Working set > 10000 K, looks like you are leaking!

Matrix test completed

[TJ_AGI]

I note that you are not dumping out the processor architecture at the beginning of the test (this is done in the test.bat file by examining the PROCESSOR_IDENTIFIER environment variable set by Windows). Whether we see a leak or not and at what size matrix and thread settings depends very much on the specific Intel processor we are running on (see the Excel spreadsheet I uploaded which lists the processors and the specific test run configuration that causes them to leak). Have you been able to reproduce any of these specific results using the 2019.0.4 build?

I note that in your run results you do not appear to have hyperthreading enabled (MKL processors is equal to OMP processors). 

I will try the latest 2020 build and see what results we get. Thanks!

[TJ_AGI]

I reran the test using 2020 rel 1 and still see a significant leak. Test results shown below. The highlighted results show the leak (scroll to the bottom 3 tests results). 

Test sequence start at Mon 07/06/2020 9:49:08.09
Processor architecture AMD64
Processor identifier Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 12, Use MKL threads? 0
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4492
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 12, Use MKL threads? 0
MKL peak mem usage [KBytes]: 18172
Working set mem usage [Kbytes]: 5844
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 1, Use MKL threads? 0
MKL peak mem usage [KBytes]: 6084
Working set mem usage [Kbytes]: 4812
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 2, Use MKL threads? 0
MKL peak mem usage [KBytes]: 12132
Working set mem usage [Kbytes]: 5088
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 4, Use MKL threads? 0
MKL peak mem usage [KBytes]: 18191
Working set mem usage [Kbytes]: 5372
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 100, Number of threads 4, Use MKL threads? 0
MKL peak mem usage [KBytes]: 24317
Working set mem usage [Kbytes]: 5416
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 6, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4472
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 1, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4468
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 2, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4472
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 4, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4468
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 6, Use MKL threads? 1
MKL peak mem usage [KBytes]: 18172
Working set mem usage [Kbytes]: 5876
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 1, Use MKL threads? 1
MKL peak mem usage [KBytes]: 6084
Working set mem usage [Kbytes]: 4796
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 2, Use MKL threads? 1
MKL peak mem usage [KBytes]: 24213
Working set mem usage [Kbytes]: 827388
Working set > 10000 K, looks like you are leaking!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 4, Use MKL threads? 1
MKL peak mem usage [KBytes]: 18227
Working set mem usage [Kbytes]: 827904
Working set > 10000 K, looks like you are leaking!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 100, Number of threads 4, Use MKL threads? 1
MKL peak mem usage [KBytes]: 60586
Working set mem usage [Kbytes]: 828516
Working set > 10000 K, looks like you are leaking!
Matrix test completed

Test sequence stop at Mon 07/06/2020 9:59:14.33

[Gennady_F_Intel]

yes, it looks like a problem. The case will be escalated and we will keep you informed as soon as any updates.

[Gennady_F_Intel]

If you like you could also try to escalate the issue to the Intel Online Service Center - https://supporttickets.intel.com/?lang=en-US

[TJ_AGI]

Thanks for investigating this Gennady. I'll open up an issue with the service center.

[TJ_AGI]

For what it's worth, we do not see the memory leak in the 2017.0.2 MKL version which would suggest it was introduced sometime between that version and the 2019 release.

[Gennady_F_Intel]

ok, thanks for the updates. it useful. We will check the changes were committed since version 2017 into this component.

[TJ_AGI]

Any updates on this issue? I've been scanning the release notes for the current updates and haven't seen anything that suggests this issue has been addressed. Thanks!

[Gennady_F_Intel]

The fix of the issue is targeting to be released in the next update of oneMKL 2021.3. The announcement of this release will be posted on the top of this forum and we will keep this thread updated either.

[TJ_AGI]

Fantastic, thanks for the update!

[KevinOB]

We see 2021.3 is now available.  Is a fix available for this issue?

[Gennady_F_Intel]

yes, the issue could be fixed in this update. Please check this update and let us know the result.