Community
cancel
Showing results for 
Search instead for 
Did you mean: 
TJ_AGI
Beginner
606 Views

Memory leak in dpotrf and dpotri

We are seeing a significant (> 800 MB) memory leak when calling dpotrf and dpotri to calculate the inverse of a symmetric matrix. We are using "MKL version Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64", building using MS Visual Studio 2019 and running on various Windows 10 PCs with different Intel processor architectures. The problem has also been seen when running the 2020 MKL release.

Whether the leak occurs or not depends on the matrix size, specific Intel processor we are running on, how many threads are being used, and whether we are configuring the number of threads using omp_set_num_threads() or mkl_set_num_threads(). 

I have attached test code that reproduces the problem, a batch file to run the code using various input parameters, and a spreadsheet that summarizes our observed cases when the leak occurs.

Any advice on how to avoid this issue would be appreciated as it's affecting our customers ability to run our software successfully. Thanks!

0 Kudos
18 Replies
TJ_AGI
Beginner
602 Views

Updated the attachment to include the batch file with test cases.

Gennady_F_Intel
Moderator
591 Views

Thanks for the report. This is an unknown issue with this version of MKL

Gennady_F_Intel
Moderator
590 Views

Linking dependencies: Psapi.lib;libiomp5md.lib;mkl_intel_lp64.lib;mkl_intel_thread.lib; mkl_core.lib;mkl_blas95_lp64.lib;

Do we need to have this lib to build the case?

TJ_AGI
Beginner
587 Views

Psapi.lib is used to support the test routine calls to GetProcessMemoryInfo() which is a Windows specific call to get the process size at the end of the test. You could rip that out and monitor the memory in some other fashion. Just watching it in the task manager is enough, just manually intensive when running lots of test cases.

Gennady_F_Intel
Moderator
555 Views

check the latest mkl 2020.1

>MatrixTest.exe 64 24 OMP MKL

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R)

64 architecture applications

Maximum number of OMP threads: 24

Maximum number of MKL threads: 24

Matrix size 64, Number of threads 24, Use MKL threads? 0

  MKL peak mem usage [KBytes]: 12456

Working set mem usage [Kbytes]: 5664

Working < 10000 K, no leak found!

Matrix test completed



Gennady_F_Intel
Moderator
540 Views

increasing the size of the problem, we could see this example reported the mem.leakage:

>MatrixTest.exe 1024 24 OMP MKL

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R)

64 architecture applications

Maximum number of OMP threads: 24

Maximum number of MKL threads: 24

Matrix size 1024, Number of threads 24, Use MKL threads? 0

  MKL peak mem usage [KBytes]: 267963

Working set mem usage [Kbytes]: 32848

Working set > 10000 K, looks like you are leaking!

Matrix test completed


TJ_AGI
Beginner
535 Views

I note that you are not dumping out the processor architecture at the beginning of the test (this is done in the test.bat file by examining the PROCESSOR_IDENTIFIER environment variable set by Windows). Whether we see a leak or not and at what size matrix and thread settings depends very much on the specific Intel processor we are running on (see the Excel spreadsheet I uploaded which lists the processors and the specific test run configuration that causes them to leak). Have you been able to reproduce any of these specific results using the 2019.0.4 build?

I note that in your run results you do not appear to have hyperthreading enabled (MKL processors is equal to OMP processors). 

I will try the latest 2020 build and see what results we get. Thanks!

TJ_AGI
Beginner
530 Views

I reran the test using 2020 rel 1 and still see a significant leak. Test results shown below. The highlighted results show the leak (scroll to the bottom 3 tests results). 

Test sequence start at Mon 07/06/2020 9:49:08.09
Processor architecture AMD64
Processor identifier Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 12, Use MKL threads? 0
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4492
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 12, Use MKL threads? 0
MKL peak mem usage [KBytes]: 18172
Working set mem usage [Kbytes]: 5844
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 1, Use MKL threads? 0
MKL peak mem usage [KBytes]: 6084
Working set mem usage [Kbytes]: 4812
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 2, Use MKL threads? 0
MKL peak mem usage [KBytes]: 12132
Working set mem usage [Kbytes]: 5088
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 4, Use MKL threads? 0
MKL peak mem usage [KBytes]: 18191
Working set mem usage [Kbytes]: 5372
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 100, Number of threads 4, Use MKL threads? 0
MKL peak mem usage [KBytes]: 24317
Working set mem usage [Kbytes]: 5416
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 6, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4472
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 1, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4468
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 2, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4472
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 32, Number of threads 4, Use MKL threads? 1
MKL peak mem usage [KBytes]: 0
Working set mem usage [Kbytes]: 4468
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 6, Use MKL threads? 1
MKL peak mem usage [KBytes]: 18172
Working set mem usage [Kbytes]: 5876
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 1, Use MKL threads? 1
MKL peak mem usage [KBytes]: 6084
Working set mem usage [Kbytes]: 4796
Working < 10000 K, no leak found!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 2, Use MKL threads? 1
MKL peak mem usage [KBytes]: 24213
Working set mem usage [Kbytes]: 827388
Working set > 10000 K, looks like you are leaking!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 60, Number of threads 4, Use MKL threads? 1
MKL peak mem usage [KBytes]: 18227
Working set mem usage [Kbytes]: 827904
Working set > 10000 K, looks like you are leaking!
Matrix test completed

Matrix test started
Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Maximum number of OMP threads: 12
Maximum number of MKL threads: 6
Note: OMP threads != MKL threads - Do you have hyperthreading enabled?
Matrix size 100, Number of threads 4, Use MKL threads? 1
MKL peak mem usage [KBytes]: 60586
Working set mem usage [Kbytes]: 828516
Working set > 10000 K, looks like you are leaking!
Matrix test completed

Test sequence stop at Mon 07/06/2020 9:59:14.33

Gennady_F_Intel
Moderator
526 Views

yes, it looks like a problem. The case will be escalated and we will keep you informed as soon as any updates.


Gennady_F_Intel
Moderator
524 Views

If you like you could also try to escalate the issue to the Intel Online Service Center - https://supporttickets.intel.com/?lang=en-US

TJ_AGI
Beginner
519 Views

Thanks for investigating this Gennady. I'll open up an issue with the service center.

TJ_AGI
Beginner
515 Views

For what it's worth, we do not see the memory leak in the 2017.0.2 MKL version which would suggest it was introduced sometime between that version and the 2019 release.

Gennady_F_Intel
Moderator
455 Views

ok, thanks for the updates. it useful. We will check the changes were committed since version 2017 into this component.


TJ_AGI
Beginner
248 Views

Any updates on this issue? I've been scanning the release notes for the current updates and haven't seen anything that suggests this issue has been addressed. Thanks!

Gennady_F_Intel
Moderator
233 Views

The fix of the issue is targeting to be released in the next update of oneMKL 2021.3. The announcement of this release will be posted on the top of this forum and we will keep this thread updated either.

TJ_AGI
Beginner
228 Views

Fantastic, thanks for the update!

KevinOB
Beginner
76 Views

We see 2021.3 is now available.  Is a fix available for this issue?

Gennady_F_Intel
Moderator
64 Views

yes, the issue could be fixed in this update. Please check this update and let us know the result.

Reply