- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

We are seeing a significant (> 800 MB) memory leak when calling dpotrf and dpotri to calculate the inverse of a symmetric matrix. We are using "MKL version Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64", building using MS Visual Studio 2019 and running on various Windows 10 PCs with different Intel processor architectures. The problem has also been seen when running the 2020 MKL release.

Whether the leak occurs or not depends on the matrix size, specific Intel processor we are running on, how many threads are being used, and whether we are configuring the number of threads using omp_set_num_threads() or mkl_set_num_threads().

I have attached test code that reproduces the problem, a batch file to run the code using various input parameters, and a spreadsheet that summarizes our observed cases when the leak occurs.

Any advice on how to avoid this issue would be appreciated as it's affecting our customers ability to run our software successfully. Thanks!

Link Copied

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Thanks for the report. This is an unknown issue with this version of MKL

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Linking dependencies: **Psapi.lib**;libiomp5md.lib;mkl_intel_lp64.lib;mkl_intel_thread.lib; mkl_core.lib;mkl_blas95_lp64.lib;

Do we need to have this lib to build the case?

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Psapi.lib is used to support the test routine calls to GetProcessMemoryInfo() which is a Windows specific call to get the process size at the end of the test. You could rip that out and monitor the memory in some other fashion. Just watching it in the task manager is enough, just manually intensive when running lots of test cases.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

check the latest mkl 2020.1

>MatrixTest.exe 64 24 OMP MKL

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R)

64 architecture applications

Maximum number of OMP threads: 24

Maximum number of MKL threads: 24

Matrix size 64, Number of threads 24, Use MKL threads? 0

MKL peak mem usage [KBytes]: 12456

Working set mem usage [Kbytes]: 5664

**Working < 10000 K, no leak found!**

Matrix test completed

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

increasing the size of the problem, we could see this example reported the mem.leakage:

>MatrixTest.exe **1024** 24 OMP MKL

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R)

64 architecture applications

Maximum number of OMP threads: 24

Maximum number of MKL threads: 24

Matrix size 1024, Number of threads 24, Use MKL threads? 0

MKL peak mem usage [KBytes]: 267963

Working set mem usage [Kbytes]: 32848

Working set > 10000 K,** looks like you are leaking**!

Matrix test completed

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

I note that you are not dumping out the processor architecture at the beginning of the test (this is done in the test.bat file by examining the PROCESSOR_IDENTIFIER environment variable set by Windows). Whether we see a leak or not and at what size matrix and thread settings depends very much on the specific Intel processor we are running on (see the Excel spreadsheet I uploaded which lists the processors and the specific test run configuration that causes them to leak). Have you been able to reproduce any of these specific results using the 2019.0.4 build?

I note that in your run results you do not appear to have hyperthreading enabled (MKL processors is equal to OMP processors).

I will try the latest 2020 build and see what results we get. Thanks!

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

I reran the test using 2020 rel 1 and still see a significant leak. Test results shown below. The highlighted results show the leak (scroll to the bottom 3 tests results).

Test sequence start at Mon 07/06/2020 9:49:08.09

Processor architecture AMD64

Processor identifier Intel64 Family 6 Model 45 Stepping 7, GenuineIntel

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 32, Number of threads 12, Use MKL threads? 0

MKL peak mem usage [KBytes]: 0

Working set mem usage [Kbytes]: 4492

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 12, Use MKL threads? 0

MKL peak mem usage [KBytes]: 18172

Working set mem usage [Kbytes]: 5844

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 1, Use MKL threads? 0

MKL peak mem usage [KBytes]: 6084

Working set mem usage [Kbytes]: 4812

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 2, Use MKL threads? 0

MKL peak mem usage [KBytes]: 12132

Working set mem usage [Kbytes]: 5088

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 4, Use MKL threads? 0

MKL peak mem usage [KBytes]: 18191

Working set mem usage [Kbytes]: 5372

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 100, Number of threads 4, Use MKL threads? 0

MKL peak mem usage [KBytes]: 24317

Working set mem usage [Kbytes]: 5416

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 32, Number of threads 6, Use MKL threads? 1

MKL peak mem usage [KBytes]: 0

Working set mem usage [Kbytes]: 4472

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 32, Number of threads 1, Use MKL threads? 1

MKL peak mem usage [KBytes]: 0

Working set mem usage [Kbytes]: 4468

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 32, Number of threads 2, Use MKL threads? 1

MKL peak mem usage [KBytes]: 0

Working set mem usage [Kbytes]: 4472

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 32, Number of threads 4, Use MKL threads? 1

MKL peak mem usage [KBytes]: 0

Working set mem usage [Kbytes]: 4468

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 6, Use MKL threads? 1

MKL peak mem usage [KBytes]: 18172

Working set mem usage [Kbytes]: 5876

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 1, Use MKL threads? 1

MKL peak mem usage [KBytes]: 6084

Working set mem usage [Kbytes]: 4796

Working < 10000 K, no leak found!

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 2, Use MKL threads? 1

MKL peak mem usage [KBytes]: 24213

Working set mem usage [Kbytes]: 827388**Working set > 10000 K, looks like you are leaking!**

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 60, Number of threads 4, Use MKL threads? 1

MKL peak mem usage [KBytes]: 18227

Working set mem usage [Kbytes]: 827904**Working set > 10000 K, looks like you are leaking!**

Matrix test completed

Matrix test started

Using MKL version Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications

Maximum number of OMP threads: 12

Maximum number of MKL threads: 6

Note: OMP threads != MKL threads - Do you have hyperthreading enabled?

Matrix size 100, Number of threads 4, Use MKL threads? 1

MKL peak mem usage [KBytes]: 60586

Working set mem usage [Kbytes]: 828516**Working set > 10000 K, looks like you are leaking!**

Matrix test completed

Test sequence stop at Mon 07/06/2020 9:59:14.33

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

yes, it looks like a problem. The case will be escalated and we will keep you informed as soon as any updates.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

If you like you could also try to escalate the issue to the Intel Online Service Center - https://supporttickets.intel.com/?lang=en-US

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Thanks for investigating this Gennady. I'll open up an issue with the service center.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

For what it's worth, we do not see the memory leak in the 2017.0.2 MKL version which would suggest it was introduced sometime between that version and the 2019 release.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

ok, thanks for the updates. it useful. We will check the changes were committed since version 2017 into this component.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Any updates on this issue? I've been scanning the release notes for the current updates and haven't seen anything that suggests this issue has been addressed. Thanks!

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

The fix of the issue is targeting to be released in the next update of oneMKL 2021.3. The announcement of this release will be posted on the top of this forum and we will keep this thread updated either.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

We see 2021.3 is now available. Is a fix available for this issue?

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

yes, the issue could be fixed in this update. Please check this update and let us know the result.

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page