Memory Accumulation: mkl_free_buffers does not seem to work

[Lange__Nils]

Hello,

I'm facing a problem with possible memory accumulation. I've programmed a FE software, wich solves the systems of equations with the Intel MKL DSS solver. My program is called from the commercial FE software Abaqus. In one solution step, several (lets say some 1000) systems of equations are solved (the program is called in parallel from Abaqus, so many soeq are solved in parallel). I call the DSS solver in the usual order: from dss_create ... dss_factor ... dss_delete. Everything seems to work fine, now I have problems with big models. Everything runs as expected for some iteration steps and then the program crashes. I called the mkl_mem_stat function (at the end of each iteration step, when all systems are solved) and it gives the following:

byte: 4127123520; n_buffer:1560
byte: 6878539200; n_buffer: 2600
byte: 9629954880; n_buffer: 3640
byte: 13069224480; n_buffer: 4940

I've tried to call mkl_free_buffers at the end of each iterations step, but it doesn't have any effect. I've also tried it in combination with mkl_disable_fast_mm. Also no effect.

Has anybody an idea how to free the memory allocated by the MKL Memory Management?

Best regards,

Nils

PS: im using INTEL ifor 2020.2, gcc 9.3.0, Abaqus 2019, CentOS Linux release 7.6

[Kirill_V_Intel]

Hello Nils,

Let me first say something about mkl_free_buffers: it releases MKL internal thread-local buffers while it tracks all memory allocated by MKL. This means, if there is a memory leak in the functionality which is not related to thread-local buffers (e.g., if it is an mkl_malloc without mkl_free), then this routine cannot remove it.

Typically we check for memory leaks in our internal tests and do so for DSS tests. So it is strange that you see a leak.

Is it possible that you provide us a reproducer for this problem?
I guess it will be enough to have one matrix and just duplicate it to mimic the 1000 systems which you're solving + have the workflow (= how you do this multi-system solving exactly). From your description it is not fully clear how you handle multiple systems, each with a separate DSS handle or not? We can get all the necessary information about the case from a reproducer. Also, it might be important to know how you link your application with MKL.

Last, but not least: I officially recommend to stop using DSS API and switch to PARDISO API. PARDISO API has greater flexibility and a lot of cool features, is better tested and is better than DSS in all aspects except one minor feature (which you are not using I hope).

Best,
Kirill

[Lange__Nils]

Hello Kirill,

many thanks for your reply. I switched to the PARDISO interface and no the memory leak disappeared. Even though that is good news, I'm still a bit disappointed, because wenn an Interface is offered, it should work properly.. Thanks anyway.

Best,

Nils

[Kirill_V_Intel]

Hello Nils,

Just to make sure I got it right, you don't see the memory leak with PARDISO API?

If so, that's good. Yet if you can, provide us some more data (one of the matrices and a code snippet which calls the solver through DSS API). Of course, we'd like to be able to fix the problem if memory leaks somewhere in the DSS interface. As I wrote earlier, it is unexpected, even for DSS.

Best,
Kirill

[Gennady_F_Intel]

regarding to DSS leaks - yes, I agree with Kirill, We have to fix the problem, but we have to have the reproducer on our end. Here is the link describing what we need to have to check the problem:

https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/How-to-post-in-the-Intel-oneAPI-Math-Kernel-Library-forum/m-p/1209982#M30079

[Gennady_F_Intel]

There is no reproducer available.

The issue is closing and we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread with reporducer. Any further interaction in this thread will be considered community only.