Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
358 Views

[Bug report] Memory leak in CPU plugin

Hello,

I am experiencing a big memory leak with the attached model. It appears that ExecutableNetwork does not deallocate everything when it is destructed.

I have only tried the cpu plugin, so I do not know if it is present in other plugins as well.

I am running the latest version of openvino (2018.5.456) on a Windows 10 machine with a Xeon W2133 cpu.

Find attached the problematic model and a code snippet to reproduce.

When executing the code, the commit size observed in Windows Task Manager continues to increase by approximately 50 MB for every iteration.

 

Regards,

Thomas

0 Kudos
45 Replies
Highlighted
160 Views

Bump

0 Kudos
Highlighted
Employee
160 Views

Hello Thomas. Here is the code in your main.cpp:

while (true)
{
    std::cout << "Press key to continue." << std::endl;
    std::cin.ignore();

    std::cout << "Loading inference plugin" << std::endl;
    auto plugin = PluginDispatcher({ L"" }).getPluginByDevice("CPU");

    CNNNetReader networkReader;
    networkReader.ReadNetwork("saved.xml");
    networkReader.ReadWeights("saved.bin");

    auto cnnNetwork = networkReader.getNetwork();
    cnnNetwork.setBatchSize(1);

    auto executableNetwork = plugin.LoadNetwork(cnnNetwork, {});
}

Your code has never left scope and all those objects are created via smart pointers. Why would you expect memory to be released while stuck in this while loop ? 

0 Kudos
Highlighted
160 Views

Hi,

The smart-pointers are local variables inside the loop and thus go out of scope at the end of each iteration of the loop. See for example here: https://stackoverflow.com/questions/6403055/object-destruction-in-c

0 Kudos
Highlighted
Valued Contributor I
160 Views

FWIW we are also seeing a statics order of destruction issue; not sure if this is related. Let's further debug this.

0 Kudos
Highlighted
Employee
160 Views

Hello Thomas, (1) does not compile - came from your sample. However (2) does compile - comes from the OpenVino samples.

1) auto plugin = PluginDispatcher({ L"" }).getPluginByDevice("CPU")

2) InferencePlugin plugin = PluginDispatcher({ FLAGS_pp, "../../../lib/intel64" , "" }).getPluginByDevice(FLAGS_d);

Can you kindly clarify ?

Thanks,

Shubha

0 Kudos
Highlighted
160 Views

This is related to whether the UNICODE preprocessor symbol is defined. The classification sample compiles without UNICODE and in this case the line should be changed to read

auto plugin = PluginDispatcher({ "" }).getPluginByDevice("CPU");

Or just use the one from the classification sample. Both will provoke the memory leak.

PS: I used to compile with UNICODE set, but found a scenario where this gave rise to an access violation (segfault) when calling from some c# code. There is a post elsewhere on this forum mentioning that. I now compile without UNICODE and still experience the memory leak.

0 Kudos
Highlighted
160 Views

Were you able to reproduce this?

0 Kudos
Highlighted
160 Views

Hi again,

I have investigated this and have found the location of the leak. It is a thread-local work buffer in MKL-DNN that doesn't get deallocated because OpenVino calls the constructors and destructors of MKL-DNN primitives from different threads.

Specifically, the global_scratchpad_t object in https://github.com/opencv/dldt/blob/2018/inference-engine/thirdparty/mkl-dnn/src/common/scratchpad.c... is allocated by a task created here https://github.com/opencv/dldt/blob/2018/inference-engine/src/mkldnn_plugin/mkldnn_graph.cpp#L1076 (and this allocation may therefore happen off the main thread). The scratchpad is deallocated when the InferenceEngine::ExecutableNetwork object in main() is destructed, which happens on the main thread. However, MKL-DNN uses thread local buffers and assumes that those two calls happen from the same thread.

The memory leak can be fixed by compiling MKL-DNN with preprocessor definition MKLDNN_ENABLE_CONCURRENT_EXEC. That may, however, increase memory usage (see https://github.com/intel/mkl-dnn/blob/master/cmake/options.cmake#L33). The real solution is probably to make sure that MKL-DNN primitives are constructed and destructed on the same thread.

- Thomas

0 Kudos
Highlighted
Valued Contributor I
160 Views

Good find Thomas, thank you! Hopefully this gets fixed soon.

I am going to try with MKLDNN_ENABLE_CONCURRENT_EXEC and keep an eye on memory usage to see if our issues get resolved.

cheers

nikos

0 Kudos
Highlighted
Employee
160 Views

Hi Thomas, with your help I have reproduced this issue and filed a bug. Thank You !

Shubha

0 Kudos
Highlighted
160 Views

Hi Shubha,

Do you know if this is fixed in the 2019 R1 release?

0 Kudos
Highlighted
Employee
160 Views

Dear Thomas, yes this leak has been fixed. Please download 2019 R1 and give it a test drive !

Thanks for using OpenVino !

Shubha

0 Kudos
Highlighted
160 Views

Hi Shubha,

I have just tried 2019 R1 and unfortunately I am still seeing this memory leak. Can you update the bug report with this information?

Thanks,

Thomas

0 Kudos
Highlighted
Employee
160 Views

Dear Thomas, 

Wow. Sorry for this. I will certainly update the bug report accordingly.

Thanks,

Shubha

0 Kudos
Highlighted
New Contributor I
160 Views

As far as I know the google gflags library has memory leak problem

0 Kudos
Highlighted
Employee
160 Views

Dear Thomas, this is not an OpenVino bug. Please review the following code:

//There is some leak per each LoadLibrary <-> FreeLibrary call on Windows. 

int main(int argc, char *argv[]) {

    LPCTSTR pluginName = "Full path to MKLDNNPlugin.dll";

    int c = 100;
    do { 

{         HMODULE object = LoadLibrary(pluginName); 

        FreeLibrary(object); 

         c–; 

    } 

 while (c > 0);
}

To avoid it, just to load plugin once:

InferencePlugin plugin = PluginDispatcher().getPluginByDevice("CPU");

 

0 Kudos
Highlighted
New Contributor I
160 Views

Hi Shubha, Could you answer my question here?

Shubha R. (Intel) wrote:

Dear Thomas, 

Wow. Sorry for this. I will certainly update the bug report accordingly.

Thanks,

Shubha

0 Kudos
Highlighted
160 Views

Hi Shubha,

Loading the plugin just once doesn't fix the leak either.

0 Kudos
Highlighted
Valued Contributor I
160 Views

Any update on this?

Is it an OpenVino bug or not?

If yes, ETA on a fix?

thanks!

Nikos

0 Kudos