[Bug report] Memory leak in CPU plugin

[Lundgaard__Thomas]

Hello,

I am experiencing a big memory leak with the attached model. It appears that ExecutableNetwork does not deallocate everything when it is destructed.

I have only tried the cpu plugin, so I do not know if it is present in other plugins as well.

I am running the latest version of openvino (2018.5.456) on a Windows 10 machine with a Xeon W2133 cpu.

Find attached the problematic model and a code snippet to reproduce.

When executing the code, the commit size observed in Windows Task Manager continues to increase by approximately 50 MB for every iteration.

Regards,

Thomas

[Shubha_R_Intel]

Dear Lundgaard, Thomas,

There is no such leak in OpenVino. It could be coming from the gflags library or somewhere else. The leak is not within Inference Engine.

Thanks,

Shubha

[Lundgaard__Thomas]

In a previous post I pointed out the location of the memory leak. Looking at the source code for OpenVino 2019 R1.1 I can see that it is still not fixed.

I can reproduce this issue without gflags, so gflags is NOT the source of the leak.

I have also tried just loading the CPU plugin once, and still observe the leak. So this is NOT caused by Windows and multiple calls to LoadLibrary and FreeLibrary.

You previously acknowledged this as a bug in OpenVino. It is still not fixed. Can you please forward this information to the developers?

Thank you, Thomas

[Shubha_R_Intel]

Dear Lundgaard, Thomas,

I have forwarded this to OpenVino Developers. They state that there is no memory leak. I can certainly try again.

Thanks,

Shubha

[Lundgaard__Thomas]

Hi Shubha,

I would appreciate it if you send it to the developers again. If they persist that this is not a leak in openvino, it would be great if they can comment on my findings described in one of the posts above (https://software.intel.com/en-us/forums/computer-vision/topic/804912#comment-1934775).

Thank you,

Thomas

[Shubha_R_Intel]

Dearest Lundgaard, Thomas,

OpenVino 2019R2 should be released soon. I don't know exactly when, however. Here's my advice to you - and it kinda sucks because it increases your workload but it will be a surefire way to convince the developers.

Using https://github.com/opencv/dldt please build a self contained little project which demonstrates the issue without gflags. And I know that you've done this before (a long time ago) - I used your little project in fact to file a bug. Don't do this just yet however. Please wait for the R2 release.

Please see my detailed response to this github forum post for hints on how to build a self-contained project.

Would that be OK for you ? 

Thanks,

Shubha

[Lundgaard__Thomas]

Hi,

Yes I can do that when 2019R2 is released.

I wont build OpenVino from scratch as there is no need for that (and my trial of the intel c++ compiler has expired). I will just compile a small example application against the OpenVino binaries you distribute -- just as I did last time.

Thanks, Thomas

[Shubha_R_Intel]

Dearest Lundgaard, Thomas,

That would be great.

Thanks !

Shubha

[Lundgaard__Thomas]

Hi Shubha,

I have tried with 2019 R2 and still get the memory leak. The attached project reproduces it.

For every iteration in the loop I get about 8 MB of increase in memory usage.

I hope this will be fixed in 2019 R3.

Best regards,

Thomas

[Shubha_R_Intel]

Dear Lundgaard, Thomas,

Thanks for the attached project. I will see if I can reproduce the memory leak on R2.

Shubha

[Shubha_R_Intel]

Dear Lundgaard, Thomas,

Please see the attached image. I have run your sample program for over 4 minutes on OpenVino 2019R2.01 and I could not produce a memory leak. What I see is a constant memory usage of around 12 MB. If there were a memory leak, the memory usage would have increased during a 4 minute period but it didn't - it simply stayed constant. So there is no memory leak.

Shubha

[Lundgaard__Thomas]

Hi,

It appears that you never got it to run inference. You have to repeatedly press a key in the console window of the application. I made it like that so the program wont eat up all your memory. Alternatively you can remove the line "std::cin.ignore();" in main.cpp.

As you can see on the attached screenshot, there is indeed a memory leak on my machine.

- Thomas

[Shubha_R_Intel]

Dear Lundgaard, Thomas,

OK I didn't repeatedly hit a key in the console window. Let me try again. Looking at your image, I do see a tiny (not big) leak.

Thanks,

Shubha

[Marchetto__Enrico]

Hi there,

just to add to Thomas findings, I'm experiencig a very similar leak under Linux.

Recently I switched to 2019R2.0.1 but still identical behavior.

Could you please rise the priority of investigation on this issue?

Thanks a lot in advance!

E.

[Shubha_R_Intel]

Dear Marchetto, Enrico,

I would be glad to prioritize this but what are you specifically doing to cause your memory leak ? Thomas is literally doing just the short piece of code which follows. So is your situation different ? And if it is, can you kindly attach code to this ticket which demonstrates the problem ?

int main(char argv[])
{
	std::cout << "Loading inference plugin" << std::endl;
	auto plugin = PluginDispatcher({ "" }).getPluginByDevice("CPU");

	CNNNetReader networkReader;
	networkReader.ReadNetwork("saved.xml");
	networkReader.ReadWeights("saved.bin");

	auto cnnNetwork = networkReader.getNetwork();
	cnnNetwork.setBatchSize(1);

	while (true)
	{
		std::cout << "Press key to continue." << std::endl;
		std::cin.ignore();

		auto executableNetwork = plugin.LoadNetwork(cnnNetwork, {});
	}
}

 Thanks,

Shubha

[Lundgaard__Thomas]

Shubha,

Were you able to reproduce this?

[Shubha_R_Intel]

Dear Lundgaard, Thomas,

OpenVino 2019R3 should arrive in about a month (or less). I tried your code (hitting enter like you said for about 3 minutes) and I could not produce the leak on R3.  At this point, it makes little sense to go backwards and try and repro it on R2. I couldn't rig Visual Studio 2017's Memory Analyzer since this required my hitting enter and within Visual Studio the code just exited (failed). I am not sure how I can hit enter through Visual Studio - i tried researching this to no avail. I tried attaching to a process and launching an exe (I tried different ways) but Memory Profiler within Visual Studio is only available by launching the app. Remember last time I did this, I did it incorrectly as i didn't hit enter repeatedly as you later instructed.

But what I did finally was run your app at the command-line hitting enter repeatedly for about 3 minutes, and I used Resource Monitor. And I saw absolutely no evidence of a memory leak on R3. So it looks like R3 fixed it.

If you still see it in R3, please let us know.

Hope it helps,

Thanks

Shubha

[Marchetto__Enrico]

Shubha R. (Intel) wrote:

Dear Marchetto, Enrico,

I would be glad to prioritize this but what are you specifically doing to cause your memory leak ? Thomas is literally doing just the short piece of code which follows. So is your situation different ? And if it is, can you kindly attach code to this ticket which demonstrates the problem ?

int main(char argv[])
{
	std::cout << "Loading inference plugin" << std::endl;
	auto plugin = PluginDispatcher({ "" }).getPluginByDevice("CPU");

	CNNNetReader networkReader;
	networkReader.ReadNetwork("saved.xml");
	networkReader.ReadWeights("saved.bin");

	auto cnnNetwork = networkReader.getNetwork();
	cnnNetwork.setBatchSize(1);

	while (true)
	{
		std::cout << "Press key to continue." << std::endl;
		std::cin.ignore();

		auto executableNetwork = plugin.LoadNetwork(cnnNetwork, {});
	}
}

 Thanks,

Shubha

Dear Shubha,

I'm running a minimal code sample actually very similar to Thomas's one.

I see that you are not able to reproduce the leak with R3, which could be good news. Could you have a feedback from the developers to know if they actually addressed the issue?

Thanks a lot!

Best,

Enrico

[Shubha_R_Intel]

Dearest Enrico,

The developers rely on me to demonstrate that there is indeed a memory leak (using code such as Thomas's above) and so far, I've not been able to demonstrate this on R3.

Hope it helps,

Shubha

[Lundgaard__Thomas]

Dear Shubha,

Unfortunately, this leak is still present in 2019R3. See the attached screenshot. After running this for about 10 further mins my memory usage grew to 20 GB, at which point I stopped it.

Can you please please please send this to the developers now? We have been writing back and forth regarding this for more than 6 months now! Just because you are not able to reproduce the issue, it doesn't mean that there is no memory leak. I have reproduced this over and over again on different machines. I have even identified the bug in the source code (see my post above https://software.intel.com/en-us/forums/computer-vision/topic/804912#comment-1934775).

When you send it to the developers, make sure that you send the information in that post along with the bug report.

Thank you,

Thomas

[Shubha_R_Intel]

Dear Lundgaard, Thomas

I can only send it to the developers if I reproduce it.  To be fair, so far I have not been able to reproduce it. First I need to ask you, how do I run  your code within Visual Studio (Release Build since Debug Build can be inaccurate) ? You have to keep hitting enter within Visual Studio and memory profiler on. Please tell me the steps on how you accomplished this. Believe me I tried and failed before. I am using Visual Studio 2017.

I did manage to run your code outside of the Visual Studio Environment but by using Windows Resource Monitor, not Visual Studio's memory profiler. And as aforementioned, I could not reproduce the issue in R2.

I was thinking of putting a breakpoint in your code within Visual Studio and hitting enter - is that how you manage ? Because Visual Studio Memory Profiler clearly tells you to use a Release Build only for accurate results.

Thanks,

Shubha

[Lundgaard__Thomas]

Hi,

Do you need to reproduce it even when I can point you to the exact line in the code where the bug is? We are currently using a custom build of OpenVino 2018R3 with the fix I proposed. And that does indeed solve the issue.

I have reproduced this with both Debug and Release build in visual studio. The screenshot I send you was with a release x64 build. I dont use the memory profiler, but simply look in the "Diagnostic window" of Visual Studio. I can also see the increasing memory usage in the Windows task manager (look at the Commit size column). I have reproduced this on both Visual Studio 2019 and 2017.

On my machine, the memory usage goes up about 3 megabytes every time i press enter.

You need to be in the root folder of the project when executing the program, such that "saved.bin" and "saved.xml" can be found by the program.

Do you see a command line window that prints "Press key to continue." each time you press enter?

BR, Thomas