Memory Leak in MFXVideoENCODE_EncodeFrameAsync

[monaco__joe]

I'm seeing a slow memory leak when using mediaSDK to perform MPEG2 encoding.  I see this for both hardware/software with the attached code. In this code the main loops continually processes the same frames and contains no memory allocation.  Yet the memory used by the process continually creeps up.  The examples below show only 60 seconds of history, but the trend continues in the same manner until memory is exhausted.  This can take many hours, but eventually the process will consume all free memory in the machine.

For hardware, the example app generates a sequence like this on my system:

./msdkTest.exe
Using HW version 1.7
6 buffers suggested
Starting frame encode
Elapsed (s) DeltaMemory
          0      737280
         10     1368064
         20     2138112
         30     2625536
         40     3149824
         50     3710976
         60     4644864
 

Likewise for software, I see something like this:

./msdkTest.exe -s
Using SW version 1.8
3 buffers suggested
Starting frame encode
Elapsed (s) DeltaMemory
          0       24576
         10       36864
         20       49152
         30      118784
         40      184320
         50      253952
         60      286720

Am I doing something wrong in this simple loop?  I've run this same code on a difference machine with HW version 1.4 and SW version 1.7 and there was no leak so I believe my loop is OK.

[Jeffrey_M_Intel1]

From a quick read through your code I'm not seeing anything obviously wrong.  However, based on more tests with running long encodes I'm also seeing what could be slow memory leaks for the Windows HW mpeg2 and h264 implementations.  I'll investigate more and get back to you.  In the meantime, any additional details about driver version(s) and processor(s) tested on your end could help.

Thanks!

Jeff  

[monaco__joe]

I'm built win32 application in vs2010 from attached C src and ran on two systems.

The first system had no memory leak using HW 1.4 and SW 1.7.  For this first system Media SDK System Analyzer gives:

Graphics Devices:

Name Version State

NVIDIA NVS 5400M 9.18.13.1269 08

Intel(R) HD Graphics 4000 9.17.10.2843 Active

System info:

CPU: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz

OS: Microsoft Windows 7 Enterprise

Arch: 64-bit

[monaco__joe]

On the second system I'm running the same application and wind up using HW 1.7 and SW 1.8.

I'm definitely seeing memory usage grow for HW 1.7 but I'm no longer seeing any issue with the SW version.  Not sure what has changed.

Hardware log looks like as follows (pretty much same as previous post).

Using HW version 1.7
6 buffers suggested
Starting frame encode
Elapsed (s) DeltaMemory
          0      778240
         10     1302528
         20     2125824
         30     2580480
         40     3072000
         50     3756032
         60     4435968
         70     4870144

Here's what I see from the analyzer:

Graphics Devices:
        Name                                         Version             State
        Intel(R) HD Graphics 4600                    10.18.10.3345       Active
        VNC Mirror Driver                            1.8.0.0             08
        NVIDIA GeForce GTX 650                       9.18.13.1407        08

System info:
        CPU:    Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
        OS:     Microsoft Windows Embedded Standard
        Arch:   64-bit

[Jeffrey_M_Intel1]

Thanks!  This gives some important clues.

[monaco__joe]

Jeff,

Any update on this leak.  For example have you been able to duplicate it? 

Do you have any suggestions for workarounds?  Currently I can use SW implementation for SD, but HD requires too many CPU cycles in my application.

[Jeffrey_M_Intel1]

I've replicated the leak and escalated to the appropriate teams.  We will get a fix out as soon as possible.  Don't know how acceptable these workarounds will be in your situation, but there are a few possibilities:

* Definitely understand that the memory leak is unacceptable for production, but is it possible to proceed with shorter tests that don't exhaust memory until a fix can be released?  
* Can development/testing requiring longer tests be done on 3rd Generation Core/Ivybridge?  I was only able to replicate the leak on  4th Generation Core/Haswell.

Hopefully this is very temporary and application development can continue.  Please let us know if this will affect any of your release deadlines.  If this is sensitive information please feel free to use a private message.

Regards, Jeff

[Andrew_C_1]

Wow, 

Excellent investigation, Carsten K! 

I'm seeing the same behavior of memory leak using hardware, but not with software. I've tried several versions of the SDK (2014, 2013 R2, 2012 R3) and they all exhibit the same behavior (only tested on my machine so far, using an i7 4750HQ) . My usage is a bit different from most others that I've seen in the forums, though.

We're encoding captured video frames in real time to be sent out in real time as h.264 (we do the audio encoding and muxing ourselves). When passing the encoder 1280x720 frames with 32bit ARGB pixels at 30 frames per second, I see an average of one MB/minute memory leakage over the course of several hours using the HW implementation. When using SW implementation, memory is extremely steady for as long as the application runs (we do overnight tests several days a week). 

If it would help, I can condense the code to the initialization and encode functionality and post it here.

-Andrew

[Jeffrey_M_Intel1]

Carsten, thanks for all of the great details!  They are very helpful.  I've passed a reproducer for the decode memory leak to the dev team as well.  

[jrberry]

Hi all,

Is there any update on this problem?  We are running tests decoding h.264 HD material and transcoding to an h.264 proxy.

Running this on a 32 core machine we run out of memory after processing about 100 clips.  Our experiments indicate that the number of cores effects the rate at which the memory is lost.

Thanks.

[Jeffrey_M_Intel1]

Sorry for the delay.  The 15.33.0.3496 driver, available now as beta from downloadcenter.intel.com, has a fix for this leak.

Decode and encode with system memory now have stable memory usage.  For example, on my test machine with the previous driver memory use increased about 50 MB per 100k frames of HD encode but now I am not seeing any increase over long runs.  

The schedule for the next non-beta release is still TBD, but the importance is understood and we're working to get this out as soon as possible.  

Regards, Jeff

[monaco__joe]

This change fixes memory issues in my tests.

The driver version is either 15.​33.​18.​3496 (32 bit) or 15.​33.​18.​64.​3496 (64 bit). Maybe that's what 15.33.0.3496 means but it wasn't clear to me.

[Anthony_P_Intel]

Hi Joe,

The 'Beta' driver was repackaged / renamed to 15.33.18.3496 now that it is no longer considered "Beta", and is considered an official / recommended release.  It is same driver, other than the name and intended purpose.