EMGD . memory leak and streaming issue

[TScar1]

Hi,

I am working on ATOM z530 +us15W chipset with Emgd enabled.I am succesfully ported EMGD 1.6 with debian and working well for dual sdvo and lvds display. when i tried to play some local video files using gstreamer its succefully plays..Then i have integrated splitted desktop vaapi library and trying to stream 720 p and 1080p. During this i have following issues. can anyone please hellp me for solving the problems

1. streaming 720p hangs after 5-10 mins .

2. streaming 1080p hangs before 1 mins

3.If streaming hangs and restarted using "ctrl+c" i am seeing a memory leak of <40Mb virtual memory(vmalloc).

4. when i am trying local playback and closing using "ctl+c" inbetween i can recreate issue 3. This will happen only once in 10 times or so.

I have one more question regarding memory allocation. is it possible to use static virtual memory instead of dynamic memory allocation every time? if possible how?

Note:I am using vaapi from spliieted desktop and emgd1.6 with debian.

Hope i will get a support.I am in a deliverble stage of my project and i am having really bad time with issue.Please help me ..

Regards and Thanks

Thomas Scariah

[Kirk_B_Intel]

Thomas,

Does anyone at your company have an Intel Premier Support Account (we call it Quad internally)? The is the best way to get support from us. The EDC forum is a "best effort" type system.

Is the reason you are using such an older version of EMGD because of support for an older kernel/X version? Is there a way to try at least 1.8 version now shipping?

Splitted Desktop sometimes provides VA API libraries beyond the recommneded ones for a particular driver. I am thinking that for 1.6 you may need to use a version of VA API earlier (like .29) than the latest one (.31 or later) but the release notes with the EMGD driver should talk about it.

If the VA API is crashing, it would not surprise me that memory allocations get left open- 40MB sounds like decode buffers that would not be closed properly if the driver panics. After a crash like that, a reboot is always a good idea.

What specific feature are you trying to use to "stream". There are some interfaces that are not supported and if you are trying to use those, it might explain your issues (long shot as they should fail immediately). I';ve been very successful taking video from ethernet from an E600 encoding video to another E600 decoding it to the screen. Here is the command line we use to do this:

TX to other

gst-launch v4l2src ! videorate ! ffmpegcolorspace ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! MixVideoEncoder encode-format=video/mpeg profile=2 rate-control=CBR bit-rate=15000000 need-display=0 ! queue ! tcpclientsink host=192.168.0.108 port=5000 protocol=1

TX to Self

gst-launch v4l2src ! videorate ! ffmpegcolorspace ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! MixVideoEncoder encode-format=video/mpeg profile=2 rate-control=CBR bit-rate=15000000 need-display=0 ! queue ! tcpclientsink host=192.168.0.109 port=5000 protocol=1

Receive from other

gst-launch tcpserversrc host=192.168.0.109 port=5000 protocol=1 ! MixVideoDecoder ! MixVideoSink

Obviously your IP Addresses and port addresses would be different- we hard code them in a stand-alone demo fashion...

Hope this helps.

[TScar1]

Hi Kirk/All,

Thanks for your immediate reply. I am attaching error log from emgd with debug enabled..

I am using ubuntu natty with emgd 1.6 and libva0.31 ,libav 0.6.2 & vaapi 0.2.5

The sender is encoding video to h264 format(DM6467) and streaming.The receiver is decoding the frames using US15W hardware decoder

I have tested the stream by decoding/playing using a standard PC+windows+vlc player and standard PC+ubuntu+gstreamer+ffdec_h264, and i am able to see the video and the setup is working for more than 6 hours On the Intel ATOM+ubuntu+gstremer+vaapi i am able to play the video but the system crashes after sometime(after 20min) with the below message

[h264 @ 0x99d0c00]number of reference frames exceeds max (probably corrupt input), discarding one

Last message repeated 16 times

[h264 @ 0x99d0c00]get_buffer() failed (-1 1 2 (nil))

[h264 @ 0x99d0c00]decode_slice_header error

[h264 @ 0x99d0c00]mmco: unref short failure

if i restart gstreamer pilepline(without rebooting system) it works for some time and again crashes i also displays the message

Since the Intel ATOM+ubuntu+gstremer+vaapi system is working for some thim it cannot be encoder-decoder incompatiability can you tell where cound be the problem and where should i check for the solution

command on Intel ATOM+ubuntu+gstremer+vaapi for playing the stream gst-launch-0.10 gstrtpbin udpsrc port=5434 ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, profile-level-id=(string)428028, sprop-parameter-sets=(string)\"Z0KAKPQCgC2IAA\\=\\=\\,aN48gAA\\=\", payload=(int)96" ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtph264depay ! "video/x-h264, framerate=(fraction)25/1, width=(int)1280, height=(int)720" ! vaapidecode ! vaapisink sync=false async=false

Regards and Thanks

Thomas Scariah

[Kirk_B_Intel]

OK, that makes a bit more sense. What is happening is that the decode is running out of memory in the middle of the decode process. When that happens, Linux bails out on the driver. You could likely correct this by making sure the graphics aperature setting in your bios sets it to the maximum (or at least 256MB) to allow the driver to allocate enough memory to the graphics. You may also be running out of free runtime memory that may require you to add more ram to the system.

The out of memory failure is not handled very elegantly, but much of that is out of our control. Video decode, especially of high bit rate content done with less than efficient encoder is a very high memory use case.

Hope that helps.

[TScar1]

Hi kirk,

I have ported 1.8 EMGD and checked the same.The issue still remains.

My aperture size setting is 256MB and RAM size is 1GB.Instead of streaming i tried playback and closed using ctrl+c. Sometime when closed using ctrl+c i am getting below error and if that error happens memory will get leaked(>40MB).

[h264 @ 0xa104de0]number of reference frames exceeds max (probably corrupt input), discarding one 

Last message repeated 19 times[h264 @ 0xa104de0]get_buffer() failed (-1 1 2 (nil))[h264 @ 0xa104de0]decode_slice_header error

Please find the log where i have done the testing. Actually i had a privilage user access but i lost the password and i am not able to recover it .

Below is the playback test I have done and teh memory info FYI..please note this is a playback test only ..

root@usbharddisk-lht:~# cat /proc/meminfo

MemTotal: 764316 kBMemFree: 666224 kBBuffers: 4712 kBCached: 28888 kBSwapCached: 0 kBActive: 19172 kBInactive: 24640 kBActive(anon): 10308 kBInactive(anon): 28 kBActive(file): 8864 kBInactive(file): 24612 kBUnevictable: 0 kBMlocked: 0 kBSwapTotal: 0 kBSwapFree: 0 kBDirty: 144 kBWriteback: 0 kBAnonPages: 10164 kBMapped: 31032 kBShmem: 104 kBSlab: 8868 kBSReclaimable: 1956 kBSUnreclaim: 6912 kBKernelStack: 600 kBPageTables: 568 kBNFS_Unstable: 0 kBBounce: 0 kBWritebackTmp: 0 kBCommitLimit: 382156 kBCommitted_AS: 134776 kBVmallocTotal: 262144 kBVmallocUsed: 18936 kBVmallocChunk: 239660 kBDirectMap4k: 40020 kBDirectMap4M: 737280 kBroot@usbharddisk-lht:/home/release/test_app# gst-launch-0.10 filesrc location=./Jeans.mp4 ! qtdemux name=demux demux.video_00 ! queue ! vaapidecode ! vaapisink demux.audio_00 ! queue ! faad ! alsasink device=plug:hdmilibva: libva version 0.32.0libva: va_getDriverName() returns 0libva: Trying to open /usr/lib/dri/emgd_drv_video.soIntel(R) Embedded Media and Graphics Driver 1.8 Build 2032Using XCB based dispatch table.libva: va_openDriver() returns 0Setting pipeline to PAUSED ...libva: libva version 0.32.0libva: va_getDriverName() returns 0libva: Trying to open /usr/lib/dri/emgd_drv_video.soIntel(R) Embedded Media and Graphics Driver 1.8 Build 2032Using XCB based dispatch table.libva: va_openDriver() returns 0Pipeline is PREROLLING ...Pipeline is PREROLLED ...Setting pipeline to PLAYING ...New clock: GstAudioSinkClock^CCaught interrupt -- handling interrupt.Interrupt: Stopping pipeline ...Execution ended after 1752461761 ns.Setting pipeline to PAUSED ...Setting pipeline to READY ...[h264 @ 0x95aee30]number of reference frames exceeds max (probably corrupt input), discarding one Last message repeated 19 times[h264 @ 0x95aee30]get_buffer() failed (-1 1 2 (nil))[h264 @ 0x95aee30]decode_slice_header error[h264 @ 0x95aee30]mmco: unref short failureSetting pipeline to NULL ...Freeing pipeline ...

root@usbharddisk-lht:/home/release/test_app# cat /proc/meminfo

VmallocTotal: 262144 kBVmallocUsed: 71704 kBVmallocChunk: 181512 kBroot@usbharddisk-lht:/home/release/test_app# gst-launch-0.10 filesrc location=./Jeans.mp4 ! qtdemux name=demux demux.video_00 ! queue ! vaapidecode ! vaapisink demux.audio_00 ! queue ! faad ! alsasink device=plug:hdmilibva: libva version 0.32.0libva: va_getDriverName() returns 0libva: Trying to open /usr/lib/dri/emgd_drv_video.soIntel(R) Embedded Media and Graphics Driver 1.8 Build 2032Using XCB based dispatch table.libva: va_openDriver() returns 0Setting pipeline to PAUSED ...libva: libva version 0.32.0libva: va_getDriverName() returns 0libva: Trying to open /usr/lib/dri/emgd_drv_video.soIntel(R) Embedded Media and Graphics Driver 1.8 Build 2032Using XCB based dispatch table.libva: va_openDriver() returns 0Pipeline is PREROLLING ...Pipeline is PREROLLED ...Setting pipeline to PLAYING ...New clock: GstAudioSinkClock^CCaught interrupt -- handling interrupt.Interrupt: Stopping pipeline ...Execution ended after 751977361 ns.Setting pipeline to PAUSED ...Setting pipeline to READY ...[h264 @ 0xa104de0]number of reference frames exceeds max (probably corrupt input), discarding one Last message repeated 19 times[h264 @ 0xa104de0]get_buffer() failed (-1 1 2 (nil))[h264 @ 0xa104de0]decode_slice_header error[h264 @ 0xa104de0]mmco: unref short failureSetting pipeline to NULL ...Freeing pipeline ...root@usbharddisk-lht:/home/release/test_app# cat /proc/meminfoMemTotal: 764316 kBMemFree: 545680 kBBuffers: 5556 kBCached: 43692 kBSwapCached: 0 kBActive: 27664 kB

XXXWHITESPACEXX...

[TScar1]

Hi Krik,

I am happy to say that issue is been solved by increasing the "buffer-size" property of "udp-src" element in gstreamer ..So I am able to stream 1080p and 720p smoothly . But still, if process is not closed smoothly we are ending up in memory leak .. I don't know how to solve this .. If you can help this it's very helpful...

Regards

Thomas Scariah

[Kirk_B_Intel]

That is excellent news. So basically, Gstreamer was overflowing one of its buffers, causing a panic, and making the kernal shut it down. Video decode takes a LOT of memory to do HD at high framerates as it is a lot of data.

My guess is that the error then memory leak is because Gstreamer has had us allocate an internal buffer for the decode when it blows up and we do not know they do not need it anymore. I wish there was a way to detect if a framework above the driver could know that it has failed, but unfortunately there is no Linux or Gstreamer mechnaism to do that. Maybe the open source community should lokk at the Gstreamer framework and beef up its overflow detection to prevent the failure from occring?

We cannot just close out a buffer unless we are explicitly told to do so, thus the "memory leak" in this case. From the driver perspective, it really is not a leak- it is a valid allocated video buffer that is open because we were told to open it, and nver told to close it. I believe this is actually one of those times a Linux reboot is required to recover fully from such a situation.

Hope that helps. Kirk

[GMcCo1]

I think I'm seeing the same memory leak.

On my system with EMGD 1.10 I've verified that gmm_unmap isn't being called for the following mapping when the X server is killed:

pvr/services4/3rdparty/emgd_displayclass/emgd_dc.c line 2210.

It appears that it is supposed to be unmapped on line 1978 of the same file but emgddc_free_a_devinfo doesn't get called.

[Kirk_B_Intel]

Yes, if you are doing video decode, and kill X in the middle of the decode, then start X again and repeat, you WILL see a memory decrease becuase of a "leak". The issue is out of our hands (and not really a true leak) because of the way video was architected by theLinux community as an add-on- it is sort of a feature right now. The kernel does not know it needs to clean-up video memory when shutting down X and the video stuff has no way to know about its memory once X is killed and restarted.

I hate to say it, but you either need to:

-WAIT for decode to finish BEFORE killing X, or

-You DO need to reboot to recover memory if you shutdown X in the middle of a decode (and likely encode also). We have found no way to prevent this issue from happening because of the way Linux works currently, or

-Don't kill X, it should not be necessary EVER during normal operation of a system. If someone is using that use case as a test, It is a silly one, because it does not reflect a normal use case.

Hope this helps.

[GMcCo1]

I'm not doing video decoding and I'm still seeing a massive leak. Just changing the runlevel back and forth between 5 and 3 will leak my system to death on Timesys Fedora Remix. I've made a patch that fixes part of the problem, I'm working on the rest now. I'm really amazed at the poor quality of this driver. Some of the comments in the kernel portion of the code are ridiculous and give me no confidence in this driver or anything to do with it (it reflects VERY badly on Intel's corporate image). FFS do you even try starting and stopping your X server to see if your kernel driver leaks memory !?! There is even a free in the code that is commented out and without it the kernel module leaks!

[GMcCo1]

Many hours and liters of diet pepsi later I've made a patch that fixes the leaks I've been referring to. I haven't finished because there might still be more leaks but at least I've found several HUGE ones. Feedback is appreciated as I've created this patch hastily with nearly no prior knowledge of the workings of this driver. I still believe there is some kind of emgd shutdown race condition plauging my production systems. It should be easier to track down now that the driver isn't leaking like a sieve.

Patch written and tested on NANO-8045 w/ Timesys Fedora Remix 14 and EMGD 1.10.0 Build 2209.

[LHale3]

it's very help full. if you like Diet Lite Please check it.

Thank