GStreamer Media SDK plugins now available in 01.org! (now with complete Windows 10 support)

[Ishmael_Sameen]

Hi All,

We have developed a set of GStreamer plugins that uses Intel Media SDK on Linux, and we'd like to share out our work here in the forums for those who might find this useful for their projects. This project was done for a major customer targeting the Apollo Lake platform, but we made sure that these GStreamer Media SDK plugins work on Haswell, Broadwell and Skylake platforms as well. The plugins can perform  hardware-accelerated decode, encode, VPP, transcode, rendering with zero-copy, etc. and works for all video formats currently supported by Intel Media Server Studio 2016 / 2017 Community, Essential or Professional Editions.

The validated stable versions can be cloned from Intel's official 01.org repository below:

https://github.com/01org/gstreamer-media-SDK

Windows support is completed, with full D3D11 hardware acceleration including zero-copy feature and deep color support when rendering with mfxsink. Please feel free to test it out from the below repos and report any issues you may find.

https://github.com/ishmael1985/gstreamer-media-SDK
https://github.com/ph0b/gstreamer-media-SDK

Any contributions to these repos are most welcome :)

Since I am no longer associated with Intel, please contact me via my personal email below strictly for GST-MSDK plugin issues / matters. As for other MSDK matters, it's best to contact Intel MSDK team since I do not represent Intel in official capacity.

ishmael1985@gmail.com

Regards,
Ishmael Sameen

[benjamin_d_3 · Accepted solution]

Hi Ishmael,

I just want to confirm that all issues that I have reported have been solved, the memory has been the same for 4 hours, we also have implemented the plugin to work with our internal application you can see the time and memory usage on the screenshot in the attachment. 

Thanks again for the help

Best Regards
Ben

[benjamin_d_3]

hi Ishmael, 

I was able to test the changes you made from the repo and now the video is working good. but we noticed 1 other issue with Audio it doesn't sync with Video.

I tried 2 same pipelines one wtih mfx264 and one with x264 and the issue is only MFX.

here are the pipelines i tried:

# mfxh264dec and mfxh264enc - Audio out of sync
gst-launch-1.0 -v --gst-debug=2 souphttpsrc location="http://127.0.0.1:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \ 
h264parse ! mfxh264dec ! videoconvert ! mfxvpp width=1280 height=720 ! mfxh264enc rate-control=1 bitrate=1700 ! "video/x-h264,stream-format=byte-stream,profile=high" ! h264parse ! \ 
flvmux name=mux ! rtmpsink location="rtmp://127.0.0.1:1935/pushrtmp/mfxtest1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \ 
mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.

# avdec_h264 and x264enc - Audio is synced fine 
gst-launch-1.0 -v --gst-debug=2 souphttpsrc location="http://127.0.0.1:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \
h264parse ! avdec_h264 ! videoconvert ! x264enc pass=0 bitrate=1700 ! "video/x-h264,stream-format=byte-stream,profile=high" ! h264parse ! \
flvmux name=mux ! rtmpsink location="rtmp://127.0.0.1:1935/pushrtmp/mfxtest1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \
mpegaudioparse ! queue ! avdec_mp2float ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.

also please find logs in attachment

Regards
Ben 

[Ishmael_Sameen]

Hi Benjamin,

I've committed a new proposed patch to my repo that may solve your audio sync issue. Please pull the latest source from my repo, and please let me know  what you get.

If it doesn't fix the issue, can you try using avdec_h264 with mfxh264enc? Also, I want to understand how does it go out of sync, is it gradual or constant, or the audio drops every now and then?

I believe some parts of your pipeline can be optimized, when muxing to flv using the MFX plugin you don't need to specify the type of H264 bitstream caps and you don't need the h264parse prior to flvmux, the conversion to avc is done automatically. By default, mfxh264enc will use High profile, unless you explicitly specify the profile in the H264 bitstream caps.

Hopefully, all is finally good with your MFX pipeline. Do keep me posted on your results.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, 

I updated gstreamer-media-SDK from your repo, then I ran the same pipeline and the issues with audio sync has been fixed :) 
now everything seems to be working perfectly.
Now that all issues have been fixed this plugin seems to be performing a lot better then VAAPI based on many test we did we are getting a much better video quality with gstreamer-media-SDK. 

again thanks for your help and great work.

Best Regards
Ben

[Ishmael_Sameen]

Hi Benjamin,

Glad to hear your issues are resolved :)

If there are any issues you encounter please let us know. We'll do our best to iron out whatever issues you may find.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, after farther testing we noticed that there is memory leak, as the Process starts with 70 MB of ran then in 30 mins goes up to 800MB. To make sure narrow down the problem i also tried avdec264 and x264 i dont get any memory leak.  

[benjamin_d_3]

Here is another post forgot to add the logs for memory leak issue.

regards
Ben

[Ishmael_Sameen]

Hi Ben,

Let's isolate which plugin may likely cause the memory leak. Can you try avdec_h264 instead of mfxh264dec? Then try mfxh264dec with x264enc. You can put mfxvpp then later on, if they check out.

I'm running a similar pipeline as well but instead using a live H264 camera (Logitech C920). Haven't noticed anything funny yet, personally I am surprised you got the memory leak, I have been running week-long simultaneous preview while encoding with the plugins. But let's see what may possibly cause your memory leak.

Also, please don't forget to pull the latest source from my personal repo.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, we updated gstreamer-media-SDK and ran many tests and found out memory leak happens only when using mfxh264dec. 
We then tried avdec_h264 with mfxh264enc and there was no more leaks for 2 hours memory was the same.

[Ishmael_Sameen]

Hi Ben,

Thanks for identifying the problematic plugin. I've pushed a new commit in my personal repo that may possibly resolve the memory leak issue. Please let me know how it works out for you.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael,

I updated and tried I see that it got a little better in memory usage but there still leakage, i forgot to mention in previous posts that it is not just memory that increase by time it's the CPU as well.

I made 2 screenshots from top cmd, one when i first started and 2nd after an hour. 

At first the memory was 60MB 20%CPU then an hour later it got to Memory 170MB and CPU around 60% 

find screenshots in the attachments:

Best Regards 
Ben

[Ishmael_Sameen]

Hi Ben,

I have committed another proposed patch to my personal repo to address the last memory leak issue you have reported.

If it doesn't work out, please try mfxh264dec live-mode=true and check if you get any memory leaks.

Personally I have not been able to reproduce your issue, but based on the log you have provided, I assume that I haven't cleaned up properly if MFX_ERR_MORE_DATA is regularly reported during live streaming after the first frames have been decoded, and that I have to somehow simulate a live stream setup that can create this scenario. If this doesn't yet solve your issue, I'll try to reproduce the reported bug under simulated conditions to better debug and resolve the issue. Such an alarming memory leak didn't happen in our validation labs, and we do extensive video testing on  the decoder for many hours without running into such issues. Thank you very much for finding this bug.

In any case, please let me know what you get.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmel, we updated the plugin but we are still getting memory leak when using mfxh264dec,

the pipeline we tried:

gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec live-mode=true ! video/x-raw,framerate=25/1 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

I have sent you a PM with some more info to troubleshoot easier,

Also I captured more logs you may find them in the attachment.

best reagrds
Ben

[Ishmael_Sameen]

Hi Ben,

Thank you very much for your help in helping to resolve the issue. I've pushed a new commit to my personal repo, please let me know your findings. So far, the memory leak on my side looks fixed, but I need your confirmation.

Also, you can use the following pipeline after you test your initial pipeline:

gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec ! mfxvpp ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

This will ensure that video memory is used by the MFX plugins in the pipeline.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, 

I updated MFX from your repo and tried the pipeline you suggested but the memory leak is still happening.
I also tried to run the pipeline with memory leak tracer "GST_DEBUG=GST_TRACER:7 GST_TRACERS=leaks" to see if this can help you pinpoint the leak.

​#pipeline
GST_DEBUG=GST_TRACER:7 GST_TRACERS=leaks gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec ! mfxvpp ! mfxh264enc rate-contrrol=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

It looks like gstreamer is detecting the leak, find the tracer logs in the attachment.
 

[Ishmael_Sameen]

Hi Ben,

I pushed a new commit to my personal repo that may finally resolve the memory leak issue. The logs and the dumpstream proved useful.

Please let me know your results. If it's still there, I'll have to do some really in-depth debugging for this issue. I would also like to know the overall memory and CPU usage if the leak is not resolved.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael,

Your last commits improved the memory now i think it's better but there is still a low memory leak for example when i first started it was 55MB then it went to 98MB an hour later please see below i copied from terminal.

AT START: RSS 55464 KB  0:06.57 gst-launch-1.0
END:         RSS 98800 KB  6:15.53 gst-launch-1.0

I attached another logs that's showing some traces I see there is less memory tracing lines now.

regards
Ben

[Ishmael_Sameen]

Hi Ben,

I pushed a new commit to my personal repo which should complete the memory leak fix. Please let me know if the fix resolves the memory leak issue in your system.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, 

I did many tests with different pipelines it looks like memory leak has been solved for the most part, as of right now memory leak happens only if framerate is used in mfxvpp.

find pipeline exmaples below:

# no memory leak with this pipeline
gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec live-mode=true ! mfxvpp width=1280 height=720 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

# this pipeline with framerate=25 gives memory leak
gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec live-mode=true ! mfxvpp width=1280 height=720 framerate=25 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

Thank You Ishmael

best regards 
Ben

[Ishmael_Sameen]

Hi Ben,

I pushed a new commit to my personal repo that should resolve the memory leak issue when setting the framerate property in mfxvpp. Also I changed the framerate property as a fraction type, please take note of that. It's because it's abit easier to check if the set framerate property value is the same as the input frame rate at the sinkpad caps if framerate is a fraction type; and it's kind of a rule in GStreamer plugins to set frame rate as a fraction instead of a double.

Please let me know if the new commit has resolved the frame rate memory leak issue in your side.

Regards,

Ishmael

[benjamin_d_3 · Accepted solution]

Hi Ishmael,

I just want to confirm that all issues that I have reported have been solved, the memory has been the same for 4 hours, we also have implemented the plugin to work with our internal application you can see the time and memory usage on the screenshot in the attachment. 

Thanks again for the help

Best Regards
Ben

[benjamin_d_3]

Ishmael, 

I also noticed a warning that shows constantly when running in debug level 2 or higher, not sure if it has do anything with performance but thought i would share with you. Please see the logs in the attachment for more details.

Best regards,
Ben