Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

GStreamer Media SDK plugins now available in 01.org! (now with complete Windows 10 support)

Ishmael_Sameen
Beginner
13,523 Views

Hi All,

We have developed a set of GStreamer plugins that uses Intel Media SDK on Linux, and we'd like to share out our work here in the forums for those who might find this useful for their projects. This project was done for a major customer targeting the Apollo Lake platform, but we made sure that these GStreamer Media SDK plugins work on Haswell, Broadwell and Skylake platforms as well. The plugins can perform  hardware-accelerated decode, encode, VPP, transcode, rendering with zero-copy, etc. and works for all video formats currently supported by Intel Media Server Studio 2016 / 2017 Community, Essential or Professional Editions.

The validated stable versions can be cloned from Intel's official 01.org repository below:

https://github.com/01org/gstreamer-media-SDK

Windows support is completed, with full D3D11 hardware acceleration including zero-copy feature and deep color support when rendering with mfxsink. Please feel free to test it out from the below repos and report any issues you may find.

https://github.com/ishmael1985/gstreamer-media-SDK
https://github.com/ph0b/gstreamer-media-SDK

Any contributions to these repos are most welcome :)

Since I am no longer associated with Intel, please contact me via my personal email below strictly for GST-MSDK plugin issues / matters. As for other MSDK matters, it's best to contact Intel MSDK team since I do not represent Intel in official capacity.

ishmael1985@gmail.com

Regards,
Ishmael Sameen

0 Kudos
2 Solutions
benjamin_d_3
New Contributor I
13,245 Views

Hi Ishmael,

I just want to confirm that all issues that I have reported have been solved, the memory has been the same for 4 hours, we also have implemented the plugin to work with our internal application you can see the time and memory usage on the screenshot in the attachment. 

Thanks again for the help

Best Regards
Ben

 

  

View solution in original post

0 Kudos
benjamin_d_3
New Contributor I
13,299 Views

Hi Ishmael & Mantas,

First I want to thank Ishmael for the hard work and making this great plugin, we already started using it in production and dropped VAAPI due to quality and bug issues. I don't know how you did this but it's amazing for what I can see you are the only one developing this and solved every bug we found, thanks to Mantas as well for helping us find bugs.

Back to my last test results: I did run a pipeline with udpsrc and audio is good, i did make the source mess up by making the bandwidth lower then what source was in my switch to see if it goes out of sync then after it went back in sync again, I wasnt able to reproduce Mantas issue, maybe last fix solved it. 

I tried running other sources as well like rtmp hls tcp they all worked fine.

As far as memory that seems to be good too i have a process running for 21 hours at only 110MB. 

View solution in original post

0 Kudos
89 Replies
benjamin_d_3
New Contributor I
2,982 Views

hi Ishmael, 

I was able to test the changes you made from the repo and now the video is working good. but we noticed 1 other issue with Audio it doesn't sync with Video.

I tried 2 same pipelines one wtih mfx264 and one with x264 and the issue is only MFX.

here are the pipelines i tried:

# mfxh264dec and mfxh264enc - Audio out of sync
gst-launch-1.0 -v --gst-debug=2 souphttpsrc location="http://127.0.0.1:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \ 
h264parse ! mfxh264dec ! videoconvert ! mfxvpp width=1280 height=720 ! mfxh264enc rate-control=1 bitrate=1700 ! "video/x-h264,stream-format=byte-stream,profile=high" ! h264parse ! \ 
flvmux name=mux ! rtmpsink location="rtmp://127.0.0.1:1935/pushrtmp/mfxtest1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \ 
mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.
# avdec_h264 and x264enc - Audio is synced fine 
gst-launch-1.0 -v --gst-debug=2 souphttpsrc location="http://127.0.0.1:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \
h264parse ! avdec_h264 ! videoconvert ! x264enc pass=0 bitrate=1700 ! "video/x-h264,stream-format=byte-stream,profile=high" ! h264parse ! \
flvmux name=mux ! rtmpsink location="rtmp://127.0.0.1:1935/pushrtmp/mfxtest1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \
mpegaudioparse ! queue ! avdec_mp2float ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.

 

also please find logs in attachment

Regards
Ben 

 

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Benjamin,

I've committed a new proposed patch to my repo that may solve your audio sync issue. Please pull the latest source from my repo, and please let me know  what you get.

If it doesn't fix the issue, can you try using avdec_h264 with mfxh264enc? Also, I want to understand how does it go out of sync, is it gradual or constant, or the audio drops every now and then?

I believe some parts of your pipeline can be optimized, when muxing to flv using the MFX plugin you don't need to specify the type of H264 bitstream caps and you don't need the h264parse prior to flvmux, the conversion to avc is done automatically. By default, mfxh264enc will use High profile, unless you explicitly specify the profile in the H264 bitstream caps.

Hopefully, all is finally good with your MFX pipeline. Do keep me posted on your results.

 

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael, 

I updated gstreamer-media-SDK from your repo, then I ran the same pipeline and the issues with audio sync has been fixed :) 
now everything seems to be working perfectly.
Now that all issues have been fixed this plugin seems to be performing a lot better then VAAPI based on many test we did we are getting a much better video quality with 
gstreamer-media-SDK. 

again thanks for your help and great work.

Best Regards
Ben

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Benjamin,

Glad to hear your issues are resolved :)

If there are any issues you encounter please let us know. We'll do our best to iron out whatever issues you may find.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael, after farther testing we noticed that there is memory leak, as the Process starts with 70 MB of ran then in 30 mins goes up to 800MB. To make sure narrow down the problem i also tried avdec264 and x264 i dont get any memory leak.  

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Here is another post forgot to add the logs for memory leak issue.

regards
Ben

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

Let's isolate which plugin may likely cause the memory leak. Can you try avdec_h264 instead of mfxh264dec? Then try mfxh264dec with x264enc. You can put mfxvpp then later on, if they check out.

I'm running a similar pipeline as well but instead using a live H264 camera (Logitech C920). Haven't noticed anything funny yet, personally I am surprised you got the memory leak, I have been running week-long simultaneous preview while encoding with the plugins. But let's see what may possibly cause your memory leak.

Also, please don't forget to pull the latest source from my personal repo.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael, we updated gstreamer-media-SDK and ran many tests and found out memory leak happens only when using mfxh264dec. 
We then tried avdec_h264 with mfxh264enc and there was no more leaks for 2 hours memory was the same.

 

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

Thanks for identifying the problematic plugin. I've pushed a new commit in my personal repo that may possibly resolve the memory leak issue. Please let me know how it works out for you.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael,

I updated and tried I see that it got a little better in memory usage but there still leakage, i forgot to mention in previous posts that it is not just memory that increase by time it's the CPU as well.

I made 2 screenshots from top cmd, one when i first started and 2nd after an hour. 

At first the memory was 60MB 20%CPU then an hour later it got to Memory 170MB and CPU around 60% 

find screenshots in the attachments:

Best Regards 
Ben

 

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

I have committed another proposed patch to my personal repo to address the last memory leak issue you have reported.

If it doesn't work out, please try mfxh264dec live-mode=true and check if you get any memory leaks.

Personally I have not been able to reproduce your issue, but based on the log you have provided, I assume that I haven't cleaned up properly if MFX_ERR_MORE_DATA is regularly reported during live streaming after the first frames have been decoded, and that I have to somehow simulate a live stream setup that can create this scenario. If this doesn't yet solve your issue, I'll try to reproduce the reported bug under simulated conditions to better debug and resolve the issue. Such an alarming memory leak didn't happen in our validation labs, and we do extensive video testing on  the decoder for many hours without running into such issues. Thank you very much for finding this bug.

In any case, please let me know what you get.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmel, we updated the plugin but we are still getting memory leak when using mfxh264dec,

the pipeline we tried:

gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec live-mode=true ! video/x-raw,framerate=25/1 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'


I have sent you a PM with some more info to troubleshoot easier,

Also I captured more logs you may find them in the attachment.

best reagrds
Ben

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

Thank you very much for your help in helping to resolve the issue. I've pushed a new commit to my personal repo, please let me know your findings. So far, the memory leak on my side looks fixed, but I need your confirmation.

Also, you can use the following pipeline after you test your initial pipeline:

gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec ! mfxvpp ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

This will ensure that video memory is used by the MFX plugins in the pipeline.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael, 

I updated MFX from your repo and tried the pipeline you suggested but the memory leak is still happening.
I also tried to run the pipeline with memory leak tracer "GST_DEBUG=GST_TRACER:7 GST_TRACERS=leaks" to see if this can help you pinpoint the leak.

 

​#pipeline
GST_DEBUG=GST_TRACER:7 GST_TRACERS=leaks gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec ! mfxvpp ! mfxh264enc rate-contrrol=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'



It looks like gstreamer is detecting the leak, find the tracer logs in the attachment.
 

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

I pushed a new commit to my personal repo that may finally resolve the memory leak issue. The logs and the dumpstream proved useful.

Please let me know your results. If it's still there, I'll have to do some really in-depth debugging for this issue. I would also like to know the overall memory and CPU usage if the leak is not resolved.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael,

Your last commits improved the memory now i think it's better but there is still a low memory leak for example when i first started it was 55MB then it went to 98MB an hour later please see below i copied from terminal.

AT START: RSS 55464 KB  0:06.57 gst-launch-1.0
END:         RSS 98800 KB  6:15.53 gst-launch-1.0

I attached another logs that's showing some traces I see there is less memory tracing lines now.

regards
Ben

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

I pushed a new commit to my personal repo which should complete the memory leak fix. Please let me know if the fix resolves the memory leak issue in your system.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
2,982 Views

Hi Ishmael, 

I did many tests with different pipelines it looks like memory leak has been solved for the most part, as of right now memory leak happens only if framerate is used in mfxvpp.

find pipeline exmaples below:

# no memory leak with this pipeline
gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec live-mode=true ! mfxvpp width=1280 height=720 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'

# this pipeline with framerate=25 gives memory leak
gst-launch-1.0 -v souphttpsrc location='http://127.0.0.1:80/oranews_HD/mpegts' ! tsdemux ! h264parse ! mfxh264dec live-mode=true ! mfxvpp width=1280 height=720 framerate=25 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true ! rtmpsink location='rtmp://127.0.0.1:1935/pushrtmp/mfdsxtest1asd'


Thank You Ishmael

best regards 
Ben

 

 

 

0 Kudos
Ishmael_Sameen
Beginner
2,982 Views

Hi Ben,

I pushed a new commit to my personal repo that should resolve the memory leak issue when setting the framerate property in mfxvpp. Also I changed the framerate property as a fraction type, please take note of that. It's because it's abit easier to check if the set framerate property value is the same as the input frame rate at the sinkpad caps if framerate is a fraction type; and it's kind of a rule in GStreamer plugins to set frame rate as a fraction instead of a double.

Please let me know if the new commit has resolved the frame rate memory leak issue in your side.

Regards,

Ishmael

0 Kudos
benjamin_d_3
New Contributor I
13,246 Views

Hi Ishmael,

I just want to confirm that all issues that I have reported have been solved, the memory has been the same for 4 hours, we also have implemented the plugin to work with our internal application you can see the time and memory usage on the screenshot in the attachment. 

Thanks again for the help

Best Regards
Ben

 

  

0 Kudos
benjamin_d_3
New Contributor I
3,038 Views


Ishmael, 

I also noticed a warning that shows constantly when running in debug level 2 or higher, not sure if it has do anything with performance but thought i would share with you. Please see the logs in the attachment for more details.

Best regards,
Ben

0 Kudos
Reply