GStreamer Media SDK plugins now available in 01.org! (now with complete Windows 10 support)

[Ishmael_Sameen]

Hi All,

We have developed a set of GStreamer plugins that uses Intel Media SDK on Linux, and we'd like to share out our work here in the forums for those who might find this useful for their projects. This project was done for a major customer targeting the Apollo Lake platform, but we made sure that these GStreamer Media SDK plugins work on Haswell, Broadwell and Skylake platforms as well. The plugins can perform  hardware-accelerated decode, encode, VPP, transcode, rendering with zero-copy, etc. and works for all video formats currently supported by Intel Media Server Studio 2016 / 2017 Community, Essential or Professional Editions.

The validated stable versions can be cloned from Intel's official 01.org repository below:

https://github.com/01org/gstreamer-media-SDK

Windows support is completed, with full D3D11 hardware acceleration including zero-copy feature and deep color support when rendering with mfxsink. Please feel free to test it out from the below repos and report any issues you may find.

https://github.com/ishmael1985/gstreamer-media-SDK
https://github.com/ph0b/gstreamer-media-SDK

Any contributions to these repos are most welcome :)

Since I am no longer associated with Intel, please contact me via my personal email below strictly for GST-MSDK plugin issues / matters. As for other MSDK matters, it's best to contact Intel MSDK team since I do not represent Intel in official capacity.

ishmael1985@gmail.com

Regards,
Ishmael Sameen

[benjamin_d_3 · Accepted solution]

Hi Ishmael,

I just want to confirm that all issues that I have reported have been solved, the memory has been the same for 4 hours, we also have implemented the plugin to work with our internal application you can see the time and memory usage on the screenshot in the attachment. 

Thanks again for the help

Best Regards
Ben

[Ishmael_Sameen]

Hi Ben,

I'm glad the plugins finally fully work out for you. And I thank you very much for pointing out all the bugs we haven't found, especially those memory leaks. You kind of ruined my weekend, but it was great fun and satisfaction fixing those bugs :) After all, they have to be fixed, either you or the customers will find them and we will have to resolve them.

The warning message is nothing to be concerned about, it won't affect the performance of the plugins, but it does pique my curiosity why it happens. I'm pretty sure the plugins performance is close to the reference sample apps, based on our internal testing in almost all use case scenarios (around 5-10% faster or slower). I will take a look on it when I have the time, but I will place this as a lower priority.

Meanwhile, you may have a few questions about the plugins, please feel free to ask. And don't forget to notify us if there are any other bugs you may find. For now, it's great to see that the GST-MSDK plugins are working out well for you.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, thank you so much and sorry i was bugging you during the weekend, as of right now we are very satisfied with performance of the plug in, now we started testing the plugin with Intel VCA cards and we are getting excellent results.

Thank again
Ben 

[jan_g_1]

hey,

Is gstreamer better than ffmpeg?

@benjamin which intel VCA do you use? You can tell a bit about how many streams with it at the same time and how many with the e3-1245 v5?

[benjamin_d_3]

hi Jan, 

with E3-1245 v5 we are doing 15 leaner transcodes in full hd 1080@30fps h264 and aac, as far VCA's, 3 cards in 1u server about 150 transcodes 1080@30fps which is a lot.. shocking i know :)

Best Regards
Ben

[benjamin_d_3]

Hi Ishmael, it's me again :)

We found another bug when transcoding live streams audio goes out of sync after 6 hours or more. 

Not sure how you can troubleshoot this but let me know if there is anything I can do to help from my side

Best Regards
Ben

[jan_g_1]

hey ben, 

thank you for your answer. this is really shocking me :-D nice. you know how many streams can a e3-1585L v5 transcodes? intel VCA are very expensive or? 2500? 

now i use a i7-4790k and this cpu transcodes 11 hd streams at the same time with ffmpeg and qsv. you mean that gstreamer works better? or why do you use gstreamer and not ffmpeg?

[benjamin_d_3]

Hi Jan, yes the price for the VCA is around there. 
We been testing E3-1585L for a 2 weeks now we were able to benchmark up to 22 linear transcodes in HD. 1080@30 h264 aac 2.5mbits

As far gstreamer vs ffmpeg, depends what  you are trying to do, gstreamer could be used as library that you can develop your app in top of it and ffmpeg is just a standalone app. Look for more info online to see what suits you best

Best Regards
Ben
 

[Mantas_D_]

benjamin dreshaj wrote:

Hi Ishmael, it's me again :)

We found another bug when transcoding live streams audio goes out of sync after 6 hours or more. 

Not sure how you can troubleshoot this but let me know if there is anything I can do to help from my side

Best Regards
Ben

 It is easy to reproduce, You have to make input live stream bad during transcoding, and it will go out of sync after that. I have the same issue allso.

[Ishmael_Sameen]

Hi All,

Looks like I can't just catch a break :)

Benjamin, Mantas,

Thanks for the findings. Based on what Mantas found, it looks like a robustness feature at the decoder may be possibly causing the sync issue. When a majorly corrupted frame is detected by the MFX decoder plugin, it skips the frame (and most likely the following few frames) and resets the decoder, until it finds a good keyframe to resume decoding. I believe that at some point of the transcode, some corrupted frames were detected and skipped, but the audio transcoding remains smooth, thus causing the gradual out-of-sync issue.

The fix, which was turned into an option in the decoder and is set to false by default, was actually committed yesterday to my personal repo, please get the latest from there and let me know whether the fix worked or not.

Jan,

We really haven't tried out and benchmarked FFMPEG QSV, so we are in no position to judge whether FFMPEG QSV is better than GStreamer MSDK or not. However, I can very confidently say that the GStreamer MSDK plugins I primarily developed is as fast as it can get (dare I say the fastest open-source broad market solution?), taking advantage of most memory and threading optimizations that the MSDK API allows. The GStreamer  MSDK will use video memory when it can, and it employs task sharing to optimize the usage of the thread pool between tasks. The performance optimizations as well as the features provided by the plugins should enable most users to accomplish the video processing tasks that would taken a lot of development effort purely using the MSDK API.

Regards,

Ishmael

[Mantas_D_]

 Hello Ishmael,

i have tried with skip-corrupted-frames=true , but still the same desync happens, video stays behind audio. This is how i do it: i stop multicast source for a second during live transcoding, and start it again. It does happen with mpeg2 source as input, as well as with h264. If a reproduce the same  situation with avdec_h264 and x264enc, video and audio stays in sync.

Best regards,

Mantas

[benjamin_d_3]

Hello Ishmael and Mantas,

I can confirm the same issue in my end as well, in fact I tried 2 different pipelines with skip-corrupted-frames=true and false but still audio went out of sync on both. My sources are http mpegts in my case I waited a couple of hours and I seen the out of sync issues.
I guess Mantas was right if there is a dropped frame that's when audio goes out of sync.

Regards
Ben

[Ishmael_Sameen]

Hi Mantas, Benjamin,

I've pushed a new commit to my personal repo that may resolve the sync-ing issues based on the assumption that dropped frames causes the out-of-sync issue. I suggest to set skip-corrupted-frames=false for now, but I'd like to also know if the fix also works when skip-corrupted-frames=true as well. Please let me know your results.

Regards,

Ishmael

[Mantas_D_]

 Hello Ishmael, Ben,

It has a slight desync from the beggining (500ms or more), but looks like glitches in stream does not make it worse. I will try to test it further tommorow.

Regards,

Mantas

[benjamin_d_3]

Hi Guys, I tested last commit with 2 pipelines one of them with skip-corrupted-frames=true and the other one with skip-corrupted-frames=false.

with false: the video was a head of audio around 500ms I can confirm I got the same results as Mantas, which is a lot better then before but still out of sync.

with true:  the video was a head of audio a lot more around almost 2 seconds.

best regards
Ben

[Ishmael_Sameen]

Hi Ben, Mantas,

Thank you for your hard work in reporting the results of the fix. In lieu of your findings I have just pushed a new commit to my repo to further improve the synchronization. Please get the latest from my repo and let me know what you get.

Regards,

Ishmael

[Mantas_D_]

 Hi all,

 It still has issues, if i use skip-corrupted-frames=true , emit-stats=true on tsdemux, and rate-control=1 on the encoder, the desync is there, bet verry small, in fact one stream worked overnight and in the morning it was allmos in perfect sync. But today i tryed other streams, and if i do not use emit stats or skip corrupted frames, i am getting bad desync from start. Hope Ben has some more insights.

Regards,

Mantas

[Ishmael_Sameen]

Hi Mantas,

I kind of anticipated that you'd find a problem (actually I got it via another way), so I updated the codebase on my repo that should fix up your issues. Please try out the latest from there and see how it works out for you. I would hope that all of your live test streams would be in sync when transcoded, but we'll find out. The only problem really here is that whether audio is sync-ed with video from the start, I believe that's not so easy to determine, perhaps that's what the streamsynchronizer element is meant for.

Hi Ben,

Please update to the latest from my repo, it may address the issues that Mantas has, and also fixes up the annoying "Can't copy metadata" error that you have reported previously. But please do take note of the memory usage, by fixing up the error message, there's a risk of memory leak, though I can't test long enough to determine it is so. So far it looked good to me with an hour-long test.

I really appreciate the thoroughness with which you guys test the plugins, please do keep those bug reports going!

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael,

I left the transcoding on overnight with your previous commit and the out of sync was just a little bit same as what Mantas reported, now i just updated it again from the repo and i will let you know in few hours if the issue has been solved for good.

Thank you guys again.
 

best regards
Ben

[Mantas_D_]

 Hello, its me again :)

  Did some more testing. This time i used one h264 hd stream and one mpeg2 sd, both comming in via http. Mpeg4 source was stable all night and the sync was perfect, mpeg2 source had many continuity counter errors and still in the morning it was in sync. Only when i restarted the input stream it went out of sync the same moment.

There is allso another observation, since this commit 8f606d5c259e4891b2257513794b15eda9e69274 , only http input streams work good, if i input same stream via udp, it is stuttering and goes in and out of sync from the beginning of transcode.

Regards,

Mantas

[benjamin_d_3]

Hi,

I had it running all night and today morning I checked it's still in sync my source is http and output rtmp . So it seems that issue has been fixed at least for my tests.  

I didn't get to try UDP though, Mantas can you show us your pipeline ? so i can check on my end as well to see if I can duplicate.

best regards 
Ben 

[Mantas_D_]

 Hello,

 this is the pipline i use :

gst-launch-1.0 -v udpsrc uri=udp://@233.0.1.108:1234 buffer-size=2097152 ! tsdemux name=demux demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! mfxh264dec ! mfxvpp ! mfxh264enc rate-control=1 bitrate=1500 ! mux. demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! mpegaudioparse ! mux. mpegtsmux name=mux ! hlssink

The same pipeline but with http input produces ok stream. Older versions before this 8f606d5c259e4891b2257513794b15eda9e69274 worked ok with udp. By the way, the http and udp stream is the same stream, so its quite odd.

Update: getting mpegtsmux mpegtsmux.c:1159:mpegtsmux_clip_inc_running_time:<mux:sink_65> ignoring DTS going backward with latest versions, this i dont see on old versions.

Mantas