GStreamer Media SDK plugins now available in 01.org! (now with complete Windows 10 support)

[Ishmael_Sameen]

Hi All,

We have developed a set of GStreamer plugins that uses Intel Media SDK on Linux, and we'd like to share out our work here in the forums for those who might find this useful for their projects. This project was done for a major customer targeting the Apollo Lake platform, but we made sure that these GStreamer Media SDK plugins work on Haswell, Broadwell and Skylake platforms as well. The plugins can perform  hardware-accelerated decode, encode, VPP, transcode, rendering with zero-copy, etc. and works for all video formats currently supported by Intel Media Server Studio 2016 / 2017 Community, Essential or Professional Editions.

The validated stable versions can be cloned from Intel's official 01.org repository below:

https://github.com/01org/gstreamer-media-SDK

Windows support is completed, with full D3D11 hardware acceleration including zero-copy feature and deep color support when rendering with mfxsink. Please feel free to test it out from the below repos and report any issues you may find.

https://github.com/ishmael1985/gstreamer-media-SDK
https://github.com/ph0b/gstreamer-media-SDK

Any contributions to these repos are most welcome :)

Since I am no longer associated with Intel, please contact me via my personal email below strictly for GST-MSDK plugin issues / matters. As for other MSDK matters, it's best to contact Intel MSDK team since I do not represent Intel in official capacity.

ishmael1985@gmail.com

Regards,
Ishmael Sameen

[benjamin_d_3 · Accepted solution]

Hi Ishmael,

I just want to confirm that all issues that I have reported have been solved, the memory has been the same for 4 hours, we also have implemented the plugin to work with our internal application you can see the time and memory usage on the screenshot in the attachment. 

Thanks again for the help

Best Regards
Ben

[benjamin_d_3]

Hi Mantas,

I started the the same pipeline as yours im going to wait few hours to see if it goes out of sync.

regards

Ben

[Ishmael_Sameen]

Hi Mantas,

I have updated my repo to address the issue where restarting the input makes your live stream out-of-sync. Please get the latest from there and let me know if the issue is fixed.

However, I haven't yet worked on the udp issues you're having. It seems that streams coming from udp sources give inconsistent timestamps causing the kind of stuttering you see. My fixes after commit id 8f606d5c essentially re-order and re-use the presentation timestamps deduced by the parser and passed downstream to the decoder and finally to the encoder, by doing so, the encoder faithfully follows the timestamps of incoming frames without having to calculate those timestamps. This way, there is no sync issue since all frames, dropped or corrupted or good, all have their own specific timestamps and the encoder does not have to worry about being out-of-sync with the decoder. Prior to the fixes, those timestamps were manually calculated, that's why udp and http streams were ok, but will have sync issues when frames are dropped.

It would help to provide a reasonably short dump of the udp stream with those inconsistent timestamps you're getting so that I can reproduce your issues and find a method to further increase the robustness of the streaming quality, which may involve intelligently deciding for each frame whether the correct valid timestamp is being used, or that the correct timestamp needs to calculated for it.

You could also use avdec_h264 instead of mfxh264dec in conjuction with mfxh264enc, just to see if my theory is right and that udp sources are streaming smoothly.

Hi Ben,

Good to know that all your issues are solved. You should also make sure the memory usage remains constant as well, please do get the latest from my repo to confirm that everything is working good for you.

I hope you guys should be pretty satisfied with the current state of the plugins, but yeah let me try to finish the job here. Please do keep in mind I'm doing this in my own free time, but I take great pride in my own  work, so rest assured that any bug you report, and is reproducible on my side, I will do what I can to address it.

Regards,

Ishmael

[Mantas_D_]

 Hello Ishmael,

Your efforts are very much appreciated and thanks for these plugins as there is in deed nothing like it freely available ! I have tryed to record a sample for You ant then tryed to encode it with the same pipe and gues what? it was fine! UDP with avdec_h264 was allso fine, but not with mfxh264dec. Then i have tryed to pass do-timestamp=false to updscr receiver of gstremer, and looks like it did the trick, mfxh264dec is allso fine using this setting. I will do more testing and report back. 

Allso,  the issue after restarting the stream is indeed fixed! Have killed the input multiple times and it did come back in sync.

 Ben, are You using udpsrc for my pipeline or souphttpsrc ? Because i had problems only with udp, using same streamer outputing to http and udp at same time i had different results. But seems that do-timestamp=false did the trick for udpsrc.

Thanks!

Mantas

[benjamin_d_3 · Accepted solution]

Hi Ishmael & Mantas,

First I want to thank Ishmael for the hard work and making this great plugin, we already started using it in production and dropped VAAPI due to quality and bug issues. I don't know how you did this but it's amazing for what I can see you are the only one developing this and solved every bug we found, thanks to Mantas as well for helping us find bugs.

Back to my last test results: I did run a pipeline with udpsrc and audio is good, i did make the source mess up by making the bandwidth lower then what source was in my switch to see if it goes out of sync then after it went back in sync again, I wasnt able to reproduce Mantas issue, maybe last fix solved it. 

I tried running other sources as well like rtmp hls tcp they all worked fine.

As far as memory that seems to be good too i have a process running for 21 hours at only 110MB. 

[benjamin_d_3]

Hi Ishmael, 

I thought the audio sync was fixed but when started doing longer tests we noticed that after longer periods of time it is still happening, after 24 hours keeps going out of sync.

Regards

Ben

[Ishmael_Sameen]

Hi Ben, And here I thought it was over :( Let's deduce what happened. Firstly, try using avdec_h264 instead of mfxh264dec and run for 24+ hours, see if it goes out of sync. If it's good then it means mfxh264dec is not robust enough to handle gradual variances in timestamping from the source plugin. Try what Mantas has suggested, do-timestamp=false in the source plugin with mfxh264dec in the pipeline. You may use an element like videorate to control the frame rate and hence the timestamping. Make sure you use video/x-raw(ANY) to keep frames in video memory. Please do let me know what you get. Meanwhile let me try to figure out how this could happen, and how I can deal with the timestamping issues in a more robust manner. Regards, Ishmael P.S. You could restart the pipeline every 24 hours to sync again but that wouldn't be a preferred solution I suppose.

[benjamin_d_3]

Hi Ishmael, 

About "do-timestamp" in souphttpsrc it's false by default: 

do-timestamp          : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: false

About framerate I tried it with set frame rate and with out, it went out of sync both with ways.

gst-launch-1.0 souphttpsrc location="http://ip:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! mfxh264dec ! mfxvpp width=1280 height=720 framerate=25/1 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true name=mux ! rtmpsink location="rtmp://ip:1935/pushrtmp/mfxtesasfht1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.

Then last thing i did, try it with avdec_h264 for 2 days and no sync issue, so it's definitely something with mfxh264dec. 

gst-launch-1.0 souphttpsrc location="http://ip:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! avdec_h264 ! videoconvert ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true name=mux ! rtmpsink location="rtmp://ip:1935/pushrtmp/mfxtesasfht1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.

Best Regards,
Ben

[Ishmael_Sameen]

Hi Ben,

Try this pipeline:

gst-launch-1.0 souphttpsrc location="http://ip:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux  ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! mfxh264dec ! videorate ! 'video/x-raw(ANY), framerate=25/1' ! mfxvpp width=1280 height=720 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true name=mux ! rtmpsink location="rtmp://ip:1935/pushrtmp/mfxtesasfht1s live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.

The videorate element should enforce the frame rate and timestamping for the decoded frames, over-riding its original timestamps when necessary. Please let me know if the out-of-sync issue remains with the new pipeline.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, 

I tried the pipeline you suggested but unfortunately the results were the same it went out of sync in couple of hours.

Thanks for your help.

Best Regards
Ben

[Ishmael_Sameen]

Hi Ben,

Sorry for the delayed response.

It seems like by 'correcting' the timestamps coming from the live stream it actually has a higher chance of being out-of-sync. This is pretty hard to debug, it proves that timestamps coming from a live stream are inconsistent and that the decoder must adapt to the changes. In avdec_h264, the FFMPEG decoder has a built-in H264 parser that is more robust to these kind of timestamping fluctuations compared to the h264parse plugin, that's why you don't see the out-of-sync issue in combination with the MFX H264 encoder.

The key for me is to try to reproduce the issue, so I'll have to run the same pipeline you have under the same conditions you are running. If you can somehow get me a dump of the stream that can cause the out-of-sync issue, that exact moment, then I have a great sample to then fix up the robustness of the MFX decoder. I think to do this you will have to run the original MFX pipeline without the videorate element, and at the same time, another pipeline dumping the original stream. The part where it starts to get out-of-sync, that's the file I would really like to have.

Also, please do get the validated version from the 01.org repo, it's actually later than my own personal repo this time, with Klockwork and some other fixes as well. We'll use my personal repo only when a proposed patch / hotfix is provided to fix issues.

https://github.com/01org/gstreamer-media-SDK

Do note the cmake and make install commands could be a bit different for your case. I think typically your sequence is something like this:

mkdir build; cd build; cmake -DUSE_VP9_DECODER=OFF -DCMAKE_INSTALL_PREFIX=/usr/lib64/gstreamer-1.0; make; sudo make install

The codebase is actually meant for Intel's customers, not really configured for the broad market.

Thanks for your hard work in testing the plugins, will continue to probe and resolve the issue from here.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael, 

Ok we are going to try that. Can you please explain about what is considered to be an Intel Customers ?

Our company uses Intel VCA in more then 50 server's and that's where we are doing are the test's in 
https://downloadcenter.intel.com/download/26395/Intel-Visual-Compute-Accelerator-Persistent-Reference-Package?product=87380

Cento 7.2 with Intel kernels and pre installed Media Server Studio

[Ishmael_Sameen]

Hi Ben,

Sorry, I wasn't being specific enough. It was meant for IOTG first-tier embedded customers, the plugins were initially developed and configured for Apollo Lake Yocto Media SDK Embedded Edition. I used to work for IOTG division, primarily developing and optimizing the GStreamer stack for these customers, but I also made sure it was generic enough so that it will also work with MSS 2017 Linux which normally is targeted for the server and broad market.

Anyway, yes, the idea is to dump the stream, and simply play and seek to the part where it goes out of sync. When you install the plugins, playbin will automatically select the installed MFX decoders as the primary plugin for decoding due to their primary ranking. Once you have found that part of the dumped stream where it goes out of sync, that's the part you could help cut out for me so that I have a great sample for debugging the out of sync issue. My gut feeling is at one point of the stream, the presentation and decode timestamps may have been reset, but really, the only way to find out is to check the timestamps of that troublesome part of the stream, and then  re-orient the decoder accordingly based on  the timestamp changes.

Please let me know when you can make that sample available for me so that I can finally reproduce and resolve that annoying out of sync issue.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael,

Thank you for the explanation, now I understand.

as far as the source link please check you PM.

best reagrds
Ben

[benjamin_d_3]

Hi Ishmael,

I have sent a PM with a stream dump as you requested, I'm writing here because not sure if you seen the PM.

I also see you have some new commits in your repo, do they include any new fixes for out of sync issue or they are about something else ?

Best Regards 
Ben

[Ishmael_Sameen]

Hi Ben,

No worries, I have gotten your message and your stream dump two days back. It's a very interesting dump, such that even gst-libav itself cannot decode it correctly with proper timestamp synchnronization as well! You could try playing it back with Totem or gst-play to verify yourself, make sure you uninstall the MFX plugins first.

However, playing back the dump via VLC or MPC would result in correct synchronization, so the problem highly likely lies with the demuxers / parsers used to play back the stream via GStreamer.

In any case since you have reported that avdec_h264 decoder works correctly with mfxh264enc in your use case, my aim is simply to match the timestamp behaviour of avdec_h264 for mfxh264dec. One of the cleanup commits possibly improved it but not entirely so, so you could try out the latest from my personal repo and report to me your results. I'm still working on the fix though, will let you know when I think it's done.

The other commits are targeted to "resolve" an interesting MPEG2 encode behaviour reported by another user, especially using multiple instances. Really interesting finding, but let's just say this is more or less a finished story.

I will also let you know if I need more samples from you, so far the sample you've provided suffices for my work.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishmael thank you,

Note.. also I tested the same source with VAAPI and I didn't get any out of sync issue. 
PART of the PIPELINE: ! vaapih264dec ! vaapipostproc width=1280 height=720 deinterlace-method=1 ! vaapih264enc rate-control=2 bitrate=1700 !

I will test your latest changes and I wil let you know.

Best Regards
Ben

[benjamin_d_3]

Hi Ishmael,

I tested with your latest commits but the issue is still there few hours later it went out of sync.

Regards,
Ben

[Ishmael_Sameen]

Hi Ben,

Can you execute the following command first prior to your test pipeline?

export GST_DEBUG=mfxdecode:6

When the out-of-sync issue happens, please provide the log and the output encoded file via PM.

Regards,

Ishmael

[benjamin_d_3]

Hi Ishamel, 

I have sent you a PM with the log and encoded output.

Also another thing i wanted to remind you of, is that the sources that im testing are all interlaced and I see mfxvpp deinterlaces them automatically, I looked if i can disable deinterlacing but there is no such option for mfxvpp.
Not sure if deinterlacing has to do with out of sync maybe it's good idea for mfxvpp to have a option to disable deinterlacing anywas like other deinterlcing elements in gstreamer. Just a suggestion.  

[Nilesh_K_]

Hi Ishmael,

I am using media-gstreamer plugins (mainly mfxdecode & mfxsink) from below branch:

https://github.com/01org/gstreamer-media-SDK/tree/release/gold

Commit ID:- b06e8efb681a5c528465d3cdcd9b7296503bdf49

I am planning to execute H264 video playback using mfxdecode & mfxsink plugins from media-gstreamers on Linux platform.

I have gone through media-gstreamer readme to understand usage of plugins.

But i am unable to locate any documentation for media-gstreamer from design & coding point of view.

Is it possible for you to share some design documentations for mfxsink plugin? This will be very much useful for me to understand plugin in depth.

Regards,

Nilesh

[Ishmael_Sameen]

Hi All,

Bumping up this thread for important updates regarding D3D11 Window 10 support.

Ishmael