Green Frame When Decoding Two Jpeg Streams With Gstreamer

[BLock]

Hello all,

I've been using the latest Intel Media SDK with Gstreamer through the msdk plugins on an Intel NUC6i7KYK mini-pc running Windows 10 64bit.

Overall, I've been loving the improvements in performance relative to using software decoding/encoding, but I'm having a strange issue when trying to decode two mjpeg streams at the same time. When I create/decode two mjpeg streams from two webcams with the command line:

 ksvideosrc device-index=0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! queue ! videorate ! jpegparse ! msdkmjpegdec hardware=1 ! videorate ! queue ! fpsdisplaysink text-overlay=0 ksvideosrc device-index=1 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! queue ! videorate ! jpegparse ! msdkmjpegdec hardware=1 ! videorate ! queue ! fpsdisplaysink text-overlay=0

I get flashes of green in one of the views, which eventually stops altogether with a "device-operation" failure. If I instead make both use hardware=0, I get no flashes of green but one view only outputs 20fps unless I manually set the pipeline latency. Using only one camera + decoder at a time works without any issue.

Has any anyone encountered an issue like this with flashes of green before a device failure? Does anyone have any suggestions for how to fix this issue when using the hardware decoder?

[Ishmael_Sameen]

Hi Brendan,

Have you tried these plugins?

https://github.com/ishmael1985/gstreamer-media-SDK

Just remove videorate from your pipeline and you're all set :)

[BLock]

Hi Ishmael,

Those plugins seem to be working excellently, much better than the mainline msdk plugins.

Would you happen to know why the pipeline doesn't seem to work if videorate is present?

And don't be surprised if you see a pull request from me on that repo :)

[Ishmael_Sameen]

Hi Brendan,

Pleased to know you've tried it. The significant performance improvement is because the decode and VPP operations are all done in video memory without any intermediate copies to system memory, including rendering (mfxsink).

Some elements such as videorate don't seem to have a good pass-through mechanism when processing such frames residing in video memory (sinkpad and srcpad caps can take a video/x-raw(ANY) type). I will further investigate this by checking the videorate source in gst-plugins-base, but if you really wish to upsample the framerate, you can use mfxvpp frame rate conversion (FRC) instead with even better performance. Note that mfxvpp FRC will only work with certain input frame rates, check the reference MSDK manual to find out more.

Sure, we will always welcome contributions to my repo :) I hope you're enjoying the plugins, it's as full-featured and fast as it can ever be, but there are some cool stuff we are still working on.

[BLock]

Hi Ishmael,

I would leave an issue on the gstreamer-media-SDK repo, but unfortunately I don't believe that repo has issues enabled.

I'm having some difficulty using your plugin with a tee. For example the pipeline:

gst-launch-1.0 -v videotestsrc ! "video/x-raw,width=2160,height=1920,format=YUY2" ! tee name=tp tp. ! queue ! identity silent=0 ! fakesink tp. ! queue ! videoconvert ! mfxh264enc rate-control=vbr preset=veryfast async-depth=4 gop-size=150 bitrate=4096 ! fakesink

Seems to stall after the first frame. However, if I remove the tee:

gst-launch-1.0 -v videotestsrc ! "video/x-raw,width=2160,height=1920,format=YUY2" ! identity silent=0 ! queue ! videoconvert ! mfxh264enc rate-control=vbr preset=veryfast async-depth=4 gop-size=150 bitrate=4096 ! fakesink

It works perfectly. Similarly if I lower the resolution:

gst-launch-1.0 -v videotestsrc ! "video/x-raw,width=1920,height=1360,format=YUY2" ! tee name=tp tp. ! queue ! identity silent=0 ! fakesink tp. ! queue ! mfxh264enc rate-control=vbr preset=veryfast async-depth=4 gop-size=150 bitrate=4096 ! fakesink

It also plays correctly.

 I just pulled/recompiled from master today, so I believe it should be fully up to date. When this is running I get no logs suggesting the cause of this issue, even with your plugin's logging set to 6. I also get about 0.1% CPU usage when this is running.

Would you happen to have any ideas what might be causing this issue? Or anything I should look into to find a solution?

Thank you for making a great plugin and any help you may be able to provide,

--Brendan

[Ishmael_Sameen]

Hi Brendan,

Thanks for letting me know about the "Issues" being disabled on my repo. I enabled it now.

You've hit an issue which I also noticed as well in the midst of development. This is a serious threading issue, and it involves some conflict with the threading mechanism between MSDK and GStreamer, both heavily threaded frameworks. Unfortunately, while I do have the idea about the root cause, I don't have the MSDK proprietary source code for Windows here. And the problem could be deeper than  imagined, but in normal use cases, separate GStreamer pipelines are generally invoked and run concurrently, so it's an issue that hasn't been reported to us.

To demonstrate the issue, if you set async-depth=1 property for mfxh264enc in the first pipeline you showed, it will work perfectly. The queue element essentially invokes a thread boundary in GStreamer, and yet MSDK can't differentiate between GStreamer threads and its own internal threads. This is probably the main reason why the deadlock occurs for your use case.

In Linux however, it doesn't actually happen as often, so I would assume  that the libva backend for MSDK is better threaded, not to mention that there are libva lock guards in the libva frame allocator for each native libva call.

A possible solution is to control the threading mechanism externally as described here:

https://software.intel.com/en-us/node/628429#MFXInitEx

This is a considerably a lot more work involved here if you use this feature, so if you can somehow refactor your use case to simply use separate GStreamer pipelines instead of tee and queue combinations, I think that would greatly ease up your development time. There will be a performance penalty from oversubscribing threads, but I suppose it's no big deal to you unless you are working in a heavily constrained hardware environments.

Thanks for the report, please keep them coming and let us know what you find.

Regards,

Ishmael