BGRA output not working when using the h264/h265 Intel's Gstreamer Media SDK

[Ben_Rush]

I've been successfully using the Intel Media SDK GStreamer plugins on Windows for a while now (specifically this branch), but I've encountered an error and I'm wondering if someone can help me out. 

What I'm trying to do is generate BGRA output from either the h264 or h265 encoders, but some negotiation with downstream elements in the pipeline is failing. NV12 works just fine, and so I know I've got things hooked up right. The failure occurs when wanting the encoder to generate any other colorspace but NV12. 

A bit more detail

I've got an RTSP stream that I'm reading, and I'm wanting to draw it onto some Windows Presentation Foundation control, which only allows BGRA/RGBA/RGB/etc. I'm using the plugin in code, but I can replicate the problem using just gst-launch command: 

gst-launch-1.0.exe rtspsrc location=rtsp://address/live ! decodebin ! 'video/x-raw(memory:MFXSurface),format=BGRA' ! autovideosink

This will fail. The decodebin does use the Intel Media plugins when building the pipeline. If you enable debugging through GST_DEBUG=4 or higher, you will get the following error/warning: 

WARN                GST_PADS gstpad.c:4230:gst_pad_peer_query:<mfxdec_h264-0:src> could not send sticky events

The net effect is the pipeline never fully transitions to the play state. If I watch the GPU and CPU, I can see it's chewing, but downstream elements don't get any data. 

Flipping BGRA to NV12, the stream will work: the pipeline transitions to PLAY and downstream elements get data and I can view my stream:

gst-launch-1.0.exe rtspsrc location=rtsp://address/live ! decodebin ! 'video/x-raw(memory:MFXSurface),format=NV12' ! autovideosink

When I inspect the MFX GStreamer elements, they all advertise they can handle BGRA. For example, here are the pad outputs generated by gst-inspect when examining the H265 encoder: 
 

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
              alignment: au
                profile: { (string)main, (string)main-10 }
          stream-format: byte-stream

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:MFXSurface)
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2, (string)ENCODED }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1

I'm running on Windows 10. My CPU is a i7-6700K @ 4GHZ. 

Any insight or thoughts would be appreciated. I'm digging through the Media SDK's GStreamer plugin code right now to find the source of the issue, but I'm hoping someone can offer a pointer or two. Thanks. 

Edit: 

I have found that the problem at least manifests itself at this point in the code. The call to gst_video_decoder_finish_frame() returns GST_FLOW_NOT_NEGOTIATED if BGRA is specified, GST_FLOW_OK if NV12 is specified. 

[Mark_L_Intel1]

Hi Ben,

Thanks for the post and it is good to know a user is using Media SDK GStreamer plugin in Windows in this forum.

As you might understand, we don't support MSDK GST plugin here but I still want to help you.

I never use this plugin before, so I want to clarify your test. First I want to know if you are using hardware accelerated codec in this case because I can't see it from the command line you posted; also the output message seems point to a software codec--"video/x-h265"

From Media SDK point of view, its decoder only outputs NV12, in order to output other format, the NV12 frame must be converted by Video Post Process component(VPP), I can't see this in your command line either.

Mark

[Ben_Rush]

Liu, Mark (Intel) wrote:

it is good to know a user is using Media SDK GStreamer plugin in Windows in this forum.

Yes, I seem to be the only one doing anything with this plugin on Windows. In fact, I seem to be one of the few using GStreamer on Windows :)

Liu, Mark (Intel) wrote:

As you might understand, we don't support MSDK GST plugin here but I still want to help you.

You say "here", is there a place where it is more well-supported? Or is the GST plugin more of a side project and a "use at your own risk" kind of thing. I understand the situation either way, but just curious. I've had to do a couple of tweaks here and there to get things to work on Windows. 

Liu, Mark (Intel) wrote:

First I want to know if you are using hardware accelerated codec in this case because I can't see it from the command line you posted; also the output message seems point to a software codec--"video/x-h265"

Yes, I am. The caps string for the GstMemory objects I get when decoding say MFXSurface on them (video/x-raw(memory:MFXSurface)) and I'm absolutely seeing the GPU being utilized when watching my process in task manager. I also know that I've had to learn how to use GStreamer to talk to the video memory buffers the plugin makes available to me. So, I'm confident it's hardware accelerated. 

Liu, Mark (Intel) wrote:

From Media SDK point of view, its decoder only outputs NV12, in order to output other format,

So, I think this is key. The decoder, at least according to the caps advertised on the output pins, explicitly advertise they can produce BGRA output, and so I thought the decoder could/would do this for me. Here are pad details taken from the "gst-inspect" command for the H265 decoder by running gst-inspect-1.0.exe mfxhevcdec: 

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
              alignment: au
                profile: { (string)main, (string)main-10 }
          stream-format: byte-stream

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:MFXSurface)
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2, (string)ENCODED }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

You can see it states BGRA as a valid output on the 'src' or output pin. Therefore, it was my understanding no further elements need to be included in the pipeline. I guess perhaps this is an error and that BGRA shouldn't be listed here. 

Liu, Mark (Intel) wrote:

the NV12 frame must be converted by Video Post Process component(VPP), I can't see this in your command line either.

These are the available elements in the MFX plugin, I see the mfxvpp plugin. I'm assuming this is the plugin that I need to integrate into the pipeline in order to do the conversion? I'll do some looking at it and see what I need to do in order to utilize it. 
 

  mfxh264dec: MFX H264 decoder
  mfxhevcdec: MFX HEVC decoder
  mfxmpeg2dec: MFX MPEG2 decoder
  mfxwmvdec: MFX WMV decoder
  mfxvc1dec: MFX VC1 decoder
  mfxvp9dec: MFX VP9 decoder
  mfxjpegdec: MFX JPEG decoder
  mfxdecode: MFX Video Decoder
  mfxvpp: MFX video postprocessing
  mfxsinkelement: MFX sink
  mfxsink: MFX Sink Bin
  mfxh264enc: MFX H.264 encoder
  mfxhevcenc: MFX H.265 encoder
  mfxjpegenc: MFX JPEG encoder

Thanks for your feedback. I appreciate you taking the time!

[Ishmael_Sameen]

This should work.

gst-launch-1.0.exe rtspsrc location=rtsp://address/live ! decodebin ! mfxvpp ! 'video/x-raw(memory:MFXSurface),format=BGRA' ! autovideosink

[Ben_Rush]

Yup! It sure does. I actually took what Mark had told me and was able to piece the above line together. Thanks for your help, guys. It was much appreciated. 

[Mark_L_Intel1]

Good to know you solved it with Ishmael's good comment.

The capability message seems to be confused; I was looking at the mfx component list and vpp is there.

Mark

[Caleb_M_Intel]

Hi, I am trying to do the same thing, but unfortunately Ishmael's command did not work for me. Sorry, I have used Media SDK quite a bit, but am pretty new when it comes to GStreamer. I am running this in Ubuntu 16.04 with GStreamer 1.8.3.

I have the Media SDK GST plugins built and installed from this repo. Here is the pipeline I am trying run based on Ishmael's comment:

rtspsrc location=\"rtsp://localhost:8554/\" latency=100 ! rtph264depay ! h264parse ! mfxdecode ! mfxvpp ! 'video/x-raw(memory:MFXSurface),format=BGRA' ! appsink drop=TRUE max-buffers=10

But I get 2 syntax errors:

1. no element "video"
2. no source element for URI "/x-raw(memory:MFXSurface),format=BGRA'"

If i remove 'video/x-raw(memory:MFXSurface),format=BGRA' part, the pipeline works fine, but the output is NV12, obviously.

Is the syntax different on Ubuntu?

Thanks,

Caleb