VPP Composition with Interlaced content

[Ken_S_]

Hi there,

I have a question about composite two video, the primary video is H264 "Interlaced" content and the overlay video is raw NV12 "Progressive" content.

1) How does MSDK handle this? Would the following pipeline work?

[h264] --> [decoder] --+
                                +--> [vpp composite] --> [output]
              [raw nv12] --+
 

2) Or, do I need to deinterface the primary video first before I can do composite it?

[h264] --> [decoder] --> [vpp deinterlace] --+
                                                              +--> [vpp composite] --> [output]
                                             [raw nv12]--+

3) Do I need to do the sync operation after "vpp deinterlace" meaning moving the data from surface to memory buffer first?


Thanks,
Ken
 

[Surbhi_M_Intel]

Hi Ken, 

Pipeline described in point 1 can be used. You can use the interlaced and progressive input together to composite. o/p video will have composition of an interlaced and progressive content. 
If you would like to deinterlace the input, then configure both composition and deinterlacer, which is your 2nd pipeline. 
You don't need to configure two sync operations when choosing to use two filters. Media SDK should handle that for. Please check sample_vpp  in the samples package to configure similar pipeline. 
Let us know if you have any follow up questions. 

Thanks,
Surbhi

[Ken_S_]

Hi Surbhi,

The pipeline above is simplified. The output in fact goes into the encoder. Here is the full pipeline.

[h264] --> [decoder] --+
                                       +--> [vpp composite] --> [sink]
                [raw nv12] --+

[source] --> [vpp resize] --> [encoder] --> [h264 output]

If I have both progressive video, this works perfectly.
Unfortunately if the primary video is interlaced and the secondary video is progressive, the encoder fails. The Sync Operation reports an error code (-12) which is MFX_ERR_ABORTED. The error code doesn't really help. The entire pipeline appears to be ok and I got a valid SyncPoint as well. But, as soon as I do a sync operation, it just failed!

I did go through the sample_vpp application. But, I have to say the sample_vpp code has very limited usage and doesn't really help me to understand why 2 progressive video composition works but interlaced with progressive video will fail.

For instance, sample_vpp uses both RAW inputs and also only ouput RAW. And, the PicStruct relies on user to set it. In contrast, our primary video is compressed video (h264 or mpeg2). The FrameInfo is decoded by MSDK decoder (calling MFXVideoDECODE_DecodeHeader function). After decode the header, the PicStruct was set to 0 which is MFX_PICSTRUCT_UNKNOWN. That doesn't seem to help setting up the vpp.In structure neither.

So, my question is that while I am debugging, what should I be looking at? PicStruct value on each frame? or else? Again, compositing two compressed progressive video work fine. But, if one of them is interlaced, it fails!

Thanks,
Ken

[Ken_S_]

Would it help if I upload the interlaced sample for you?

I have tried both h264 interlaced content as well as mpeg2 interlaced content. Both failed on sync operation with an ABORT error.

[Surbhi_M_Intel]

Hi Ken, 

I understand your problem, workaround to this issue- after the decode, can you overwrite the pictstruct to be progressive, so that pictstruct is not unknown in the VPP structure. What this will do is take the inputs as progressive and o/p composite of an interlaced and progressive content. 
Another alternate would be to de-interlace the content, before calling composite filter in your pipeline.  The pipeline you have explained in point1 originally is not fully supported(sorry for the confusion). Basically we cannot composite interlaced content, it has to be progressive. That's why we suspect workaround might work if you are telling application it is a progressive content, treat it like that. 

Please let us know if that works for you, send me a reproducer of your issue in case it doesn't. We will try to help the best we can!

Thanks,
Surbhi

[Ken_S_]

Hi Surbhi,

I tried your suggestion by forcing PicStruct to Progressive after the Decoder. It doesn't work. As soon as I do SyncOperation, it fails with MFX_ERR_ABORT again.

Thus, I modified the pipeline to deinterlace first before doing composition. At least I could get it to encode. But, it still has other issues (frame stuttering/hesitation).
The pipeline I have is as follows:
- 1st session handles the decode and deinterlacing
- 2nd session just handles the composition between the raw image and the decoded/deinterlaced content
- 3rd session will do the resizing on the composed frame and encode to H264

[h264 interlaced] --> [Decoder] --> [VPP Deinterlace] -- (session #1)

(session #1) -----------+
                              +----> [VPP Composite]  -- (session #2)
[raw RGB4 frame]----+

(session #2) ---> [VPP Resizer] --> [Encoder] --> [output to file]  -- (session #3)

With this pipeline, I could at least get the content to encode instead of receiving MFX_ERR_ABORT. Unfortunately, the video stutters a lot. It seems to me that the encode order is incorrect.

I posted two video for you to look at. One is with composition that has video stuttering/hesitation. The second one is without composition and the encoded content looks fine.

1) w/ VPP composition (video stutters), it gets very severe at 00:00:15
http://youtu.be/nkzXCV808R0

2) w/o VPP composition (video is OK)
http://youtu.be/OLvwJNMDkYk

The main difference between #1 and #2 is that #2 doesn't have the [VPP Composite] component in the pipeline.

Can you tell me what could go wrong?
Does the Decoder decode frame in sequential order?

My wild guess is that the Decoding is not in sequential order and the encoder loses the "decode order" after composition.
I traced the FrameOrder going in/out of each component on both scenarios. Without the composition (scenario #2), the frame surface carries the FrameOrder number into the encoder. With the composition (scenario #1), the FrameOrder number is always 0. I tried to inject the FrameOrder number sequentially but it still doesn't work :(


Thanks,
Ken
 

[Ken_S_]

Hi Surbhi,

Any updates or suggestions from Intel regarding to the frame hesitation issue?

Thanks,
Ken
 

[Surbhi_M_Intel]

Hi Ken, 

Sorry I missed your last post. I am discussing this issue with experts. So far we are not able to reproduce the issue, it will be great if you could send us a reproducer and input for this issue. How many inputs are there in session? This will make the process faster and will be able to resolve the issue soon. 

Thanks,
Surbhi 

[Ken_S_]

Hi Surbhi,

Reproducer - do you mean a test sample that can reproduce the issue? We are using our own application. Should I send you our code to reproduce the issue?

I uploaded the interlaced content to google drive for your reference.
https://drive.google.com/file/d/0BzzniwLw7a4DMXF0Z1BxZWgycnM/view?usp=sharing

By the way, the issue is getting even more interesting. If I do multiple encode sessions (multiple outputs), then the video were completely fine (no stuttering). Meaning the video stuttering only occurs when there is one encode session. Nothing is changed. It is identical pipeline but just have an additional encode session as follows:

[h264 interlaced] --> [Decoder] --> [VPP Deinterlace] -- (session #1)

(session #1) -----------+
                                      +----> [VPP Composite]  -- (session #2)
[raw RGB4 frame]----+

(session #2) ---> [VPP Resizer] --> [Encoder] --> [output to file #1]  -- (session #3)

(session #2) ---> [VPP Resizer] --> [Encoder] --> [output to file #2]  -- (session #4)

Ken
 

[Surbhi_M_Intel]

Hi Ken, 

That's really interesting to know. Can you send the application, which can reproduce this issue or successfully replicate this behavior through a private message or you can email me on surbhi.madan@intel.com. I have a question about your application - After composition, is your o/p RGB4 frames? Are you doing color conversion at any point in your pipeline?

Thanks,
Surbhi

[Ken_S_]

Hi Surbhi,

Thanks for pointing out the MSDK version. I confirmed that the latest R6 version does resolve the video stuttering problem on interlaced content.

Ken S.

[Pavel_P_1]

Hello,

I'm trying to make composition with video and ARGB picture on 2017r2 and have the same problem.

My target sheme of execution is:

(session 1: [decoder]) ---> (session 2: [VPP]) ---> (session 3: [composite VPP] ---> [encoder])
                                                                                (ARGB surface) ---^
I use separate session for [VPP] because there can be multiple variants of resolution and frame rate at the same time.

The result of encoding looks like the video with glitches from Ken S.

I made some experiments with different varaints of execution scheme, and here are the results:

scheme A (works fine):
(session 1: [decoder]) ---> (session 2: [composite VPP] ---> [encoder])
                                           (ARGB surface) ---^

scheme B (works fine):
(session 1: [decoder]) ---> (session 2: [composite VPP]) ---> (session 3: [encoder])
                                            (ARGB surface) ---^
                            
scheme C (works fine):
(session 1: [decoder]) ---> (session 2: [composite VPP]) ---> (session 3: [VPP] ---> [encoder])
                                            (ARGB surface) ---^
                            
scheme D (works with glitches):
(session 1: [decoder]) ---> (session 2: [composite VPP]) ---> (session 3: [VPP]) ---> (session 4: [encoder])
                                            (ARGB surface) ---^
                            
scheme E (works with glitches):
(session 1: [decoder] ---> [VPP]) ---> (session 2: [composite VPP] ---> [encoder])
                                                            (ARGB surface) ---^
                
All the next experiments I made are only with scheme E.

Video - timer, picture - png with dices.

Firstly, I added 250 surfaces to frame pool of [VPP]. Additionally I numbered empty surfaces using vertical light lines (number of light lines is the index of the surface) and received the following result:

https://www.youtube.com/watch?v=Wpuhqh-E_F0

It shows me that [composite VPP] uses surfaces that were taken from [VPP] but not filled with data yet. I.e. [composite VPP] can draw surfaces i and i-1 while [VPP] fills only surface i-1, and i-th is empty (or old).

So I added MFXVideoCORE_SyncOperation() in [VPP] and the glitches are gone. Because it's a rather slow solution, I'm searching for an alternative approach. I changed the direction of drawing surfaces (first - ARGB picture, second - frame from [VPP]) and glitches were gone again (without MFXVideoCORE_SyncOperation()). Next I made the following scheme: I draw the video frame, then ARGB picture, and third - invisible video frame from [VPP] again, and there were no glitches too (without MFXVideoCORE_SyncOperation()). But it's a slow solution too.

What else can I do to find a correct solution of my problem?

[Pavel_P_1]

Hello, i made the other topic for my question.

https://software.intel.com/en-us/forums/intel-media-sdk/topic/734639

Pavel.