Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

Question about VPP Composition

Ken_S_
Beginner
636 Views

Hi there,

I have a question about VPP Composition in sample_vpp sample code.
In the sample code, the logic is as follow assuming we have two video to composite:

    get a free surface
    load video 1
    RunFrameVPPAsync
    if (status == ERR_MORE_SURFACE)
    {
        get a free surface
        load video 2
    }
    RunFrameVPPAsync
    and so forth

So, my question is that when RunFrameVPPAsync returns ERR_MORE_SURFACE, how do we know which Video we should load more data? The sample code simply alternates between Video 1 and 2.


Thanks in advance as always,
Ken S.

 

0 Kudos
7 Replies
Ken_S_
Beginner
636 Views

Hi there,

Another question regarding to VPP Composition. Basically if I have two input files that I would like to do composition, one file is H264 file and another is a raw NV12 file. The H264 file will require to decode and send to VPP and another one would just copy the data to the Frame Surface. Diagram below illustrates what I am trying to accomplish:

CompositeVPP.png

Couple questions:
1) Can I use Opaque Surface for "Surface #1" and "Surface #2"? If so, how do I configure it?
2) Can I mix the surface type? Meaning "Surface #1" uses Opaque Surface and "Surface #2" uses Video Memory Surface?

In general, I am a bit confused what type of Frame Surface I could use for "Surface #1" and "Surface #2". I couldn't find any sample code that does decode and composite :(

Any help and information will be greatly appreciated!
Ken

 

0 Kudos
Ken_S_
Beginner
636 Views

To be more specific,

When configuring the opaque frame surface between the mfxDecoder and mfxVPP (Composite),
[Decoder Parameter]
- mfxVideoParam.IOPattern  = MFX_IOPATTERN_OUT_OPAQUE_MEMORY
- mfxExtOpaqueSurfaceAlloc.Out.Type = MFX_MEMTYPE_OPAQUE_FRAME | MFX_MEMTYPE_FROM_DECODE | MFX_MEMTYPE_DXVA2_DECODER_TARGET

[VPP Composite Parameter]
- mfxVideoParam.IOPattern = MFX_IOPATTERN_IN_OPAQUE_MEMORY | MFX_IOPATTERN_OUT_OPAQUE_MEMORY
- mfxExtOpaqueSurfaceAlloc.In = Decoder.mfxExtOpaqueSurfaceAlloc.Out
- m_OpaqueSurfaceAlloc.Out.Type = MFX_MEMTYPE_OPAQUE_FRAME | MFX_MEMTYPE_DXVA2_PROCESSOR_TARGET | MFX_MEMTYPE_FROM_VPPIN

This implies that the opaque surface between the Decoder and VPP (Composite) are basically hard-wired to a specific surface's type.
So, when I tried to configure the frame surface between the NV12RawImage and VPP (Composite) in the diagram above, I am not sure what kind of frame surface I can use since the "frame surface" between Decoder & VPP is tightly coupled :(

Questions are:
1) Does it mean opaque surface won't work in my scenario?
2) Even if I use video surface, how do I allocate surfaces such that the VPP Composite can process the data from both the Decoder or the NV12RawImage?

Thanks,
Ken

0 Kudos
Sravanthi_K_Intel
636 Views

Hi Ken,

Regarding your first question/message - If you look at the Media SDK applications (samples or tutorials alike), the encode/decode/vpp functions are asynchronous calls. Meaning they are non-blocking calls that operate on the surface and issue commands (if you will) to the driver to be processed. When the number of surfaces run out, you want to synchronize the in-use surfaces to free them up for the next calls. In the case of VPP, the surface is allocated for the "composited" frame (or output). So, the more surface required means that the output frame is ready after synchronization, and it needs more surfaces at output for additional output.

What you are referring to could be need more data return status. For this case, the we maintain a "bDoNotUpdateIn" flag that tells the application if we did not read all the data for the input stream (in which case, do not move to the next input). Hope this helps your understanding. If not, please let me know and can go into some more details.

0 Kudos
Sravanthi_K_Intel
636 Views

Hi Ken,

The composition example can be extended from VPP to Decode+VPP. Looks like you are having trouble understanding how to use opaque surfaces - did you have a chance to look at the simple_5_transcode_opaque_async_vppresize tutorial? It shows how to use opaque surfaces between the components of the application (dec, vpp and enc). 

With that information, you could add a DecodeFrameAsync call to your while() loop that is processing the VPP Async call. The output of the DecodeFrame will feed the VPPAsync call. You will manage "more data" and "more surface" return status. (Again, transcode tutorial will show you how - extend the VPPAsync part of it with the composition-specific code).

Opaque surfaces reduce the overhead for the developer to worry about how surface allocation is handled internally. The tutorial is a good start - https://software.intel.com/en-us/intel-media-server-studio-support/training. That combined with your understand of composition should be helpful.

Let me know if you have more questions.

0 Kudos
Ken_S_
Beginner
636 Views

Hi Sravanthi,

Thanks for your response. Indeed I have gone through all the sample codes and tutorials. And, I was able to construct a simple transcoder with opaque memory without any problem.

However the problem I have is that when I tried to composition (for example adding a logo on the video in the transcoding process), I am stuck. It seems to me that I can't use opaque memory if I want to composite two sources together.

So, let me rephrase my questions again.
Q1) Can we use opaque memory if we are loading a raw NV12 video to VPP (RunFrameVPPAsync), not decoding a frame?
For instance, Intel provides a sample code, called sample_vpp. The sample illustrates video composition with Video Memory. So, can you tell me if the same sample_vpp can also use Opaque Memory?
Inside the sample app, there is code that "MOVES/LOADS" raw data onto a surface
   yuvReaders[nInStreamInd].LoadNextFrame( &(pInSurf[nInStreamInd]->Data), &(inFrameInfo[nInStreamInd]));
And, I don't see how this can be done with Opaque Memory. Based on my debugging observation, the Opaque memory doesn't assign real memory address to the surface (to the user-space). Any attempt to manually "move" data to the Opaque Memory surface will get access violation!

Q2)
The simple transcode illustrated below works.
H264  --> Decode --> (Opaque) --> VPP --> (Opaque) --> Encode

But, if I want to do the following,
H264  --> Decode --> (Opaque) --+
                                                +-->  VPP --> (Opaque) --> Encode
Logo Image ----------> (  ????  )  --+ 

What is my option?

Thanks,
K

0 Kudos
Ken_S_
Beginner
636 Views

Just FYI,
I can at least get the composition works with Video Memory. Still don't know how to do it with Opaque Memory.

K

0 Kudos
Sravanthi_K_Intel
636 Views

Hi Ken, Apologies for the late response (remained in drafts).

Opaque memory is intended for straight-forward pipelines that are end-to-end. For the one you mention above, video memory is the way to go as you have already observed. It gives the best performance and allows passing surface handles between pipeline stages. In opqaue, it is a "black-box" approach if the developer does not want to concern how the memory is handled, and does not have pointers for surfaces to be passed around (which is what you'd need). 

Apologies again for the delayed response. But glad to see you are already on the correct path.

0 Kudos
Reply