topic Re: Re:Transparent surface for composition in Media (Intel® Video Processing Library, Intel Media SDK)

Transparent surface for composition

hardik_panchal — Thu, 15 Apr 2021 12:12:28 GMT

Hi,

In our application use case, we require to composite multiple video stream and render to the display.

Out pipeline for composition will be something like below:

decode1--> |

decode2--> |

|------> VPP Compositor --> Renderer

decode(N-1) |

decodeN--> |

Note: Input streams can be different FPS (i.e. stream1 is of 25 FPS and stream2 is of 30 FPS). What I am expecting is that in this case compositor will be composing all the decoded frames at constant rate which is highest of the frame rate of all decoded streams.

All media session are running in parallel threads.

Now in this case, there will a situation where VPP Compositor request decoded frame from each of the decoder session. The decoder which is running at lower FPS will not be able to provided decoded surface at that point. I want my compositor to don't wait for slowest decoder and keep doing its task of composition at constant rate.

For this I need to provided a dummy or we can say transparent frame to VPP compositor in place of slowest decoder's surface.

Question: How can I generate a transparent frame which can be passed to VPP compositor?

Re:Transparent surface for composition

ArunJ_Intel — Fri, 16 Apr 2021 10:05:52 GMT

Could you please provide us a reproducer of the composite pipeline that you have built so we could reproduce this scenario from our end. Also please let us know which operating system you are using.

Thanks

Arun

Re: Re:Transparent surface for composition

hardik_panchal — Fri, 16 Apr 2021 10:28:38 GMT

Hi Anuj,

I am using Intel Atom 3940 ported with Yocto Embedded Linux.

Should I provide you with my sample application or architecture overview?

Re:Transparent surface for composition

ArunJ_Intel — Mon, 19 Apr 2021 06:27:52 GMT

I was primarily looking for the sample application, could you share the architecture overview as well if possible.

Thanks

Arun

Re: Re:Transparent surface for composition

hardik_panchal — Mon, 19 Apr 2021 07:06:48 GMT

Hi @ArunJ_Intel,

Thanks for the prompt response.

I have attached my sample application for your reference.

Architecture overview:

- We have taken reference for our sample application from Sample_multi_trasncode provided with Intel Media SDK.

- Sample application will decode two h264 streams (1. 704x576 at 25FPS and 2. 2592x1952 at 30 FPS). Decoded streams will be used for composition in 2x2 window layout.

- There are 3 Media SDK session present here. 1st in for VPP Composition , 2nd and 3rd for Decoding. Vpp session is used at parent session and all decoder session is joined with it.

- All Media SDK session are running in independent threads.

- For sharing surfaces between decoder and composition session, linked list of surface buffers is maintained. Each decoder will decode one frame, put the decoded frame in it's surface buffer and give asynchronous signal to compositor session to read the frame.

Re:Transparent surface for composition

ArunJ_Intel — Wed, 21 Apr 2021 09:34:44 GMT

Generating a dummy / transparent frame to compositor would be a complicated approach. A recommended solution in this case would be to use frame rate conversion option in media sdk to convert all streams which are input to compositor to have the same FPS. Please find more information about frame rate conversion in the below link.

https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-man.md#mfxExtVPPFrameRateConversion

Thanks

Arun Jose

Re: Re:Transparent surface for composition

hardik_panchal — Thu, 22 Apr 2021 04:32:07 GMT

Hi Arun,

Thanks for guidance. I will try it out.

Re:Transparent surface for composition

ArunJ_Intel — Mon, 26 Apr 2021 04:09:39 GMT

Hi hardik_panchal,

Hope that answers your query, Is there anything else you need help with regarding this query.

Thanks

Arun

Re: Re:Transparent surface for composition

hardik_panchal — Mon, 26 Apr 2021 04:18:25 GMT

Hi @ArunJ_Intel

Thanks for your support.

Re:Transparent surface for composition

ArunJ_Intel — Tue, 27 Apr 2021 05:37:19 GMT

Thank you for the confirmation. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Regards

Arun