Transparent surface for composition

[hardik_panchal]

Hi,

In our application use case, we require to composite multiple video stream and render to the display.

Out pipeline for composition will be something like below:

decode1-->     |

decode2-->     |

                           |------> VPP Compositor --> Renderer

decode(N-1)    | 

decodeN-->     |

Note: Input streams can be different FPS (i.e. stream1 is of 25 FPS and stream2 is of 30 FPS). What I am expecting is that in this case compositor will be  composing all the decoded frames at constant rate which is highest of the frame rate of all decoded streams.

All media session are running in parallel threads.

Now in this case, there will a situation where VPP Compositor request decoded frame from each of the decoder session. The decoder which is running at lower FPS will not be able to provided decoded surface at that point. I want my compositor to don't wait for slowest decoder and keep doing its task of composition at constant rate. 

For this I need to provided a dummy or we can say transparent frame to VPP compositor in place of slowest decoder's surface.

Question: How can I generate a transparent frame which can be passed to VPP compositor?

[ArunJ_Intel]

Hi 

Could you please provide us a reproducer of the composite pipeline that you have built so we could reproduce this scenario from our end. Also please let us know which operating system you are using.

Thanks

Arun

[hardik_panchal]

Hi Anuj,

I am using Intel Atom 3940 ported with Yocto Embedded Linux.

Should I provide you with my sample application or architecture overview?

[ArunJ_Intel]

Hi

I was primarily looking for the sample application, could you share the architecture overview as well if possible.

Thanks

Arun

[hardik_panchal]

Hi @ArunJ_Intel,

Thanks for the prompt response.

I have attached my sample application for your reference.

Architecture overview:

-  We have taken reference for our sample application from Sample_multi_trasncode provided with Intel Media SDK.

- Sample application will decode two h264 streams (1. 704x576 at 25FPS and 2. 2592x1952 at 30 FPS). Decoded streams will be used for composition in 2x2 window layout.

- There are 3 Media SDK session present here. 1st in for VPP Composition , 2nd and 3rd for Decoding. Vpp session is used at parent session and all decoder session is joined with it.

- All Media SDK session are running in independent threads. 

- For sharing surfaces between decoder and composition session, linked list of surface buffers is maintained. Each decoder will decode one frame, put the decoded frame in it's surface buffer and give asynchronous signal to compositor session to read the frame. 

[ArunJ_Intel]

Hi 

Generating a dummy / transparent frame to compositor would be a complicated approach. A recommended solution in this case would be to use frame rate conversion option in media sdk to convert all streams which are input to compositor to have the same FPS. Please find more information about frame rate conversion in the below link.

https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-man.md#mfxExtVPPFrameRateConversion

Thanks

Arun Jose

[hardik_panchal]

Hi Arun,

Thanks for guidance. I will try it out.

[ArunJ_Intel]

Hi hardik_panchal,

Hope that answers your query, Is there anything else you need help with regarding this query.

Thanks

Arun

[hardik_panchal]

Hi @ArunJ_Intel 

Thanks for your support.

[ArunJ_Intel]

Thank you for the confirmation. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Regards

Arun