Intel® oneAPI Threading Building Blocks
Ask questions and share information about adding parallelism to your applications when using this threading library.
2467 Discussions

Is there some Intel TBB example about video processing

applemax82
Beginner
2,847 Views

Hi,

 

Intel oneTBB is powerfull. Is the library suitable for developing video processing program, such as video transcoding?

 

Where can I find sample code about video processing?

 

Thanks!

 

Andrew

0 Kudos
18 Replies
SeshaP_Intel
Moderator
2,807 Views

Hi,

 

Thank you for posting in Intel Communities.

 

Intel OneAPI Threading Building Blocks (oneTBB) is a runtime-based parallel programming model for C++ code that uses threads.

You can use a parallel pipeline from oneTBB to develop video processing code. 

Please refer to the below link for more details.

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Working_on_the_Assembly_Line_pipeline.html

Please refer to the below link for more examples on oneTBB.

https://github.com/oneapi-src/oneTBB/tree/67c1171696fc539ba0eb57ad51d8963d63c2f5f6

 

Thanks and Regards

Pendyala Sesha Srinivas

 

0 Kudos
applemax82
Beginner
2,797 Views

What's the difference between pipeline and flow graph? Is flow graph suitable for video transcoding? Which one is better?

 

Thanks!

 

Regards

 

Andrew

0 Kudos
SeshaP_Intel
Moderator
2,760 Views

Hi,


A parallel_pipeline applies a linear series of filters to input items as they flow through a pipeline and they support basic pipelines.

Flow Graph offers a far more general solution that should be used when that generality is needed and supports both pipelines and generalized pipelines.

Based on the use case we can make use of any method to develop video processing code.


>>Is flow graph suitable for video transcoding?

Yes, we can use flow graphs for video transcoding.


Please refer to the below link for more details regarding flow graphs

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph.html


Thanks and Regards

Pendyala Sesha Srinivas


0 Kudos
SeshaP_Intel
Moderator
2,693 Views

Hi,


We haven't heard back from you. Could you please provide an update on your issue?


Thanks and Regards,

Pendyala Sesha Srinivas


0 Kudos
applemax82
Beginner
2,670 Views

I plan to construct flow graph by using the following nodes:

input_node/function_node/multifunction_node/queue_node/function_node/buffer_node

 

If I implement the body of function_node and input_node, how to report error to up-level program when running flow graph?

 

Thanks

 

Regards

 

Andrew

0 Kudos
applemax82
Beginner
2,663 Views

Hi,

 

For transcoding application, I should use some nodes to construct flow graph.

 

For example, input_node is used to read media file and demuxer audio/video streams. function_node is used to decode audio/video stream. Because decoding process is slower than AV demuxer, I guess queue_node should be inserted between demuxer(input_node) and deocder(function_node), just like demuxer -> buffer -> decoding. Is there some method to set the size of queue_node? How can I control the speed in which demuxer sends audio/video encoded data to decoder, to avoid too many audio/video encoded data are buffered in queue_node?

 

Thanks!

 

Regards

 

Andrew

0 Kudos
applemax82
Beginner
2,653 Views

hi,

 

multifunction_node can receive messages from one input port, and send messages to several output ports.

 

But the number of its output ports should be decided by its template parameter, i.e. the second tuple. Is there a way to set the number of output ports dynamically? Or I should create my customized node class to support this feature?

 

Thanks!

 

Regards

 

Andrew

0 Kudos
SeshaP_Intel
Moderator
2,625 Views

Hi,


>>> how to report error to up-level program when running flow graph?


The flow graph supports exceptions and will throw any exception that is not handled within a node at the callsite of wait_for_all().

Please refer to the below link for topics about exception handling and cancellation.

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph_exception_tips.html


>>> Is there some method to set the size of queue_node?


No, queue_node grows dynamically. You can limit the growth of a queue_node through other mechanisms such as limiter_node. 

Please refer to the below link for more details.

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph_resource_tips.html


>>> How can I control the speed in which demuxer sends audio/video encoded data to decoder, to avoid too many audio/video encoded data are buffered in queue_node?


You can use a limiter_node, a token-based system to control the speed. 

Please refer to the below link for more details. 

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph_resource_tips.html.


>>> Is there a way to set the number of output ports dynamically? Or I should create my customized node class to support this feature?


No, currently the number of ports for a multifunction_node are set at compile-time. 

One work-around is to keep an array of references to successor nodes in the user-provided body of a multifunction_node, similar to the output_ports tuple that is passed as the 2nd argument; 

however, the array would be maintained within the functor – maybed captured by reference. 

You would need to manually manage what’s in that array and do an explicit try_put to the successor node you’d like to send to. So it’s definitely less user-friendly.


Thanks and Regards,

Pendyala Sesha Srinivas


0 Kudos
applemax82
Beginner
2,610 Views

Hi,

 

>>> Is there a way to set the number of output ports dynamically? Or I should create my customized node class to support this feature?

 

>>>>>> No, currently the number of ports for a multifunction_node are set at compile-time. 

>>>>>> One work-around is to keep an array of references to successor nodes in the user-provided body of a multifunction_node, >>>>>> similar to the output_ports tuple that is passed as the 2nd argument; 

>>>>>> however, the array would be maintained within the functor – maybed captured by reference. 

>>>>>> You would need to manually manage what’s in that array and do an explicit try_put to the successor node you’d like to send to. So it’s definitely less user-friendly.

 

If I use the work-around to keep an array of references to successor nodes,  g.wait_for_all() will return after several seconds. How to make flow graph know successor nodes are alive and wait until all tasks are completed? Should I send continue_msg to successor nodes frequently?

 

Thanks!

 

Regards

 

Andrew

0 Kudos
applemax82
Beginner
2,601 Views

Hi,

 

I construct my flow graph using some nodes. But sometimes flow graph halt before all tasks are complete. Is there a way to check the cause why flow graph stops?

 

Thansk!

 

Regards

 

andrew

0 Kudos
applemax82
Beginner
2,583 Views

Hi,

 

I use several video decoders to decode each gop video stream whose task are run by each multifunction_node. Then deocded frames are sent to sequencer_node in order to make decoded video frame in pts order.

 

sequencer_node has the ability to make each item in order, but video frame's pts is not continuous. In order to make frame in pts order, how many frames does sequencer_node know is proper?

 

Below is my sample code about sequencer_node:

oneapi::tbb::flow::sequencer_node<frame_msg_type> frames_ordering(g, [](const frame_msg_type& frame_msg) -> int64_t {
return frame_msg->pts;
});

 

Thanks!

 

Regards

 

Andrew

0 Kudos
SeshaP_Intel
Moderator
2,559 Views

Hi,


>>> How to make flow graph know successor nodes are alive and wait until all tasks are completed? Should I send continue_msg to successor nodes frequently?

Yes, you can use continue_msg to send message to successor nodes frequently.

Please refer to the below link for more details.

https://spec.oneapi.io/versions/latest/elements/oneTBB/source/flow_graph/dependency_flow_graph_example.html


>>> Is there a way to check the cause why flow graph stops?

You can use the try/catch block to check where the flowgraph stops. You need to do a g.wait_for_all() outside the try/catch block.

Please refer to the below link for more details:

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph_waiting_tips.html


>>>In order to make frame in pts order, how many frames does sequencer_node know is proper?

A sequencer_node can reestablish the order of the messages, but it does require us to assign the sequence number. 

To obtain the number from an incoming message, we need to provide a body to the sequencer_node.

You can refer to the ProTBB Textbook, page no:484.


Thanks and Regards,

Pendyala Sesha Srinivas


0 Kudos
applemax82
Beginner
2,546 Views

Hi,

 

I have one problem about the usage of limiter_node. I create three nodes, i.e. source node, decoder node, and image processor node, which correspond to input_node, multifunction_node, and multifunction_node.

 

source node read video packets, and send video packets to its successor, i.e. decoder node. Then decoder node decodes video packets and output raw frames, and send raw frames to image processor node. I intent to limit the number of alive frames between decoder node and image processor node, and insert limiter_node between decoder node and image processor node. image processor node will send continue_msg to limiter_node's decrement port.

 

To verify limiter_node functions correctly, I use std::atomic<int> to count the number of alive frame, where decoder node increases the count and image processor node decrease the count, and add extra sleep code in image processor node:

std::this_thread::sleep_for(std::chrono::milliseconds(some large time))

 

10 is set to limiter_node's threshold. When flow graph is running, I find std::atomic<int> count is increasing. It seems limiter_node doesn't work. Is there something wrong with my usage of oneTBB?

 

Thanks!

 

regards

 

andrew

0 Kudos
applemax82
Beginner
2,504 Views

Hi,

 

There is one problem when throwing exception in flow graph.

 

I have read the following webpages, and try to throw some exception in some nodes. main thread can succeed in catch some exception from some nodes. But the program is pending, when some exception is thrown from some nodes.

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph_exception_tips.html

 

Do you have this kind of problem? Thanks

 

regards

 

andrew

0 Kudos
SeshaP_Intel
Moderator
2,441 Views

Hi,


Could you please share your sample reproducer code and steps to reproduce so that we can investigate more on your issue from our end.


Thanks and Regards,

Pendyala Sesha Srinivas


0 Kudos
SeshaP_Intel
Moderator
2,403 Views

Hi,


We haven't heard back from you. Could you please provide an update on your issue?


Thanks and Regards,

Pendyala Sesha Srinivas


0 Kudos
SeshaP_Intel
Moderator
2,353 Views

Hi,


We have not heard back from you. This thread will no longer be monitored by Intel. If you need further assistance, please post a new question.


Thanks and Regards,

Pendyala Sesha Srinivas


0 Kudos
applemax82
Beginner
2,346 Views

Hi,

 

I think it's better to add the issue to the Intel oneTbb's github, where the developers can help track the problem.

 

Thanks for your help.

 

Regrads

 

Andrew

0 Kudos
Reply