flow_graph large data handling/passing - best practice ?

[nyue]

I am considering splitting up the various task I currently have with geometry processing of point-cloud/lidar-scan data and rewrite/wrap them as tasks and connect them up via flow_graph.

As the dataset will not be small, I plan to pass them between the flow_graph nodes via pointer/smart-pointer.

If the node modify the data, the node is responsible for duplicating (and registering it with some singleton data manager of some sort which I have to figure out and implement) the data before modifying and passing them on downstream.

Are there threading/concurrency aspect that I should be mindful of ? Is the above approach feasible ?

I believe these are common problems that many people have encountered in the past, are there any write up about such things ?

Kind regards

[SeshaP_Intel · Accepted solution]

Hi,

Thank you for posting in Intel Communities.

The flow graph passes data structures from node to node. The messages can be primitive types, objects, and pointers. You can use pointers as messages.

You can use buffering nodes like buffer_node or write_once_node which forward the message to successor. Please refer to the below links for more details.

https://spec.oneapi.io/versions/latest/elements/oneTBB/source/flow_graph/buffer_node_cls.html

https://spec.oneapi.io/versions/latest/elements/oneTBB/source/flow_graph/write_once_node_cls.html

You can use the concurrency limits on nodes to limit resource consumption. Please refer to the below link for more details.

https://oneapi-src.github.io/oneTBB/main/tbb_userguide/Flow_Graph_resource_tips.html

Thanks and Regards,

Pendyala Sesha Srinivas

[SeshaP_Intel]

Hi,

Thanks for accepting our solution. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks and Regards,

Pendyala Sesha Srinivas