Waiting for multiple communicating graphs executed in different arenas

[hawkmooneternal]

Hello Community,

A) Problem Description:

I'm trying to build an application using multiple flow_graphs which are communicating with each other via try_put messages. Every graph should be executed in a separate arena, mapped to a specific NUMA node (that is only specific cores belonging to the same NUMA node should work on this graph). Also it is not possible to know which graph finishes last at compile time. As it is possible that some graph is idle for a short period of time, until it receives another try_put_message from another graph, graph.wait_for_all is possibly returning before the end of execution.

B) What I've tried so far:

1. I created the arenas like

arenas->initialize(tbb::task_arena::constraints(numaIDs,4));

2. I call graphs.reset() inside those arena to reset set the graph's arena to this arena.

3. I start graph execution like this:

arenas->execute([&task_groups,i,descr] {
	task_groups.run([i,descr]{
		descr.start_nodes->try_put(continue_msg());
	});
});

4. //TODO: Wait until all graphs have finished

I messed with task_group.wait() and graph.wait_for_all() but didn't succeed.

All of my attempts so far either produced deadlocks or did not block until all graphs had finished (resulting in segfaults).

C) So my questions are:

1. Is there a way to wait until all nodes of all graphs are executed?

2. Is there a way to map graph-nodes to specific cores / NUMA-node other then representing the graph by multiple subgraphs communicating with each other?

3. If I would use an atomic counter in the node bodies and also check for reaching the number of nodes in the bodies (which would basically check if the current node is the last one of all graphs): Is there a way to use this to signal a waiting thread but still make it possible for the waiting (master)-thread to join the arena/graph execution?

I know that this is a very specific issue but any help would be highly appreciated.

D) Here an example and some more details of my application that may or may not help:

I have a graph representing the dependencies between a source vector and a destination vector in sparse-matrix-vector-multiplications.

Let's say I want to execute the upper part of the graph on NUMA node 0 and the lower part on NUMA node 1. There are dependencies between the two parts of the graph.

I devide the graph into two subgraphs (TBB flow_graph with continue_nodes) (upper and lower) with edges for dependencies within the subgraph and explicit try_put(continue_message) calls after the execution of a node body for all dependencies between the subgraphs.

I start the graphs with a call to one broadcast_node for each subgraph which puts forward a continue_message to all nodes of the first level of this graph  (which is the the first multiplication).

E) Best Regards

[hawkmooneternal]

I finally found a solution for my problem using nested graphs:

I created another graph G with one broadcast_node B as root with edges to the node N_i for each graph g_i I want to execute and wait for. So inside these nodes I call g_i.try_put() on the corresponding start node of the graph and after that g_i.wait_for_all(). Finally after calling B.try_put() I call G.wait_for_all() to wait for all graphs g_i to finish.

To prevent the g_i.wait_for_all() from returning during idle time (waiting for a graph g_j to send a try_put_message) I call increment_wait_count() on g_i for everyone of its nodes, and each node calls decrement_wait_count() when finishing the work. This way, g_i.wait_for_all() will not return before all of its nodes are executed.

Best regards