Link Copied
Hello, Chris,
Thanks for your comments and patch, but we are not using limiter_node. Unfortunately there we don't have simple app to reproduce the issue. When our app was compiled with -DTBB_DEPRECATED_FLOW_ENQUEUE hanging issue has disappeared.
Right now we're suspecting that problem is with our ill-formed graph and few questions arise about correctness of flow graph usage. Let me first explain how flowgraph is used inside app... In mentioned app flow graph is used as mechanism to drive dependent tasks of rendering engine. All data dependencies and execution order between computation(CPU only) and rendering tasks(CPU + GPU) are hand-wired in graphical editor, which generates source file with nodes and edge construction, we then implement nodes bodies where needed.
Ofcourse rendering tasks could not be executed inside TBB worker threads because multithreaded OpenGL usage is prohibited (without any wrappers, GL context switches and call-deffering). So actual rendering tasks were executed in separate dedicated render thread, by means of message passing through concurrent_queue from wrapping-render-task (body of function node executed in TBB working threads) to dedicated thread. Each actual render task (inside render thread) has pointer(s) to queue_node, which have no predecessor, but connected to dependent nodes (tasks or join\broadcast). When actual render task has executed in dedicated rendering thread, it outputs its results to queue_nodes and execution of dependent nodes proceeds as expected. <b>We now suspect that those "hanging in air" queue_nodes which have no predecessors may result in ill formed graph which hangs 4.2.<b>
Final node (task EndFrame) in graph passes 'restart' message to dedicated render thread which leads render thread to:
1) swaps buffers (GL)
2) call wait_for_all on graph to ensure it finishes (which never resulted in rendering thread beign executing "computation tasks", because last has already finished)
3) puts continue_msg to first node - BeginFrame (do {} while ( ..->try_put(..) )
4) wait for messages that render task can be executed
Old implementation of described system used barriers to ensure that render task wrapping body (executed in function node) will not finish until actual render task finishes its execution in dedicated rendering thread and then copied output(s) from it, so no "hanging in air" queue nodes were used. But that implementation resulted in TBB worker threads doing less usefull work and poorer engine performance. So described above implementation was developed.
Below images with simplest possible drawn in editor graph and its tbb implementation are present for clarity.
What is hand drawn in editor: (Also attached in minimal_src.png)
All lines (connections) between nodes represent make_edge operation. All have concurency limit set to 1 and all default policies.
BeginFrame, EndFrame, BurnCPU_1, BurnCPU_2 are bodies executed in function nodes and implemented by developer.
GPUClearScreen is simple wrapper body, which actually passes message to render thread with information about rendering task requiered to execute.
tbb_continue_msg, join_node, queue_node and broadcast_node correspond to tbb::continue_msg, tbb::join_node, tbb::queue_node and tbb::broadcast_node respectively.
What is actually gets generated from above graph: (Also attached in minimal_dst.png)
Above is real picture of what is going on for hand-drawn graph. Grey rectangle covers code executed in TBB working threads. Violet rectangle covers code executed in
dedicated rendering thread. Rendering thread is simple, it infinitely spins and tries to pop (try_pop) from concurrent_queue (spining was cheaper than conditional waiting, in our case) and execute rendering task, or swap buffers.
As it is seen from picture GPUClearScreen has no successors, and is a simple function node, with body pushing message to renderer's concurrent_queue. Actual rendering code is executed in rendering thread when it receives message from concurrent_queue, and real output result of GPUActualClearScreen is put to queue_node.
It is important to note again, that queue_node has no predecessors connected to it, and results are hand-pushed (actually done by generated code) to it.
-------------------
Sorry for quite long message and cumbersome explanations.
Are queue_nodes without predecessors cause of graph hanging, or it could be something else. Any ideas how hanging queue_nodes problem could be solved with minimal stalling?
I suspect that it would be hard to understand without any code, so if needed i`ll start working on minimalistic as possible version of engine which will be easy to build and debug.
Regards, Max.
