problems with tbb::task_scheduler_init

[ugavnholt]

I'm building an application where a few cores are dedicated to an iocp, and the rest of the cores (on a 8 core system) are meant to be assigned the task scheduler in tbb, because of thisI need set the maximum number of workers, the task scheduler thread pool will start to the number of cores available-the number of iocp threads.

After watching the number of threads spawned by task_scheduler_init, it appears as it is ignoring the number of threads specified as a parameter, no matter what I set it to.

I also had a problem with the threads that I spawned outside of tbb, in contradiction to what is written in the documentation, I had to manually init the task scheduler in each thread that I start, or the stack would be corrupted - is this somehow related? (task_scheduler_init is called before these threads are spawned, in the main thread)

I've put the call to task_scheduler_init to the very first line of code in main(), and I'm running tbb version 30_20110427oss, and I'm using windows task list to monitor the number of threads active in my program.

Anyone have some pointers that can help me out here?

[RafSchietekat]

You don't "call" this, you instantiate it, e.g., "tbb::task_scheduler_init l_anonymous(tbb::task_scheduler_init::default_num_threads()-1);". If you leave out the instance's name, perhaps assuming that task_scheduler_init is a function, the compiler constructs and immediately destroys a temporary object, whereas the instance should persist throughout the thread's involvement with TBB.

[ugavnholt]

Hi Raf,

Thank you for your answer, I create the object the following way

tbb::task_scheduler_init *task_init = new tbb::task_scheduler_init(nWorkers);

and destroy it with delete before the program terminates, I'm sorry if its my grammar that is a bit off.

[RafSchietekat]

Do you mean that this is what failed, or that you successfully modified the solution I suggested?

I don't suppose it's illegal to allocate a task_scheduler_init instance on the heap (otherwise the class-specific new/delete operators could have been made unavailable), but you may need to be careful to destroy it from the same thread that constructed it (the Reference Manual is not explicit about that but does seem to imply it), which happens automatically... with an automatic variable (if the program framework does not prevent such an arrangement).

[ugavnholt]

No what I described was how the solution was working when the problem started.

The main thread is the one both creating and destroying the task_scheduler_init object, and I tried the usual approach with an automatic variable, unfortunately with the same result. The reason for creating it on the heap is that I want the number of threads to be configurable from the registry,once I get this problem sorted.

According to the documentation it is no longer needed to create the task_scheduler_init object, so I was wondering whether it gets created as an automatic variable thorugh the library, thus ignoring the number of threads option?

Thank you for your help so far

[RafSchietekat]

Sorry, I'm fresh out of ideas that aren't just speculative.

(Added 2011-08-10) Let's speculate anyway... Have you watched the test programs that vary the number of threads employed to see elapsed-time evidence that it works? Have you inspected those programs and tried to do the same in your program? Another possibility is that the first task_scheduler_init in the program doesn't get to set a global maximum, so any additional thread using TBB has to be careful to have its own task_scheduler_init to override default_num_threads(), and maybe that is what is going on here; a solution for that would be to inspect all explicit thread creations, and/or use your framework's means of executing code on entry and exit of implicitly created threads, and to add appropriate task_scheduler_init instances; a useful addition to TBB might then be a global limit on number of threads used.

[Gennadi_Sushko · Accepted solution]

Hello.

> After watching the number of threads spawned by task_scheduler_init, it appears as it is ignoring the number of threads specified as a parameter, no matter what I set it to.

I have found same problem while using tbb 3.x (didn't have this problem while using 2.x). I was initialisingtask_scheduler_init for 4 threads (3+main) on 8-cores system(i7 920 SMT) and found additional threads performing tbb tasks. I fixed this problem on my system by commenting line 105 in src/tbb/market.cpp. It raises the variable max_num_workers from input value (parameter of market::global_market function) to governor::default_num_threads().

[ugavnholt]

Hi Gennadi,

Thank you for your suggestion, that did it for me, its a work-around, but it will do the trick

[Kirill_R_Intel]

Ugavnholt,

Is it possible to provide a reproducerof your problem? We can make the thread private. This looks like an issue in TBB that we would like to investigate and address.

Gennady,

Thanks for your investigation and workaround.

Regards,
Kirill

[ugavnholt]

hi Kirill,

I'm a bit hung up on a project I need to finish before I go on vacation, when I come back, i will be more than happy to try to reproduce the problem, I'll get back in touch

/Uffe

[Alexey-Kukanov]

Let me explain what you saw, at least to some extent.
Since TBB 3.0, we changed the behavior of the scheduler to isolate work (tasks)supplied by different application threads. So now each application thread has a separate place to share tasks, andit will be served by as manyworker threads as it requested via task_scheduler_init.
The behavior of old TBB versions when the first thread to initalize TBB set the total number of available workers has also changed; now TBB will give the first thread the number of workers it requested, but may create more workers for other application threads until hardware is fully subscribed. I believe this is what you observe. The workaround that works for you basically reverts the behavior back to TBB 2.x.

What concerns me is the stack corruption you reported. It seems like a bug. The expected behavior is that if an application thread does not explicitly create task_scheduler_init, TBB should initialize its internal structures automatically at the first time the app thread uses the scheduler. More details, or ideally a reproducer, are necessary to understand what's going on there.