Intel® oneAPI Threading Building Blocks
Ask questions and share information about adding parallelism to your applications when using this threading library.

Enqueue tasks and wait for the workers completion

Estelle_D_
Beginner
1,802 Views

Hello,

 

I have an application in which my application thread spawns an std::thread at the beginning of the program. I define two task_arena and two task_group that are shared by the two master threads of my application. I want the first thread to use the first arena and first group and the second thread to use the second arena and second group.

For the moment my code looks like that:

double mydata[ARRAY_SIZE];

tbb::task_arena* arena1;
tbb::task_arena* arena2;
tbb::task_group* group1;
tbb::task_group* group2;

void run()
{
   arena2->execute( [&] {
      group2->run_and_wait( [&] {
         tbb::parallel_for( /* parallel work on mydata */ );
      });
   });

   // do some other work
}

int main()
{

   arena1 = new tbb::task_arena(4);    // I have 4 cores on my machine
   arena2 = new tbb::task_arena(4);
   group1 = new tbb::task_group;
   group2 = new tbb::task_group;

   std::thread second_thread( run );

   arena1->execute( [&] {
      group1->run_and_wait( [&] {
         tbb::parallel_for( /* parallel work on mydata */ );
      });
   });

   second_thread.join();

   delete arena1;
   delete arena2;
   delete group1;
   delete group2;

   return 0;

}

 

However, I would like the second master thread to not participate in the arena2 computation and directly to go in the other computation part. I therefore thought of using arena2->enqueue instead. But in this case, the second master thread may exit run() function and may reach the second_thread.join() statement and the end of the program before the worker threads completed the execution of tbb::parallel_for.

 

My question is : is there a way for a thread that called arena->enqueue on a task_group to wait for the completion of the task_group? Something similar to the following?

arena->enqueue( [&] { // enqueue the computation
   group->run( [&] {
      tbb::parallel_for ( /* some parallel computation */ );
   });
});

// do some other work

group->wait(); // wait for the worker threads completion

 

Thank you very much for your answer,

Estelle Dirand

0 Kudos
9 Replies
Alexei_K_Intel
Employee
1,802 Views

Hi,

Your idea will work. However, the calling thread will not participate in work related to the arena. To overcome this issue, just call the wait inside the arena:

arena->execute( [&] { // Enter to the arena
    group->wait(); // wait inside the arena
} );

Just curiosity, why do you use dynamic memory for arenas and task_groups? And why do you need specify the number of threads explicitly and not to use the default behavior?

Regards,
Alex

0 Kudos
Estelle_D_
Beginner
1,802 Views

Hi,

Thank you for your answer.

I tried what you told me and sometimes, I have the second thread that waits the completion of the worker threads but sometimes it does not and TBB throws an exception:

terminate called after throwing an instance of 'tbb::missing_wait'
  what():  wait() was not called on the structured_task_group
Aborted

I attach to the post a C++ file to illustrate my problem.

I want the output to be:

First thread wrote mydata
Second thread enqueued tasks
First thread wrote mydata2
Average is 99999.5
Second thread waited for completion

If I launch several times the corresponding executable, I sometimes get the correct output but other times I get this:

First thread wrote mydata
Second thread enqueued tasks
Second thread waited for completion
First thread wrote mydata2
Average is 333.377
terminate called after throwing an instance of 'tbb::missing_wait'
  what():  wait() was not called on the structured_task_group
Aborted

with different values for the average.

Do you have any idea of what could cause this issue?

Finally, to answer your question, I am working on an application where I look at different arena sizes (not only the default behavior) and I use dynamic memory for arenas and groups because they are defined as global variables at the beginning of my program and I want to initialize TBB later in the program.

Regards,

Estelle

0 Kudos
Alexei_K_Intel
Employee
1,802 Views

I missed that the work is enqueued into the arena. It leads to a race between a moment when enqueued work is processed and the wait method is called. I.e. the thread can call the wait method before the task is added to the task group and it will not wait until the task will be run by enqueued work. To overcome this issue you can use the execute method instead of the enqueue method. You will still have "asynchronous" semantics but the execute method will guarantee that the task is added (but not executed) before the wait method is called.

Regards,
Alex

0 Kudos
Estelle_D_
Beginner
1,802 Views

I tried your approach and it works fine but in this situation, the thread that calls the execute method will participate to the execution.

Is there a way to exclude the calling thread from a task_arena when using an execute method?

In particular, I saw that arenas can be constructed with

task_arena(int max_concurrency = automatic, unsigned reserved_for_masters = 1)

and I tried to put the reserved_for_masters value to 0. In this case I get the following warning:

TBB Warning: The number of workers is currently limited to 3. The request for 4 workers is ignored. Further requests for more workers will be silently ignored until the limit changes.

and my calling thread still seems to participate in the task execution.

Do you know why I cannot use reserved_for_masters = 0 ?

Regards,

Estelle

 

0 Kudos
Alexei_K_Intel
Employee
1,802 Views

Hi,

You can specify "reserved_for_master" to be 0. But at the moment arena creation, TBB is already initialized with 3 workers.

It looks like I do not understand what you are trying to do. Correct me if I wrong:

  1. You want to create work within an arena by some external thread (e.g main/master) and do not wait for the completion.
    arena->execute( [&] { // We need to enter the arena to guarantee that work is initiated
       group->run( [&] { // Initiate the work but do not wait its completion
          tbb::parallel_for ( /* some parallel computation */ );
       });
    });
    

     

  2. Then you want to do something different
    // do some other work
  3. After some other work is finished you want to wait for the arena's work completion
    arena->execute( [&] { // Enter to the arena
        group->wait(); // wait inside the arena
    } );

And I do not understand what do you mean with "and my calling thread still seems to participate in the task execution"? Is not it the desired behavior? Or how you want to wait for the work completion but you do not want master thread to participate in work processing? What are you trying to do with "reserved_for_master=0"? Why do you need/want to manage number of threads explicitly?

Regards,
Alex

 

0 Kudos
Estelle_D_
Beginner
1,802 Views

Hi,

 

The second thread that I spawn at the beginning of the application is actually a monitoring thread that waits for instructions of the application main thread to launch tasks in the second arena.

I would like the monitoring thread to not participate in the group->run part because I want it to be available to get new instructions almost immediately and I want it to participate in the group->wait in order to wait for the arena's work completion before going further.

I tried to ensure that my second thread does not participate in the group->run part by setting reserved_for_master=0 but I realized that it would also prevent it to participate in the group->wait...

 

So my question is, is there a way to exclude the second thread from the group->run part but to guarantee that it participates to group->wait? Perhaps it is just not possible but I wanted to ask in the case there is something I do not know with TBB.

 

I hope I was more clear on my question. Thank you very much for your time.

 

Regards,

Estelle

 

0 Kudos
Lucian_T_
Beginner
1,802 Views

Hi Estelle

I'm not sure if this is 100% compatible with your problem, but I had a similar problem: one thread initialize multiple arenas, without participating into them. I solved my problem by deriving from tbb::task_arena instead of using it with a task_group. For some reasons, I had to make my main thread to do some work in that arena so that everything starts moving; with my solution, the main thread doesn't care about the tasks "enqued" into arenas. Here is a sample code of my arena class:

struct TaskArenaEx : tbb::task_arena
{
    //! Getter for the context of the tasks in this arena
    tbb::task_group_context* GetContext() const {
        return my_context;  // Protected in tbb::task_arena_base
    }

    //! Wait for all the tasks in this arena to finish
    //! and then terminate the arena
    void Shutdown(bool cancelTasks = true) {
        if ( cancelTasks )
        {
            // Cancel all the tasks in our context
            my_context->cancel_group_execution();
        }

        // Wait for all the work in this arena to complete
        internal_wait();

        // Terminate the arena
        terminate();
    }
};

In my case, I wanted at the app shutdown to quickly finish the operations in the arena and then to exit.

To add work into the arena, you can simply use the enqueue function of the arena:

arena.enqueue(task, tbbPrio);

The main thread will not take part into the work, until the Shutdown. You can mannually make it a part of one task_arena later, by making it wait for different tasks.

Hope this helps,

LucTeo

0 Kudos
Alexei_K_Intel
Employee
1,802 Views

Hi Estelle,

Does not the proposed solution suite your needs?

Katranov, Alexei wrote:

  1. You want to create work within an arena by some external thread (e.g main/master) and do not wait for the completion.

     

    arena->execute( [&] { // We need to enter the arena to guarantee that work is initiated
       group->run( [&] { // Initiate the work but do not wait its completion
          tbb::parallel_for ( /* some parallel computation */ );
       });
    });
    

     

  2. Then you want to do something different
    // do some other work
  3. After some other work is finished you want to wait for the arena's work completion
    arena->execute( [&] { // Enter to the arena
        group->wait(); // wait inside the arena
    } );

Regards,
Alex

 

0 Kudos
Alexei_K_Intel
Employee
1,802 Views

Hi Lucian,

It is not recommended to use the "internal_wait" method because its semantics is not documented and can be changed in future release (or even the interface can be removed because it is not exposed with documented API). To wait all tasks within arena it is better to use explicit task_group object.

Your implementation also uses the "my_context" member that also can be reworked/renamed/removed in future releases.

Regards,
Alex

0 Kudos
Reply