Scheduler internals questions

[AJ13]

After a break for the holidays I'm back and ready for some more scheduler investigations. Thanks to responses from everyone last time I better understand some parts of the code. I would like to ask for some help understanding the role that some components play in the scheduler. I'm looking to understand the big picture of how these pieces fit. What is an Arena? Task proxy? Mailbox? Thanks.

[robert-reed]

Quoting - AJ
I would like to ask for some help understanding the role that some components play in the scheduler. I'm looking to understand the big picture of how these pieces fit. What is an Arena? Task proxy? Mailbox? Thanks.

I'll take a stab at getting this dialog started. In TBB TheArena is the data structure that keeps track of tasks and threads. It's created the first time task_scheduler_init::initialize is called with a non-deferred number of threads, when that function calls Arena::allocate_arena() and comprises three sections. The middle section is called the ArenaPrefix and contains a collection of miscellaneous data used for managing scheduling such as worker_list, the collection of worker threads attached to this arena.

After the prefix is an array of slots which is where the scheduled TBB tasks are hung. Arena allocation creates twice as many slots as available threads. Each slotmaintains a TaskPool which is a possiblearray of tasks with a couple extra fields to track status. allocate_area() reserves a slot for each worker thread and marks the rest as unused.

On the other side of the ArenaPrefix is an array of mailboxes that are indexed in the opposite direction from the slots and are used for passing tasks between threads to sort out affinities, which is where task proxies come into play.

[irisshinra308]

Hello~

I have an additional question here.

I have read the wait_for_all() in task.cpp and find that the only way to push a task into the mailbox of threads is calling the spawn() function, am I right?

Thanks for reply!!

Dennis

[Alexey-Kukanov]

Quoting - irisshinra308

I have read the wait_for_all() in task.cpp and find that the only way to push a task into the mailbox of threads is calling the spawn() function, am I right?

Right. When a task is spawned, in case it has a non-zero affinity_id, it is also put into the mailbox corresponding to that id.

[irisshinra308]

Thanks for replying!This really helps me.

I know the set_affinity() is used to set the task's affinity_id.

However, I still confused about what the purpose of note_affinity() is?

The reference manual only depicts when note_affinity() would be invoked.

Thanks again for your patience and kindness

Dennis

[Alexey-Kukanov]

Quoting - irisshinra308

However, I still confused about what the purpose of note_affinity() is?

As I recently wrote in another forum thread, task affinity mechanism in TBB does not guarantee that the affinitized task will be executed on the thread it was mailed to. In other words, affinity_id is a hint, not an order. And the note_affinity() callback is there to help you noticing when the hint was not followed, and make adjustments if necessary.

[osarood]

Quoting - Robert Reed (Intel)

Quoting - AJ
I would like to ask for some help understanding the role that some components play in the scheduler. I'm looking to understand the big picture of how these pieces fit. What is an Arena? Task proxy? Mailbox? Thanks.

I'll take a stab at getting this dialog started. In TBB TheArena is the data structure that keeps track of tasks and threads. It's created the first time task_scheduler_init::initialize is called with a non-deferred number of threads, when that function calls Arena::allocate_arena() and comprises three sections. The middle section is called the ArenaPrefix and contains a collection of miscellaneous data used for managing scheduling such as worker_list, the collection of worker threads attached to this arena.

After the prefix is an array of slots which is where the scheduled TBB tasks are hung. Arena allocation creates twice as many slots as available threads. Each slotmaintains a TaskPool which is a possiblearray of tasks with a couple extra fields to track status. allocate_area() reserves a slot for each worker thread and marks the rest as unused.

On the other side of the ArenaPrefix is an array of mailboxes that are indexed in the opposite direction from the slots and are used for passing tasks between threads to sort out affinities, which is where task proxies come into play.

Hello, I am new to TBB
I am confused that why do we have 2*num_threads slots? Why dont we have slots=num_threads?

osman

[Alexey-Kukanov]

Quoting - osarood

Hello, I am new to TBB
I am confused that why do we have 2*num_threads slots? Why dont we have slots=num_threads?

osman

There are more arena slots than HW threads in order to allow multiple masters (i.e. threads not created by TBB itself) to share their work. I.e. if an application already have a couple of threads to perform different functions, and both of those can benefit from data driven parallelism, both could use TBB algorithms, at the same or at different times.

[Charles_Tucker]

Just to make sure I understand - given P processors and a default constructued initializer, there should be 1 master thread and P-1 workers spawned. In addition, up to P-1 other master threads can join the Arena. In that case, isn't the machine over-subscribed by a factor of 2 at this point? And what happens if a P+1th master thread attempts to join the arena?

[Alexey-Kukanov]

Quoting - Charles Tucker

Just to make sure I understand - given P processors and a default constructued initializer, there should be 1 master thread and P-1 workers spawned. In addition, up to P-1 other master threads can join the Arena. In that case, isn't the machine over-subscribed by a factor of 2 at this point? And what happens if a P+1th master thread attempts to join the arena?

Since master threads are external to TBB, reserving space for more than 1 master does not cause oversubscription by itself.

If more masters want to join arena simultaneously than there are slots for them, those who are late will not join, and so their tasks won't get stolen. They might join the arena later however if some slots got freed; if I am not mistaken each task spawn checks for a slot in the arena, if not yet there.