Link Copied
The following code:
#include <iostream>
#include <map>
#include <set>
#include <thread>
#include <tbb/tbb.h>
const int P = 12; //12 threads all together will be used
thread_local int my_tid = -1;
std::vector<std::set<std::string>> tid_regions(3*P);
//tbb::atomic<int> next_tid;
std::atomic<int> next_tid = {0};
void noteParticipation(const std::string& name) {
if (my_tid == -1) {
//my_tid = next_tid.fetch_and_increment();
std::atomic_fetch_add(&next_tid, 1);
my_tid = next_tid;
}
tid_regions[my_tid].insert(name);
}
void dump_participation() {
int end = next_tid;
std::map<std::string, int> m;
for (int i = 0; i < end; ++i) {
for (auto n : tid_regions[i]) {
m[n] += 1;
}
}
for (auto& kv : m) {
std::cout << kv.second << " working threads participated in " << kv.first << std::endl;
}
}
void doWork(const std::string& name, double seconds) {
noteParticipation(name);
tbb::tick_count t0 = tbb::tick_count::now();
while ((tbb::tick_count::now() - t0).seconds() < seconds);
}
int main() {
int N = 10*P;
std::cout << "There are " << tbb::info::default_concurrency() << " logical cores." << std::endl;
tbb::global_control gc(tbb::global_control::max_allowed_parallelism, P + 1); //one more thread
tbb::task_arena arenaA(6); // Create the custom task_arena A with 6 threads
tbb::task_arena arenaB(6); // Create the custom task_arena B with 6 threads
tbb::parallel_invoke(
[&]{
arenaA.execute([&]{
std::cout << "arenaA max concurrency = " << tbb::this_task_arena::max_concurrency() << std::endl;
// long work A here
tbb::parallel_for(0, N, [](int) { doWork("arenaA pfor", 0.01); });
});
},
[&]{
arenaB.execute([&]{
std::cout << "arenaB max concurrency = " << tbb::this_task_arena::max_concurrency() << std::endl;
// long work B here
tbb::parallel_for(0, N, [](int) { doWork("arenaB pfor", 0.01); });
});
}
);
dump_participation();
return 0;
}
can be compiled with
icpx -g -O2 invoke-forum.cpp -o invoke-forum.x -tbbIt produces this output
bash-4.4$ ./invoke-forum.x
There are 224 logical cores.
arenaB max concurrency = 6
arenaA max concurrency = 6
5 working threads participated in arenaA pfor
6 working threads participated in arenaB pfor
bash-4.4$This specific system (Sapphire Rapids) had a lot of logical cores, but I used a subset of 12 cores. One of the arenas also used main thread besides working threads. After that, you could run Vtune to debug performance:
vtune -collect hotspots -result-dir r001-forum-hs -- ./invoke-forum.xVtune should produce something similar to the following
Also, if you'd like more like OpenMP performance you could try TBB static_partitioner in the above sample tbb::parallel_for(s).
You could also use other TBB parallel algorithms, e.g., parallel_for_each, parallel_pipeline, flow graph and even lower level task_groups to setup parallel execution. The advantage of parallel_invoke is that it is simple and expresses logic of what you're trying to do (as I understand it).
BTW, even a simpler 2 arenas with 2 pfors snippet without parallel_invoke, would not run sequentially. I will expand on this more (and give you some references) in the next post here but this should help you to get started.
I realized that I misunderstood your first post. You can setup a parallel execution with task groups too. I'd just remove lines 18 and 20 and leave simply line 19: "dummyGroup.wait()" in your initial example. With your example, with the task group responsible for the parallel execution, Vtune shows more "spin and overhead" (comparing picture below vs Vtune results above with parallel_invoke). However, the parallel_for body (I used) is rather contrived example. The performance (e.g., Vtune) studies need further investigation. Said that, parallel_invoke is a high-level function that executes the provided tasks in arenas in parallel and it was designed for that. The task_group is a more flexible and general-purpose method for managing tasks. But at the end, the performance depends on your specific platform and specific workload. Could you share your Vtune data or at least provide more specifics about your workload?
