Hello!
The problem is reproducible on Windows machines with big number (4 or more) of NUMA nodes.
My testing machine has 4 Intel(R) Xeon(R) CPU E7-8860 v3 with HyperThreading, so it has 4 Processor Groups (NUMA nodes) with 32 cores in each.
I ran 4 instances of parallel TBB application (each starts 20 TBB threads).
I would expect that all the applications will be distributed evenly on all NUMA nodes.
I even tried to do it explicitly (.bat file):
start /node 0 .\Release\tbb_affinity_test.exe
start /node 1 .\Release\tbb_affinity_test.exe
start /node 2 .\Release\tbb_affinity_test.exe
start /node 3 .\Release\tbb_affinity_test.exe
The result is not as I expected. Main thread of each instance is located on correct NUMA node indeed. But almost all the other threads are on 3rd (the last one) NUMA node.
So 3rd CPU is oversubscribed while all the other CPUs are almost idle. It is proven by Task Manager as well as app's output (see below).
What can I do to evenly distribute workload?
I would expect that TBB threads should first be assigned to the same node as the main thread, how can I achieve it?
I ran my test on Windows 10 Enterprise, Version 20H2
Using TBB version 2021.5.2 (the same problem was found on older version of TBB as well)
Example was compiled on Microsoft Visual Studio Professional 2015, Version 14.0.25431.01 Update 3
The machine has 4 (four) Intel(R) Xeon(R) CPU E7-8860 v3 with HyperThreading, so it has 4 Processor Groups (NUMA nodes) with 32 cores in each.
The sample application is below:
#include <stdio.h>
#include <oneapi/tbb/parallel_for.h>
#include <oneapi/tbb/global_control.h>
#include <oneapi/tbb/enumerable_thread_specific.h>
int main()
{
oneapi::tbb::global_control global_limit(oneapi::tbb::global_control::max_allowed_parallelism, 20);
oneapi::tbb::enumerable_thread_specific< std::pair<int, int> > tls;
oneapi::tbb::enumerable_thread_specific< double > tls_dummy;
oneapi::tbb::parallel_for(0, 1000, [&](int i)
{
// dummy work
double & b = tls_dummy.local();
for(int k = 0; k < 10000000; ++ k)
b += pow(i * k, 1./3.);
PROCESSOR_NUMBER proc;
if (GetThreadIdealProcessorEx(GetCurrentThread(), &proc))
{
tls.local() = std::pair<int, int>(proc.Group, proc.Number);
}
});
for (const auto it : tls)
printf("Group: %d, Number %d\n", it.first, it.second);
system("pause");
return 0;
}
The output of 4 simultaneous run of the application is below:
Task Manager:
Thank you,
Maksim Popov
Link Copied
