Hi,
I wrote this short program to understand how memory access analysis and NUMA works. I am running a Dual-Xeon E5-2660v4. I am surprised to see that the QPI is heavily used despite the fact that my program is using the first touch policy. Can anyone explain me why there is so much traffic here?
int main() {
int n = 1000000000;
double* a = new double;
double* b = new double;
double* c = new double;
#pragma omp parallel for
for (int k = 0; k < n; ++k) {
a = 0.0;
b = 0.0;
c = 0.0;
}
#pragma omp parallel for
for (int k = 0; k < n; ++k) {
a = b + c;
}
delete[] c;
delete[] b;
delete[] a;
return 0;
}
Link Copied
Hi,
Here is the result with "export KMP_AFFINITY=compact". I have changed the code so we can see side by side the usage of first touch policy and what happen if we don't use it. Here is the code:
#include <chrono>
#include <cstdio>
#include <vector>
int main() {
int n = 1000000000;
double* a = new double;
double* b = new double;
double* c = new double;
#pragma omp parallel for
for (int k = 0; k < n; ++k) {
a = 0.0;
b = 0.0;
c = 0.0;
}
auto begin = std::chrono::high_resolution_clock::now();
#pragma omp parallel for
for (int k = 0; k < n; ++k) {
a = b + c;
}
auto end = std::chrono::high_resolution_clock::now();
double time =
1.0e-9 *
std::chrono::duration_cast<std::chrono::nanoseconds>(end - begin).count();
std::printf(" With first touch policy: %7.2f\n", time);
delete[] c;
delete[] b;
delete[] a;
std::vector<double> va(n, 0.0);
std::vector<double> vb(n, 0.0);
std::vector<double> vc(n, 0.0);
begin = std::chrono::high_resolution_clock::now();
#pragma omp parallel for
for (int k = 0; k < n; ++k) {
va = vb + vc;
}
end = std::chrono::high_resolution_clock::now();
time =
1.0e-9 *
std::chrono::duration_cast<std::chrono::nanoseconds>(end - begin).count();
std::printf("Without first touch policy: %7.2f\n", time);
return 0;
}
and the result of the memory analysis. We can see a QPI bandwidth around 9 GB/s in the first computational phase which is using the first touch policy. I agree that it is way lower than what we have in the second phase. But what is it ?
I am puzzled because I thought that I could use this QPI graph to check for NUMA problems, but it seems that we can't use it for that.
Hi John. Here is the annotated timeline. I have 32 GB of RAM on this computer (16 GB on each socket) and as the 3 vectors use 24 GB of RAM, that explains the fact that some of the memory commited for std::vector are not commited on the original bank. It explains the QPI communication during the initialization of the std::vector.
