topic kmp_set_affinity_mask_proc in Intel® Moderncode for Parallel Architectures

kmp_set_affinity(4,&new_omp_mask) return error 22

Ying_H_Intel — Thu, 14 Dec 2017 03:37:55 GMT

one of our customer reported, they had tried do the omp affinity bind in pthread code on xeon skx server 32core. intel libiomp5.so 2018 version,

static void test(int thread_num, int core_num) {

int num = omp_get_num_threads();
omp_set_num_threads(thread_num);

#pragma omp parallel default (shared)
 {
  int ompTid = omp_get_thread_num();
 

  kmp_affinity_mask_t new_omp_mask;
  kmp_create_affinity_mask(&new_omp_mask);
  kmp_set_affinity_mask_proc(ompTid, &new_omp_mask);
  kmp_set_affinity_mask_proc(ompTid + core_num, &new_omp_mask);
 
int ret = kmp_set_affinity(&new_omp_mask)

  if(ret != 0 ) {
   printf("Error::kmp_set_affinity() for thread %d,  ret %d \n", ompTid, ret);
  }
}

}

and it give the error Error: KMP_Set_affnity(8, ret 22)

I check intel Compiler , haven't found the error message about 22, could you please tell what is exact mean of error 22. we haven't test code, just get the error message, so any suggest to debug the issue are appreciated.

Thanks

Ying

kmp_set_affinity_mask_proc

jimdempseyatthecove — Thu, 14 Dec 2017 16:37:24 GMT

kmp_set_affinity_mask_proc(int LogicalProcessorNumberBitToSet, kmp_affinity_mask_t* mask);

Your line 14 is hard to interpret what you intentions are - please explain what you are trying to do.

Note, If the last thread team member number of the parallel region is the (near) the highest logical processor number of the system (available to your process), then "+ core_num" will likely index beyond the bits available to the affinity bit mask. This may corrupt memory or set unavailable logical processor bits in reserved bits of the mask.

Jim Dempsey

Hi Jim,

Ying_H_Intel — Mon, 18 Dec 2017 02:15:44 GMT

Hi Jim,
thank you for your checking.

I'm not sure what purpose of the line 14, just guess, the user may want to handle the module (like ompTid % core_num). we get some new information by export KMP_AFFINITY=verbose. it shows that

OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63}
OMP: Info #156: KMP_AFFINITY: 32 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 16 cores/pkg x 2 threads/core (16 total cores)
OMP: Info #242: KMP_AFFINITY: pid 1390 tid 1390 thread 0 bound to OS proc set {16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63}
OMP: Info #242: KMP_AFFINITY: pid 1390 tid 1431 thread 1 bound to OS proc set {16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63}
OMP: Info #242: KMP_AFFINITY: pid 1390 tid 1432 thread 2 bound to OS proc set
KMP_AFFINITY: pid 1390 tid 1472 thread 38 bound to OS proc set {16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63}
Debug: thread_num 16, core_num 32, ompTid 0, ompTid + core_num 32
Error: kmp_set_affinity(0, &new_omp_mask) ret 22
OMP: Info #242: KMP_AFFINITY: pid 1390 tid 1467 thread 34 bound to OS proc set

Thus the problem seems that the OS proc ID is not from 1 - 32 core, but 16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63} . while in the kmp_set_affinity_mask_proc, we suppose int LogicalProcessorNumberBitToSet is from omp 0 to os proc 15.

Maybe to use the system affinity will workaround the problem. ask user to try and will feedback if it works

CPU_ZERO(&new_mask);
CPU_SET(ompTid, &new_mask);
if (sched_getaffinity(0, sizeof(was_mask), &was_mask)
== -1) {printf("Error: sched_getaffinity(%d, sizeof(was_mask),
&was_mask)\n", ompTid);
}
if (sched_setaffinity(0, sizeof(new_mask), &new_mask)
== -1) {printf("Error: sched_setaffinity(%d, sizeof(new_mask),
&new_mask)\n", ompTid);
}
printf("tid=%d new_mask=%08X was_mask=%08X\n", ompTid,
*(unsigned int*)(&new_mask), *(unsigned int*)(&was_mask));
}
Thanks
Ying

The 2nd argument to shed

jimdempseyatthecove — Mon, 18 Dec 2017 17:32:08 GMT

The 2nd argument to shed_setaffinity and sched_getaffinity is the cpusetsize (IOW number of valid bits in the mask). You are passing sizeof(cpu_set_t) which is the number of bytes in the mask. Note, you cannot use number of bytes * 8 as the valid bit field will be .LE. that number of bits.

Also note, depending on your O/S build, the header bytesize for cpu_set_t could potentially be incorrect. The default number of bits is 1024, your system values may vary from this.

See: https://linux.die.net/man/3/cpu_set

Jim Dempsey