KMP_AFFINITY

[Hongwei_Zhou]

Hi,

Does anyone know I can internally change KMP_AFFINITY in the sub-process invoked from my program?  My experiment shows it does not work with intel compiler but however it is under gcc compiler.

here is the example:

let's say I have KMP_AFFINITY=scatter, which is for my main process. Then inside main process before invoking another executable as the sub-process, putenv is used to modify KMP_AFFINITY=none for the sub-process.

is this supposed to work? my run shows the KMP_AFFINITY=none does not apply to the sub-process if intel compiler is used to compile and link my main program. but it is with gcc compiler.

when I double check the environment, in the sub-process, there is one extra environment variable for my exe with intel compiler

__KMP_REGISTERED_LIB_23907=0xacfa1d0-cafe8af0-libiomp5.a

what does this guy do and how to explain such difference? Thank you

Hongwei

[Zhen_Z_Intel]

Hi Hongwei,

Since your problem is more relevant to Intel compiler not MKL, I will transfer your issue to compiler group. Thanks.

Best regards,
Fiona

[Judith_W_Intel]

Looking in the code it does seem that the compiler resets KMP_AFFINITY to none, i.e. I see this in paropt/paropt_openmp.c:

/*
** (void) __kmpc_set_defaults(char *str);
*/
extern STMT PAROPT_WRN_Gen_Kmpc_Set_Affinity(
   CHAR     *affinity,
   IL0_SRC  src
)
{
...
   CHAR      *envar = "KMP_AFFINITY=";
 

Does using this option help?

-par-affinity=[<modifier>,...]<type>[,<permute>][,<offset>]
          tune application performance by setting different thread affinity
 

[Hongwei_Zhou]

Hi Judith,

thanks for the reply. My main target is to seek a way that different KMP_AFFINITY variables can be set between the main process and sub-process since their performance behavior is different with respect to KMP_AFFINITY value.

Hongwei

[SergeyKostrov]

Could you try: ... kmp_set_defaults("KMP_AFFINITY=scatter"); // for main process ... kmp_set_defaults("KMP_AFFINITY=compact"); // for sub process ...

[Hongwei_Zhou]

Hi Sergey,

I have only the executable for sub-process. If I have kmp_set_defaults call before launching sub-process, there is a message below

OMP: Warning #213: KMP_AFFINITY must be set prior to first parallel region or certain API calls; ignored.

so for sub-process, it looks like the AFFINITY is inherited from parent process and there is no way to change?

hongwei

[SergeyKostrov]

Just reproduced OMP: Warning #213 on a KNL server: ... OMP: Warning #213: KMP_AFFINITY must be set prior to first parallel region...ignored ... >>...so for sub-process, it looks like the AFFINITY is inherited from parent process and there is no way to change? It looks like Yes. Could you try to do it in a different way something like: ... putenv( "KMP_AFFINITY=scatter" ); // for main process ... putenv("KMP_AFFINITY=compact"); // for sub process ... I'll verify it as well...

[SergeyKostrov]

>>... >>putenv( "KMP_AFFINITY=scatter" ); // for main process >>... >>putenv("KMP_AFFINITY=compact"); // for sub process >>... >> >>I'll verify it as well... It doesn't work.

[TimP]

If you mean nested openmp, omp_proc_bind=spread,false should do.

[Hongwei_Zhou]

I did not use nested openmp. My program uses system call to execute another executable directly.

system("sub.exe")

I just want to change the "KMP_AFFINITY" value for sub.exe.

Hongwei

[TimP]

Then it would appear preferable to start a script (e.g. .bat) which includes the environment variable settings.  system() is a function (not void) with return value, in case that makes a difference.

[Hongwei_Zhou]

Yes, I tried already

system("./run.sh");

run.sh:

echo $KMP_AFFINITY >> kmp_aff.txt
export KMP_AFFINITY=none
echo $KMP_AFFINITY >> kmp_aff.txt
env >> kmp_aff.txt
$sub.exe > sub.out 2>&1

I can see the two lines in kmp_aff.txt

scatter

none

which means KMP_AFFINITY is reset to none, but the sub.exe behavior is still like KMP_AFFINITY=scatter. There is no effect at all.  I also found the following env 

__KMP_REGISTERED_LIB_23907=0xacfa1d0-cafe8af0-libiomp5.a

what is this for? 

thanks,

Hongwei

[jimdempseyatthecove]

>>...so for sub-process, it looks like the AFFINITY is inherited from parent process and there is no way to change?

You can change the process affinity to any sub-set (including entire set) of the system affinity using the pthread_setaffinity_np(... in your sub-process prior to instatiating its OpenMP thread pool. Note, there is a pthread API to obtain the system affinity. Do not assume that the system affinity is always 0-based and contiguous (though you may never experience it differently).

Jim Dempsey