mic_kmp_place_threads

[Jen_B_]

Just a quick question, if I set for example mic_kmp_place_threads=60c,1t/60c,2t/60c,3t and mic_omp_proc_bind=close, then I assume the above means that I will have 60 cores utilized and in each of them I will have 1, 2, and 3 threads running respectively, am I right? But what is the job of mic_omp_proc_bind=close? Thank!

[TimP]

Without omp_proc_bind , threads could move around among the designated hardware contexts.  When you run 2 or 3 threads per core, if your application shares data among adjacent threads, proc_bind=close will place adjacent threads on a core so as to improve cache sharing.

[Jen_B_]

If I have kmp_affinity set to scattered or balanced, then do I need the omp_proc_bind or not? Thanks!

[McCalpinJohn]

KMP_PLACE_THREADS defines the set of logical processors that can be used by an OpenMP program, but does not do any additional binding of threads within that set.   So you still want something to limit the possible motion of each of the threads -- either OMP_PROC_BIND or KMP_AFFINITY.   (I recommend against using both at the same time -- you might have trouble figuring out which one takes precedence.)

It sounds like you want three cases:

1. 1 thread/core:  OMP_NUM_THREADS=60, KMP_PLACE_THREADS=60c,1t
2. 2 threads/core: OMP_NUM_THREADS=120, KMP_PLACE_THREADS=60c,2t
3. 3 threads/core: OMP_NUM_THREADS=180, KMP_PLACE_THREADS=60c,3t

For all of these cases you need to use either KMP_AFFINITY=verbose,compact  or   KMP_AFFINITY=verbose,balanced.    Because KMP_PLACE_THREADS limits the total set of cores available, these two distributions give exactly the same result.   KMP_AFFINITY=scatter would give a different (interleaved, rather than "bunched") layout when using more than 1 thread per core.

Although OMP_PROC_BIND works fine, I prefer KMP_AFFINITY because of the "verbose" option.

[Jen_B_]

Thank you both, John and Tim. 1) Does setting kmp_affinity (without omp_proc_bind) make sure that threads do not migrate within the core, so for example if I have 3 threads on a mic core (60c,3t) can they migrate or not? Also, if mic_env_prefix=mic is set, will omp_proc_bind's default or set value be used on mic side? Thanks!!!

[TimP]

omp_proc_bind is a portable alternative for kmp_affinity for the most important cases.  If you have set a kmp_affinity option which conflicts with omp_proc_bind, the expectation is the proc_bind setting will be ignored. omp_proc_bind=close is equivalent to kmp_affinity=compact, proc_bind=spread is equivalent to kmp_affinity=scatter.  You can check these by setting kmp_affinity=verbose.  By itself, that verbose setting works the same regardless of the means for setting affinity. 

kmp_affinity also offers the option to specify a full exact list, as does omp_places.  kmp_place_threads takes care of some cases which previously had to be specified by a full list and a corresponding num_threads setting.  You will see such a case in the Jeffers, Reinders textbook.

When you are running offload mode, the kmp_affinity or omp_proc_bind without prefix controls host side, the one with mic_env_prefix controls coprocessor side, or at least over-rides the unprefixed setting on the coprocessor side.  For MIC native mode, you must set the host and MIC values in the respective shells (then mic_env_prefix has no effect).

If you don't set an affinity on the MIC side, there is no control over how the threads are distributed among cores. and it will change from time to time.  There is no default setting for affinity or proc_bind, unless you allude to the host value being carried over to MIC (which is not a useful default).

kmp_affinity=balanced is meant for distributing threads evenly across cores in the case where OMP_NUM_THREADS may be reduced from the value set by default or by KMP_PLACE_THREADS.  It's useful for MPI jobs, but I haven't heard it recommended for offload.

[Jen_B_]

OK, thanks! Also, just to be sure, is there any difference in setting mic_env_prefix to "mic" or "mic_"?

[James_C_Intel2]

If you don't set an affinity on the MIC side, there is no control over how the threads are distributed among cores. and it will change from time to time.

Sorry, that  is incorrect.  If no affinity is set on the Xeon Phi a default affinity of "scatter,granularity=fine" (see  https://software.intel.com/en-us/articles/openmp-thread-affinity-control) is used. Therefore, because of the "granularity=fine", the OS has no liberty to move threads.

Unless you explicitly set a different granularity the default "granularity=fine" will apply. (e.g. simply setting "KMP_AFFINITY=compact" will still inherit the "granularity=fine")

Therefore, unless you explicitly tell it to do otherwise the OpenMP runtime  on the Xeon Phi will bind threads to individual logical CPUs, and leave the OS no freedom to move them around. This is the default because it normally provides better performance on Xeon Phi, where you own the machine, and are likely worrying about cache-locality (which would be destroyed if the OS is bouncing your threads around the machine).

[Jen_B_]

Thanks James! About my last question is there any difference between mic_env_prefix being set to "mic" or "mic_"?

[jimdempseyatthecove]

If you use "mic_" for the mic_env_prefix then you would use double "_". (mic__KMP_AFFINITY=compact")

Jim Dempsey

[Jen_B_]

Thanks Jim! But why then in the book "High-performance computing on the Intel Xeon Phi" by Endong Wang et al., specifically page 103, they set mic_env_prefix to "mic_" but other env. variables realted to mic they start only with "mic_" only, i.e., without double "__", for example they have "mic_omp_num_threads" and "mic_env_prefix=mic_". You can search keyword "mic_env_prefix=mic_" if you don't have access to the book. Thanks!

[McCalpinJohn]

It should only take a moment to check the syntax with a test case.

I try to always set the "verbose" flag in KMP_AFFINITY to make sure that I get enough information to verify that I have set the Xeon Phi environment variables correctly.  This has saved me from embarrassment on several occasions.

[jimdempseyatthecove]

"look the gift horse in the mouth". iow run a test as John suggests.

In "Intel Xeon Phi Coprocessor High-Performance Programming" by Jim Jeffers and James Reinders, chapter 2, page 48:

$export MIC_ENV_PREFIX=MIC
$export MIC_OMP_NUM_THREADS=120
$export MIC_KMP_AFFINITY=scatter

It would appear that underbar ("_") automatically intercedes between your chosen prefix and the environment variable name you wish to set on the coprocessor.

Jim Dempsey

[Kevin_D_Intel]

Early implementations required the user to include the underscore as part of the prefix string setting (which the book captured) but the current implementation does not; however, it does expect the underscore appear as a separator between the prefix and the environment variable name when setting the newly named environment variable.

It is perhaps better to use an alternative to "MIC" as some have confused that setting as being applicable to certain compiler named environment variables that begin with "MIC". If you will be setting the a MIC_ENV_PREFIX, another common one used is PHI.

There is more detail about the MIC_ENV_PREFIX discussed here.

[Jen_B_]

Thanks Kevin!