topic Hi Vineet, in Intel® MPI Library

Selective MPI process placements on processors

soni__vineet — Wed, 20 Jan 2016 16:11:45 GMT

Hello,

I am looking for a way to place some selected MPI process ranks (MPI_COMM_WORLD ranks) on different sockets of the nodes.

For example, if I have 4 MPI processes say 0,1,2,3. And, I have two nodes each having 2 sockets. Is there any way I can specify that the processes 0 and 4 be on two different sockets of node 1, and similarly processes 1 and 3 be on node 2?

Basically, I am looking for an equivalent to 'rankfile' of Open MPI on Intel MPI.

I hope I am clear enough.

Regards,

Vineet

Hi Vineet,

Artem_R_Intel1 — Thu, 21 Jan 2016 07:10:10 GMT

Hi Vineet,

You can find the related controls and its description in the Intel® MPI Library for Linux* OS Reference Manual / chapter "3.2. Process Pinning".

Hi Artem,

soni__vineet — Thu, 21 Jan 2016 10:21:00 GMT

Hi Artem,

Thank you for the reply.

I have gone through the link you mentioned. So, for the example that I mentioned initially (e.g. MPI processes - 0,1,2,3; with the group of processes 0&3 on node1, and processes 1&2 on node2), it should work in the following way:

mpirun -host node1 -genv I_MPI_PIN 1 I_MPI_PIN_PROCESSOR_LIST 0,3 -n 2 ./a.out : \

-host node2 -genv I_MPI_PIN 1 I_MPI_PIN_PROCESSOR_LIST 1,2 -n 2 ./a.out

I am confused here with the numbers following the variable I_MPI_PIN_PROCESSOR_LIST. Would you please tell me, if it shows the logical processor number or the MPI process rank? Also, I am not sure if I should use '-perhost 2' to place these processes on individual sockets.

Also, how will it handle the OpenMP threads, if it has any? Let say each socket has a quad core processor, and I use OMP_SET_NUM_THREADS(4). Will it use all physical cores of a socket as OpenMP threads for a single MPI process?

Regards,

Vineet

Hi Vineet,

Artem_R_Intel1 — Mon, 25 Jan 2016 07:58:03 GMT

Hi Vineet,

As far as I understand your scenario in the initial message it should work as you want with the default process pinning schema - you can see it with I_MPI_DEBUG=4 (see the cpuinfo utility for the detailed information about your CPU configuration):

$ I_MPI_DEBUG=4 mpirun -ppn 1 -n 4 -hosts node01,node02 IMB-MPI1 bcast -npmin 4
...

[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 24472 node01 {0,1,2,3,4,5,6,7,8,9,10,11,12,13,28,29,30,31,32,33,34,35,36,37,38,39,40,41}
[0] MPI startup(): 1 122065 node02 {0,1,2,3,4,5,6,7,8,9,10,11,12,13,28,29,30,31,32,33,34,35,36,37,38,39,40,41}
[0] MPI startup(): 2 24473 node01 {14,15,16,17,18,19,20,21,22,23,24,25,26,27,42,43,44,45,46,47,48,49,50,51,52,53,54
,55}
[0] MPI startup(): 3 122066 node02 {14,15,16,17,18,19,20,21,22,23,24,25,26,27,42,43,44,45,46,47,48,49,50,51,52,53,54
,55}

...