Two jobs pinned to the same CPU's

[John_M_8]

Two jobs are pinned to the same CPU's (cores actually) on RHEL 5 U 4:

standard output file has this evidence:

MPDBOOT DONE

[0] MPI startup(): socket data transfer mode [1] MPI startup(): socket data transfer mode [0] MPI Startup(): process is pinned to CPU00 on node compute-7-24.local [1] MPI Startup(): process is pinned to CPU03 on node compute-7-24.local

[0] Rank Pid Node name Pin cpu

[0] 0 10190 compute-7-24.local 0

[0] 1 10189 compute-7-24.local 3

[0] Init(): I_MPI_DEBUG=6

[0] Init(): I_MPI_DEVICE=sock

[0] Init(): MPICH_INTERFACE_HOSTNAME=192.168.9.236

And the other says:

MPDBOOT DONE

[0] MPI startup(): socket data transfer mode [1] MPI startup(): socket data transfer mode [1] MPI Startup(): process is pinned to CPU03 on node compute-7-24.local [0] MPI Startup(): process is pinned to CPU00 on node compute-7-24.local

[0] Rank Pid Node name Pin cpu

[0] 0 10283 compute-7-24.local 0

[0] 1 10282 compute-7-24.local 3

[0] Init(): I_MPI_DEBUG=6

[0] Init(): I_MPI_DEVICE=sock

[0] Init(): MPICH_INTERFACE_HOSTNAME=192.168.9.236

So we see how MPI is pinning those processes to the same cores.

And also from those two files:

[root@compute-7-24 spool]# grep debug 37257.cronus.OU

debug: starting

debug: launch cmd= env I_MPI_JOB_TAGGED_PORT_OUTPUT=1 /opt/intel/impi/3.2.1.009/bin64/mpd.py --ncpus=1 --myhost=compute-7-24 -e -d -s 1

debug: mpd on compute-7-24 on port 38764

debug: info for running mpd: {'ip': '192.168.9.236', 'ncpus': 1, 'list_port': 38764, 'entry_port': '', 'host': 'compute-7-24', 'entry_host': '', 'ifhn': ''}

[root@compute-7-24 spool]# grep debug 37258.cronus.OU

debug: starting

debug: launch cmd= env I_MPI_JOB_TAGGED_PORT_OUTPUT=1 /opt/intel/impi/3.2.1.009/bin64/mpd.py --ncpus=1 --myhost=compute-7-24 -e -d -s 1

debug: mpd on compute-7-24 on port 38699

debug: info for running mpd: {'ip': '192.168.9.236', 'ncpus': 1, 'list_port': 38699, 'entry_port': '', 'host': 'compute-7-24', 'entry_host': '', 'ifhn': ''}

Thus,

Regards,

John Matrow

Wichita State University

[Dmitry_S_Intel]

DearJohn!

Is it correct that you have two independent applications running on same nodes and poccesses of the applications are pinned to same CPUs?

Pinning of processes of one MPI application doesn't depend on other MPI applications running on node.

You can disable pinning withI_MPI_PIN=off to allow OS to manage pinning. There area set of environment variables to control pinning (see Reference Manual).

Best regards,
Dmitry

[John_M_8]

It is two copies of the same application but with different data sets.

The result is that each job gets 1/2 of the CPU's assigned to both.

We ran three jobs and each got 1/3 of the CPU's assigned to all three.

Why isn't it assigning other CPU's?

John

[TimP]

In case you didn't pick up on Dmitry's comment, if you set environment variable I_MPI_PIN=off, the OS should be able to assign each job to distinct logical CPUs when available. By default, the Intel MPI pinning tries to select the best core assignments on the assumption that no other jobs are running, but it would pick the same group for jobs with same number of ranks.
If you run your jobs in separate shell sessions, you could set each job to a different set of logical CPUs, preferably such that each job resides on a different physical CPU package. Doing so would be much more important on a 4 socket platform.