# Scatter the processes across sockets **URL:** \ **Category:** Intel® MPI Library\ **Tags:** intel-mpi\ **Created:** [June 27, 2024, 12:10am UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658 "2024-06-27T00:10:59Z")\ **Posts on this page:** 7\ **Page:** 1

**Author:** ![ZQyouOSC](https://avatars.discourse-cdn.com/v4/letter/z/276EF2/32.png) [@ZQyouOSC](https://community.intel.com/u/ZQyouOSC)\ **Post date:** [June 27, 2024, 12:10am UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/1 "2024-06-27T00:10:59Z")

Hello, I am researching how to pin processes across sockets using the I\_MPI\_PIN\_DOMAIN and I\_MPI\_PIN\_ORDER environment variables. I have tried many combinations, but none of them have worked. I used Intel MPI 2021.12.1 and ran [TACC](https://github.com/TACC/amask) [amask](https://github.com/TACC/amask) on two systems: one equipped with two Xeon Platinum 8470 CPUs and the other equipped with two Xeon CPU Max 9470 CPUs. With the default settings, I ran the following command: ```auto mpiexec -n 4 amask_mpi ``` and I got the process placement in bunch order as expected: ```auto    Each row of matrix is a mask for a Hardware Thread (hwt).
   CORE ID = matrix digit + column group # in |...|
   A set mask bit (proc-id) = core id + add 104 to each additional row.

rank | 0 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
0000 0---4---8---2---6---0---4---8---2---6---0---4---8---2---6---0---4---8---2---6---0---4---8---2-----------
0001 --2---6---0---4---8---2---6---0---4---8---2---6---0---4---8---2---6---0---4---8---2---6---0---4---------
0002 -1---5---9---3---7---1---5---9---3---7---1---5---9---3---7---1---5---9---3---7---1---5---9---3----------
0003 ---3---7---1---5---9---3---7---1---5---9---3---7---1---5---9---3---7---1---5---9---3---7---1---5-------- ``` The first two processes are bound to the first and second NUMA nodes in the first socket, and the remaining two processes are bound to the first and second NUMA nodes in the second socket. I can also achieve the process placement in compact order with the following variables: ```auto I_MPI_PIN_DOMAIN = core
I_MPI_PIN_ORDER = compact ``` and I got the processes placed in the first NUMA node: ```auto    Each row of matrix is a mask for a Hardware Thread (hwt).
   CORE ID = matrix digit + column group # in |...|
   A set mask bit (proc-id) = core id + add 104 to each additional row.

rank | 0 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
0000 0-------------------------------------------------------------------------------------------------------
0001 --------8-----------------------------------------------------------------------------------------------
0002 ----------------6---------------------------------------------------------------------------------------
0003 ------------------------4------------------------------------------------------------------------------- ``` I have been experimenting with different values of I\_MPI\_PIN\_DOMAIN and I\_MPI\_PIN\_ORDER to achieve process placement in a scatter order across sockets. For example, I want the first and third processes to be bound to the first socket, and the second and fourth processes to be bound to the second socket. However, I have not had any success in finding a working combination. Could you please provide any suggestions? Thank you. ---

**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [July 1, 2024, 4:35pm UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/2 "2024-07-01T16:35:37Z")

@ZQyouOSC Can you please post the output after setting export I\_MPI\_DEBUG=10 ? ---

**Author:** ![ZQyou](https://avatars.discourse-cdn.com/v4/letter/z/3700C4/32.png) [@ZQyou](https://community.intel.com/u/ZQyou)\ **Post date:** [July 3, 2024, 3:09pm UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/3 "2024-07-03T15:09:03Z")

Sure. The following is the output with I\_MPI\_DEBUG=10, I\_MPI\_PIN\_DOMAIN=core and I\_MPI\_PIN\_ORDER=scatter: ```auto [0] MPI startup(): Intel(R) MPI Library, Version 2021.10 Build 20230619 (id: c2e19c2f3e)
[0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): shm segment size (342 MB per rank) * (4 local ranks) = 1368 MB total
[0] MPI startup(): libfabric loaded: libfabric.so.1
[0] MPI startup(): libfabric version: 1.18.0-impi
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): File "/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/intel-oneapi-mpi/intel/2021.10.0/2021.10.0-a2ei2t4/mpi/2021.10.0/etc/tuning_spr_shm-ofi_mlx_100.dat" not found
[0] MPI startup(): Load tuning file: "/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/intel-oneapi-mpi/intel/2021.10.0/2021.10.0-a2ei2t4/mpi/2021.10.0/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): tag bits available: 20 (TAG_UB value: 1048575)
[0] MPI startup(): source bits available: 21 (Maximal number of rank: 2097151)
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 776163 c1002.ten.osc.edu {0}
[0] MPI startup(): 1 776164 c1002.ten.osc.edu {8}
[0] MPI startup(): 2 776165 c1002.ten.osc.edu {16}
[0] MPI startup(): 3 776166 c1002.ten.osc.edu {24}
[0] MPI startup(): I_MPI_CC=icc
[0] MPI startup(): I_MPI_CXX=icpc
[0] MPI startup(): I_MPI_FC=ifort
[0] MPI startup(): I_MPI_F90=ifort
[0] MPI startup(): I_MPI_F77=ifort
[0] MPI startup(): I_MPI_ROOT=/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/intel-oneapi-mpi/intel/2021.10.0/2021.10.0-a2ei2t4/mpi/2021.10.0
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0
[0] MPI startup(): I_MPI_HYDRA_BRANCH_COUNT=-1
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=0
[0] MPI startup(): I_MPI_PIN_DOMAIN=core
[0] MPI startup(): I_MPI_PIN_ORDER=scatter
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_DEBUG=10

   Each row of matrix is a mask for a Hardware Thread (hwt).
   CORE ID = matrix digit + column group # in |...|
   A set mask bit (proc-id) = core id + add 104 to each additional row.

rank | 0 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
0000 0-------------------------------------------------------------------------------------------------------
0001 --------8-----------------------------------------------------------------------------------------------
0002 ----------------6---------------------------------------------------------------------------------------
0003 ------------------------4------------------------------------------------------------------------------- ``` ---

**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [July 3, 2024, 3:56pm UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/4 "2024-07-03T15:56:20Z")

It seems you are using Slurm. If you are using Slurm, please use srun und manage pinning through Slurm. To use mpiexec/mpirun and ignore the Slurm settings please follow the settings here: > **[Job Schedulers Support](https://www.intel.com/content/www/us/en/docs/mpi-library/developer-guide-linux/2021-13/job-schedulers-support.html)** > > This Developer Guide contains instructions for running, debugging, and tuning the Intel® MPI Library. ---

**Author:** ![ZQyou](https://avatars.discourse-cdn.com/v4/letter/z/3700C4/32.png) [@ZQyou](https://community.intel.com/u/ZQyou)\ **Post date:** [July 3, 2024, 4:21pm UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/5 "2024-07-03T16:21:52Z")

Yes, I am using Slurm, but I am sure that I have used the IntelMPI Hydra process manager. When I use Slurm, I have set the following: ```auto I_MPI_HYDRA_BOOTSTRAP=slurm
I_MPI_PMI_LIBRARY=/usr/lib64/libpmi2.so ``` I had these settings to use the Hydra process manager when I obtained the previous output I sent: ```auto export -n I_MPI_HYDRA_BOOTSTRAP I_MPI_PMI_LIBRARY ``` I am confident because Slurm CPU binding control is not working. I also tried using SSH as the Hydra bootstrap, but I got the same result. ```auto $ export -n I_MPI_HYDRA_BOOTSTRAP I_MPI_PMI_LIBRARY
$ export I_MPI_HYDRA_BOOTSTRAP=ssh
$ mpiexec -n 4 bin/amask_mpi

[0] MPI startup(): Intel(R) MPI Library, Version 2021.10 Build 20230619 (id: c2e19c2f3e)
[0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): shm segment size (342 MB per rank) * (4 local ranks) = 1368 MB total
[0] MPI startup(): libfabric loaded: libfabric.so.1
[0] MPI startup(): libfabric version: 1.18.0-impi
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): File "/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/intel-oneapi-mpi/intel/2021.10.0/2021.10.0-a2ei2t4/mpi/2021.10.0/etc/tuning_spr_shm-ofi_mlx_100.dat" not found
[0] MPI startup(): Load tuning file: "/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/intel-oneapi-mpi/intel/2021.10.0/2021.10.0-a2ei2t4/mpi/2021.10.0/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): tag bits available: 20 (TAG_UB value: 1048575)
[0] MPI startup(): source bits available: 21 (Maximal number of rank: 2097151)
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 783094 c1002.ten.osc.edu {0}
[0] MPI startup(): 1 783095 c1002.ten.osc.edu {8}
[0] MPI startup(): 2 783096 c1002.ten.osc.edu {16}
[0] MPI startup(): 3 783097 c1002.ten.osc.edu {24}
[0] MPI startup(): I_MPI_CC=icc
[0] MPI startup(): I_MPI_CXX=icpc
[0] MPI startup(): I_MPI_FC=ifort
[0] MPI startup(): I_MPI_F90=ifort
[0] MPI startup(): I_MPI_F77=ifort
[0] MPI startup(): I_MPI_ROOT=/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/intel-oneapi-mpi/intel/2021.10.0/2021.10.0-a2ei2t4/mpi/2021.10.0
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0
[0] MPI startup(): I_MPI_HYDRA_BRANCH_COUNT=-1
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=ssh
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=0
[0] MPI startup(): I_MPI_PIN_DOMAIN=core
[0] MPI startup(): I_MPI_PIN_ORDER=scatter
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_DEBUG=10

   Each row of matrix is a mask for a Hardware Thread (hwt).
   CORE ID = matrix digit + column group # in |...|
   A set mask bit (proc-id) = core id + add 104 to each additional row.

rank | 0 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
0000 0-------------------------------------------------------------------------------------------------------
0001 --------8-----------------------------------------------------------------------------------------------
0002 ----------------6---------------------------------------------------------------------------------------
0003 ------------------------4------------------------------------------------------------------------------- ``` ---

**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [July 8, 2024, 10:28am UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/6 "2024-07-08T10:28:50Z")

@ZQyouOSC can you please try the pinning simulator: > **[Pinning Simulator for Intel® MPI Library](https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library-pinning-simulator.html)** > > Visualize your pinning configuration to select the optimal settings using Intel® MPI Library. Also please set I\_MPI\_PIN\_RESPECT\_HCA=0 I\_MPI\_PIN\_RESPECT\_CPUSET=0 Best ---

**Author:** ![ZQyouOSC](https://avatars.discourse-cdn.com/v4/letter/z/276EF2/32.png) [@ZQyouOSC](https://community.intel.com/u/ZQyouOSC)\ **Post date:** [July 10, 2024, 6:03pm UTC](https://community.intel.com/t/scatter-the-processes-across-sockets/124658/7 "2024-07-10T18:03:10Z")

Hello, I have tried the simulator, and it suggests the following command line to achieve scatter pinning across sockets: ```auto I_MPI_PIN_DOMAIN=core I_MPI_PIN_ORDER=scatter I_MPI_PIN_CELL=unit mpiexec -n 4 ``` I used the same environment variables along with the other two you suggested previously. However, I still got the compact-order result as I reported.