topic Re: IntelMPI cpuinfo showing one thread per core in Intel® MPI Library

IntelMPI cpuinfo showing one thread per core

mgHarish — Tue, 11 Feb 2025 21:45:48 GMT

My machine has 96 Physical Cores (192 logical cpu), but the cpuinfo comes with intelmpi show threads per core = 1.

lscpu

cpuinfo (intelmpi)

Because of this I was not able to pin processors to the logical cpus.

Re: IntelMPI cpuinfo showing one thread per core

TobiasK — Mon, 17 Feb 2025 10:36:12 GMT

@mgHarish thanks for reporting that, however, it would be really helpful if you also provide your OS version and the version of Intel MPI used to generate this output.

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Mon, 17 Feb 2025 10:54:32 GMT

Hi @TobiasK, Thank for the response!

OS Version

IntelMPI Version

Re: IntelMPI cpuinfo showing one thread per core

TobiasK — Mon, 17 Feb 2025 11:15:20 GMT

@mgHarish thanks, can you please provide the output of?

export I_MPI_DEBUG=10 mpirun IMB-MPI1 barrier

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Mon, 17 Feb 2025 11:38:54 GMT

@TobiasK If you see the pinning information, each pin cpu {0-95} has two mpi ranks. My expectation is pin cpu should be 0-191 with one mpi ranks on each cpu. Thanks!

[0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20241121 (id: e7829d6)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric loaded: libfabric.so.1
[0] MPI startup(): libfabric version: 1.21.0-impi
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): libfabric provider: efa
[0] MPI startup(): shm segment size (150 MB per rank) * (192 local ranks) = 28870 MB total
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa_100_x1.dat" not found
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa_100.dat" not found
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa.dat" not found
[0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: async_progress coll split: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): tag bits available: 30 (TAG_UB value: 1073741823)
[0] MPI startup(): source bits available: 30 (Maximal number of rank: 1073741823)
[0] MPI startup(): Number of NICs: 1
[0] MPI startup(): ===== NIC pinning on efa-od-az-1a-dy-c7i-48xlarge-1 =====
[0] MPI startup(): Rank Thread id Pin nic Nic id
[0] MPI startup(): 0 0 rdmap150s0-rdm 0
[0] MPI startup(): 1 0 rdmap150s0-rdm 0
[0] MPI startup(): 2 0 rdmap150s0-rdm 0
[0] MPI startup(): 3 0 rdmap150s0-rdm 0
[0] MPI startup(): 4 0 rdmap150s0-rdm 0
[0] MPI startup(): 5 0 rdmap150s0-rdm 0
[0] MPI startup(): 6 0 rdmap150s0-rdm 0
[0] MPI startup(): 7 0 rdmap150s0-rdm 0
[0] MPI startup(): 8 0 rdmap150s0-rdm 0
[0] MPI startup(): 9 0 rdmap150s0-rdm 0
[0] MPI startup(): 10 0 rdmap150s0-rdm 0
[0] MPI startup(): 11 0 rdmap150s0-rdm 0
[0] MPI startup(): 12 0 rdmap150s0-rdm 0
[0] MPI startup(): 13 0 rdmap150s0-rdm 0
[0] MPI startup(): 14 0 rdmap150s0-rdm 0
[0] MPI startup(): 15 0 rdmap150s0-rdm 0
[0] MPI startup(): 16 0 rdmap150s0-rdm 0
[0] MPI startup(): 17 0 rdmap150s0-rdm 0
[0] MPI startup(): 18 0 rdmap150s0-rdm 0
[0] MPI startup(): 19 0 rdmap150s0-rdm 0
[0] MPI startup(): 20 0 rdmap150s0-rdm 0
[0] MPI startup(): 21 0 rdmap150s0-rdm 0
[0] MPI startup(): 22 0 rdmap150s0-rdm 0
[0] MPI startup(): 23 0 rdmap150s0-rdm 0
[0] MPI startup(): 24 0 rdmap150s0-rdm 0
[0] MPI startup(): 25 0 rdmap150s0-rdm 0
[0] MPI startup(): 26 0 rdmap150s0-rdm 0
[0] MPI startup(): 27 0 rdmap150s0-rdm 0
[0] MPI startup(): 28 0 rdmap150s0-rdm 0
[0] MPI startup(): 29 0 rdmap150s0-rdm 0
[0] MPI startup(): 30 0 rdmap150s0-rdm 0
[0] MPI startup(): 31 0 rdmap150s0-rdm 0
[0] MPI startup(): 32 0 rdmap150s0-rdm 0
[0] MPI startup(): 33 0 rdmap150s0-rdm 0
[0] MPI startup(): 34 0 rdmap150s0-rdm 0
[0] MPI startup(): 35 0 rdmap150s0-rdm 0
[0] MPI startup(): 36 0 rdmap150s0-rdm 0
[0] MPI startup(): 37 0 rdmap150s0-rdm 0
[0] MPI startup(): 38 0 rdmap150s0-rdm 0
[0] MPI startup(): 39 0 rdmap150s0-rdm 0
[0] MPI startup(): 40 0 rdmap150s0-rdm 0
[0] MPI startup(): 41 0 rdmap150s0-rdm 0
[0] MPI startup(): 42 0 rdmap150s0-rdm 0
[0] MPI startup(): 43 0 rdmap150s0-rdm 0
[0] MPI startup(): 44 0 rdmap150s0-rdm 0
[0] MPI startup(): 45 0 rdmap150s0-rdm 0
[0] MPI startup(): 46 0 rdmap150s0-rdm 0
[0] MPI startup(): 47 0 rdmap150s0-rdm 0
[0] MPI startup(): 48 0 rdmap150s0-rdm 0
[0] MPI startup(): 49 0 rdmap150s0-rdm 0
[0] MPI startup(): 50 0 rdmap150s0-rdm 0
[0] MPI startup(): 51 0 rdmap150s0-rdm 0
[0] MPI startup(): 52 0 rdmap150s0-rdm 0
[0] MPI startup(): 53 0 rdmap150s0-rdm 0
[0] MPI startup(): 54 0 rdmap150s0-rdm 0
[0] MPI startup(): 55 0 rdmap150s0-rdm 0
[0] MPI startup(): 56 0 rdmap150s0-rdm 0
[0] MPI startup(): 57 0 rdmap150s0-rdm 0
[0] MPI startup(): 58 0 rdmap150s0-rdm 0
[0] MPI startup(): 59 0 rdmap150s0-rdm 0
[0] MPI startup(): 60 0 rdmap150s0-rdm 0
[0] MPI startup(): 61 0 rdmap150s0-rdm 0
[0] MPI startup(): 62 0 rdmap150s0-rdm 0
[0] MPI startup(): 63 0 rdmap150s0-rdm 0
[0] MPI startup(): 64 0 rdmap150s0-rdm 0
[0] MPI startup(): 65 0 rdmap150s0-rdm 0
[0] MPI startup(): 66 0 rdmap150s0-rdm 0
[0] MPI startup(): 67 0 rdmap150s0-rdm 0
[0] MPI startup(): 68 0 rdmap150s0-rdm 0
[0] MPI startup(): 69 0 rdmap150s0-rdm 0
[0] MPI startup(): 70 0 rdmap150s0-rdm 0
[0] MPI startup(): 71 0 rdmap150s0-rdm 0
[0] MPI startup(): 72 0 rdmap150s0-rdm 0
[0] MPI startup(): 73 0 rdmap150s0-rdm 0
[0] MPI startup(): 74 0 rdmap150s0-rdm 0
[0] MPI startup(): 75 0 rdmap150s0-rdm 0
[0] MPI startup(): 76 0 rdmap150s0-rdm 0
[0] MPI startup(): 77 0 rdmap150s0-rdm 0
[0] MPI startup(): 78 0 rdmap150s0-rdm 0
[0] MPI startup(): 79 0 rdmap150s0-rdm 0
[0] MPI startup(): 80 0 rdmap150s0-rdm 0
[0] MPI startup(): 81 0 rdmap150s0-rdm 0
[0] MPI startup(): 82 0 rdmap150s0-rdm 0
[0] MPI startup(): 83 0 rdmap150s0-rdm 0
[0] MPI startup(): 84 0 rdmap150s0-rdm 0
[0] MPI startup(): 85 0 rdmap150s0-rdm 0
[0] MPI startup(): 86 0 rdmap150s0-rdm 0
[0] MPI startup(): 87 0 rdmap150s0-rdm 0
[0] MPI startup(): 88 0 rdmap150s0-rdm 0
[0] MPI startup(): 89 0 rdmap150s0-rdm 0
[0] MPI startup(): 90 0 rdmap150s0-rdm 0
[0] MPI startup(): 91 0 rdmap150s0-rdm 0
[0] MPI startup(): 92 0 rdmap150s0-rdm 0
[0] MPI startup(): 93 0 rdmap150s0-rdm 0
[0] MPI startup(): 94 0 rdmap150s0-rdm 0
[0] MPI startup(): 95 0 rdmap150s0-rdm 0
[0] MPI startup(): 96 0 rdmap150s0-rdm 0
[0] MPI startup(): 97 0 rdmap150s0-rdm 0
[0] MPI startup(): 98 0 rdmap150s0-rdm 0
[0] MPI startup(): 99 0 rdmap150s0-rdm 0
[0] MPI startup(): 100 0 rdmap150s0-rdm 0
[0] MPI startup(): 101 0 rdmap150s0-rdm 0
[0] MPI startup(): 102 0 rdmap150s0-rdm 0
[0] MPI startup(): 103 0 rdmap150s0-rdm 0
[0] MPI startup(): 104 0 rdmap150s0-rdm 0
[0] MPI startup(): 105 0 rdmap150s0-rdm 0
[0] MPI startup(): 106 0 rdmap150s0-rdm 0
[0] MPI startup(): 107 0 rdmap150s0-rdm 0
[0] MPI startup(): 108 0 rdmap150s0-rdm 0
[0] MPI startup(): 109 0 rdmap150s0-rdm 0
[0] MPI startup(): 110 0 rdmap150s0-rdm 0
[0] MPI startup(): 111 0 rdmap150s0-rdm 0
[0] MPI startup(): 112 0 rdmap150s0-rdm 0
[0] MPI startup(): 113 0 rdmap150s0-rdm 0
[0] MPI startup(): 114 0 rdmap150s0-rdm 0
[0] MPI startup(): 115 0 rdmap150s0-rdm 0
[0] MPI startup(): 116 0 rdmap150s0-rdm 0
[0] MPI startup(): 117 0 rdmap150s0-rdm 0
[0] MPI startup(): 118 0 rdmap150s0-rdm 0
[0] MPI startup(): 119 0 rdmap150s0-rdm 0
[0] MPI startup(): 120 0 rdmap150s0-rdm 0
[0] MPI startup(): 121 0 rdmap150s0-rdm 0
[0] MPI startup(): 122 0 rdmap150s0-rdm 0
[0] MPI startup(): 123 0 rdmap150s0-rdm 0
[0] MPI startup(): 124 0 rdmap150s0-rdm 0
[0] MPI startup(): 125 0 rdmap150s0-rdm 0
[0] MPI startup(): 126 0 rdmap150s0-rdm 0
[0] MPI startup(): 127 0 rdmap150s0-rdm 0
[0] MPI startup(): 128 0 rdmap150s0-rdm 0
[0] MPI startup(): 129 0 rdmap150s0-rdm 0
[0] MPI startup(): 130 0 rdmap150s0-rdm 0
[0] MPI startup(): 131 0 rdmap150s0-rdm 0
[0] MPI startup(): 132 0 rdmap150s0-rdm 0
[0] MPI startup(): 133 0 rdmap150s0-rdm 0
[0] MPI startup(): 134 0 rdmap150s0-rdm 0
[0] MPI startup(): 135 0 rdmap150s0-rdm 0
[0] MPI startup(): 136 0 rdmap150s0-rdm 0
[0] MPI startup(): 137 0 rdmap150s0-rdm 0
[0] MPI startup(): 138 0 rdmap150s0-rdm 0
[0] MPI startup(): 139 0 rdmap150s0-rdm 0
[0] MPI startup(): 140 0 rdmap150s0-rdm 0
[0] MPI startup(): 141 0 rdmap150s0-rdm 0
[0] MPI startup(): 142 0 rdmap150s0-rdm 0
[0] MPI startup(): 143 0 rdmap150s0-rdm 0
[0] MPI startup(): 144 0 rdmap150s0-rdm 0
[0] MPI startup(): 145 0 rdmap150s0-rdm 0
[0] MPI startup(): 146 0 rdmap150s0-rdm 0
[0] MPI startup(): 147 0 rdmap150s0-rdm 0
[0] MPI startup(): 148 0 rdmap150s0-rdm 0
[0] MPI startup(): 149 0 rdmap150s0-rdm 0
[0] MPI startup(): 150 0 rdmap150s0-rdm 0
[0] MPI startup(): 151 0 rdmap150s0-rdm 0
[0] MPI startup(): 152 0 rdmap150s0-rdm 0
[0] MPI startup(): 153 0 rdmap150s0-rdm 0
[0] MPI startup(): 154 0 rdmap150s0-rdm 0
[0] MPI startup(): 155 0 rdmap150s0-rdm 0
[0] MPI startup(): 156 0 rdmap150s0-rdm 0
[0] MPI startup(): 157 0 rdmap150s0-rdm 0
[0] MPI startup(): 158 0 rdmap150s0-rdm 0
[0] MPI startup(): 159 0 rdmap150s0-rdm 0
[0] MPI startup(): 160 0 rdmap150s0-rdm 0
[0] MPI startup(): 161 0 rdmap150s0-rdm 0
[0] MPI startup(): 162 0 rdmap150s0-rdm 0
[0] MPI startup(): 163 0 rdmap150s0-rdm 0
[0] MPI startup(): 164 0 rdmap150s0-rdm 0
[0] MPI startup(): 165 0 rdmap150s0-rdm 0
[0] MPI startup(): 166 0 rdmap150s0-rdm 0
[0] MPI startup(): 167 0 rdmap150s0-rdm 0
[0] MPI startup(): 168 0 rdmap150s0-rdm 0
[0] MPI startup(): 169 0 rdmap150s0-rdm 0
[0] MPI startup(): 170 0 rdmap150s0-rdm 0
[0] MPI startup(): 171 0 rdmap150s0-rdm 0
[0] MPI startup(): 172 0 rdmap150s0-rdm 0
[0] MPI startup(): 173 0 rdmap150s0-rdm 0
[0] MPI startup(): 174 0 rdmap150s0-rdm 0
[0] MPI startup(): 175 0 rdmap150s0-rdm 0
[0] MPI startup(): 176 0 rdmap150s0-rdm 0
[0] MPI startup(): 177 0 rdmap150s0-rdm 0
[0] MPI startup(): 178 0 rdmap150s0-rdm 0
[0] MPI startup(): 179 0 rdmap150s0-rdm 0
[0] MPI startup(): 180 0 rdmap150s0-rdm 0
[0] MPI startup(): 181 0 rdmap150s0-rdm 0
[0] MPI startup(): 182 0 rdmap150s0-rdm 0
[0] MPI startup(): 183 0 rdmap150s0-rdm 0
[0] MPI startup(): 184 0 rdmap150s0-rdm 0
[0] MPI startup(): 185 0 rdmap150s0-rdm 0
[0] MPI startup(): 186 0 rdmap150s0-rdm 0
[0] MPI startup(): 187 0 rdmap150s0-rdm 0
[0] MPI startup(): 188 0 rdmap150s0-rdm 0
[0] MPI startup(): 189 0 rdmap150s0-rdm 0
[0] MPI startup(): 190 0 rdmap150s0-rdm 0
[0] MPI startup(): 191 0 rdmap150s0-rdm 0
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 15233 efa-od-az-1a-dy-c7i-48xlarge-1 {48}
[0] MPI startup(): 1 15234 efa-od-az-1a-dy-c7i-48xlarge-1 {49}
[0] MPI startup(): 2 15235 efa-od-az-1a-dy-c7i-48xlarge-1 {50}
[0] MPI startup(): 3 15236 efa-od-az-1a-dy-c7i-48xlarge-1 {51}
[0] MPI startup(): 4 15237 efa-od-az-1a-dy-c7i-48xlarge-1 {52}
[0] MPI startup(): 5 15238 efa-od-az-1a-dy-c7i-48xlarge-1 {53}
[0] MPI startup(): 6 15239 efa-od-az-1a-dy-c7i-48xlarge-1 {54}
[0] MPI startup(): 7 15240 efa-od-az-1a-dy-c7i-48xlarge-1 {55}
[0] MPI startup(): 8 15241 efa-od-az-1a-dy-c7i-48xlarge-1 {56}
[0] MPI startup(): 9 15242 efa-od-az-1a-dy-c7i-48xlarge-1 {57}
[0] MPI startup(): 10 15243 efa-od-az-1a-dy-c7i-48xlarge-1 {58}
[0] MPI startup(): 11 15244 efa-od-az-1a-dy-c7i-48xlarge-1 {59}
[0] MPI startup(): 12 15245 efa-od-az-1a-dy-c7i-48xlarge-1 {60}
[0] MPI startup(): 13 15250 efa-od-az-1a-dy-c7i-48xlarge-1 {61}
[0] MPI startup(): 14 15253 efa-od-az-1a-dy-c7i-48xlarge-1 {62}
[0] MPI startup(): 15 15254 efa-od-az-1a-dy-c7i-48xlarge-1 {63}
[0] MPI startup(): 16 15258 efa-od-az-1a-dy-c7i-48xlarge-1 {64}
[0] MPI startup(): 17 15265 efa-od-az-1a-dy-c7i-48xlarge-1 {65}
[0] MPI startup(): 18 15269 efa-od-az-1a-dy-c7i-48xlarge-1 {66}
[0] MPI startup(): 19 15270 efa-od-az-1a-dy-c7i-48xlarge-1 {67}
[0] MPI startup(): 20 15273 efa-od-az-1a-dy-c7i-48xlarge-1 {68}
[0] MPI startup(): 21 15276 efa-od-az-1a-dy-c7i-48xlarge-1 {69}
[0] MPI startup(): 22 15279 efa-od-az-1a-dy-c7i-48xlarge-1 {70}
[0] MPI startup(): 23 15285 efa-od-az-1a-dy-c7i-48xlarge-1 {71}
[0] MPI startup(): 24 15287 efa-od-az-1a-dy-c7i-48xlarge-1 {72}
[0] MPI startup(): 25 15293 efa-od-az-1a-dy-c7i-48xlarge-1 {73}
[0] MPI startup(): 26 15294 efa-od-az-1a-dy-c7i-48xlarge-1 {74}
[0] MPI startup(): 27 15297 efa-od-az-1a-dy-c7i-48xlarge-1 {75}
[0] MPI startup(): 28 15298 efa-od-az-1a-dy-c7i-48xlarge-1 {76}
[0] MPI startup(): 29 15299 efa-od-az-1a-dy-c7i-48xlarge-1 {77}
[0] MPI startup(): 30 15303 efa-od-az-1a-dy-c7i-48xlarge-1 {78}
[0] MPI startup(): 31 15304 efa-od-az-1a-dy-c7i-48xlarge-1 {79}
[0] MPI startup(): 32 15307 efa-od-az-1a-dy-c7i-48xlarge-1 {80}
[0] MPI startup(): 33 15311 efa-od-az-1a-dy-c7i-48xlarge-1 {81}
[0] MPI startup(): 34 15312 efa-od-az-1a-dy-c7i-48xlarge-1 {82}
[0] MPI startup(): 35 15314 efa-od-az-1a-dy-c7i-48xlarge-1 {83}
[0] MPI startup(): 36 15315 efa-od-az-1a-dy-c7i-48xlarge-1 {84}
[0] MPI startup(): 37 15316 efa-od-az-1a-dy-c7i-48xlarge-1 {85}
[0] MPI startup(): 38 15321 efa-od-az-1a-dy-c7i-48xlarge-1 {86}
[0] MPI startup(): 39 15325 efa-od-az-1a-dy-c7i-48xlarge-1 {87}
[0] MPI startup(): 40 15326 efa-od-az-1a-dy-c7i-48xlarge-1 {88}
[0] MPI startup(): 41 15328 efa-od-az-1a-dy-c7i-48xlarge-1 {89}
[0] MPI startup(): 42 15329 efa-od-az-1a-dy-c7i-48xlarge-1 {90}
[0] MPI startup(): 43 15330 efa-od-az-1a-dy-c7i-48xlarge-1 {91}
[0] MPI startup(): 44 15336 efa-od-az-1a-dy-c7i-48xlarge-1 {92}
[0] MPI startup(): 45 15337 efa-od-az-1a-dy-c7i-48xlarge-1 {93}
[0] MPI startup(): 46 15340 efa-od-az-1a-dy-c7i-48xlarge-1 {94}
[0] MPI startup(): 47 15344 efa-od-az-1a-dy-c7i-48xlarge-1 {95}
[0] MPI startup(): 48 15348 efa-od-az-1a-dy-c7i-48xlarge-1 {0}
[0] MPI startup(): 49 15349 efa-od-az-1a-dy-c7i-48xlarge-1 {1}
[0] MPI startup(): 50 15352 efa-od-az-1a-dy-c7i-48xlarge-1 {2}
[0] MPI startup(): 51 15356 efa-od-az-1a-dy-c7i-48xlarge-1 {3}
[0] MPI startup(): 52 15362 efa-od-az-1a-dy-c7i-48xlarge-1 {4}
[0] MPI startup(): 53 15364 efa-od-az-1a-dy-c7i-48xlarge-1 {5}
[0] MPI startup(): 54 15367 efa-od-az-1a-dy-c7i-48xlarge-1 {6}
[0] MPI startup(): 55 15372 efa-od-az-1a-dy-c7i-48xlarge-1 {7}
[0] MPI startup(): 56 15373 efa-od-az-1a-dy-c7i-48xlarge-1 {8}
[0] MPI startup(): 57 15377 efa-od-az-1a-dy-c7i-48xlarge-1 {9}
[0] MPI startup(): 58 15378 efa-od-az-1a-dy-c7i-48xlarge-1 {10}
[0] MPI startup(): 59 15381 efa-od-az-1a-dy-c7i-48xlarge-1 {11}
[0] MPI startup(): 60 15386 efa-od-az-1a-dy-c7i-48xlarge-1 {12}
[0] MPI startup(): 61 15387 efa-od-az-1a-dy-c7i-48xlarge-1 {13}
[0] MPI startup(): 62 15389 efa-od-az-1a-dy-c7i-48xlarge-1 {14}
[0] MPI startup(): 63 15393 efa-od-az-1a-dy-c7i-48xlarge-1 {15}
[0] MPI startup(): 64 15395 efa-od-az-1a-dy-c7i-48xlarge-1 {16}
[0] MPI startup(): 65 15396 efa-od-az-1a-dy-c7i-48xlarge-1 {17}
[0] MPI startup(): 66 15402 efa-od-az-1a-dy-c7i-48xlarge-1 {18}
[0] MPI startup(): 67 15403 efa-od-az-1a-dy-c7i-48xlarge-1 {19}
[0] MPI startup(): 68 15404 efa-od-az-1a-dy-c7i-48xlarge-1 {20}
[0] MPI startup(): 69 15405 efa-od-az-1a-dy-c7i-48xlarge-1 {21}
[0] MPI startup(): 70 15406 efa-od-az-1a-dy-c7i-48xlarge-1 {22}
[0] MPI startup(): 71 15410 efa-od-az-1a-dy-c7i-48xlarge-1 {23}
[0] MPI startup(): 72 15413 efa-od-az-1a-dy-c7i-48xlarge-1 {24}
[0] MPI startup(): 73 15414 efa-od-az-1a-dy-c7i-48xlarge-1 {25}
[0] MPI startup(): 74 15417 efa-od-az-1a-dy-c7i-48xlarge-1 {26}
[0] MPI startup(): 75 15418 efa-od-az-1a-dy-c7i-48xlarge-1 {27}
[0] MPI startup(): 76 15420 efa-od-az-1a-dy-c7i-48xlarge-1 {28}
[0] MPI startup(): 77 15421 efa-od-az-1a-dy-c7i-48xlarge-1 {29}
[0] MPI startup(): 78 15422 efa-od-az-1a-dy-c7i-48xlarge-1 {30}
[0] MPI startup(): 79 15423 efa-od-az-1a-dy-c7i-48xlarge-1 {31}
[0] MPI startup(): 80 15424 efa-od-az-1a-dy-c7i-48xlarge-1 {32}
[0] MPI startup(): 81 15425 efa-od-az-1a-dy-c7i-48xlarge-1 {33}
[0] MPI startup(): 82 15426 efa-od-az-1a-dy-c7i-48xlarge-1 {34}
[0] MPI startup(): 83 15427 efa-od-az-1a-dy-c7i-48xlarge-1 {35}
[0] MPI startup(): 84 15428 efa-od-az-1a-dy-c7i-48xlarge-1 {36}
[0] MPI startup(): 85 15429 efa-od-az-1a-dy-c7i-48xlarge-1 {37}
[0] MPI startup(): 86 15430 efa-od-az-1a-dy-c7i-48xlarge-1 {38}
[0] MPI startup(): 87 15431 efa-od-az-1a-dy-c7i-48xlarge-1 {39}
[0] MPI startup(): 88 15432 efa-od-az-1a-dy-c7i-48xlarge-1 {40}
[0] MPI startup(): 89 15433 efa-od-az-1a-dy-c7i-48xlarge-1 {41}
[0] MPI startup(): 90 15434 efa-od-az-1a-dy-c7i-48xlarge-1 {42}
[0] MPI startup(): 91 15435 efa-od-az-1a-dy-c7i-48xlarge-1 {43}
[0] MPI startup(): 92 15436 efa-od-az-1a-dy-c7i-48xlarge-1 {44}
[0] MPI startup(): 93 15437 efa-od-az-1a-dy-c7i-48xlarge-1 {45}
[0] MPI startup(): 94 15438 efa-od-az-1a-dy-c7i-48xlarge-1 {46}
[0] MPI startup(): 95 15439 efa-od-az-1a-dy-c7i-48xlarge-1 {47}
[0] MPI startup(): 96 15440 efa-od-az-1a-dy-c7i-48xlarge-1 {48}
[0] MPI startup(): 97 15441 efa-od-az-1a-dy-c7i-48xlarge-1 {49}
[0] MPI startup(): 98 15442 efa-od-az-1a-dy-c7i-48xlarge-1 {50}
[0] MPI startup(): 99 15443 efa-od-az-1a-dy-c7i-48xlarge-1 {51}
[0] MPI startup(): 100 15444 efa-od-az-1a-dy-c7i-48xlarge-1 {52}
[0] MPI startup(): 101 15445 efa-od-az-1a-dy-c7i-48xlarge-1 {53}
[0] MPI startup(): 102 15446 efa-od-az-1a-dy-c7i-48xlarge-1 {54}
[0] MPI startup(): 103 15447 efa-od-az-1a-dy-c7i-48xlarge-1 {55}
[0] MPI startup(): 104 15448 efa-od-az-1a-dy-c7i-48xlarge-1 {56}
[0] MPI startup(): 105 15450 efa-od-az-1a-dy-c7i-48xlarge-1 {57}
[0] MPI startup(): 106 15451 efa-od-az-1a-dy-c7i-48xlarge-1 {58}
[0] MPI startup(): 107 15452 efa-od-az-1a-dy-c7i-48xlarge-1 {59}
[0] MPI startup(): 108 15453 efa-od-az-1a-dy-c7i-48xlarge-1 {60}
[0] MPI startup(): 109 15455 efa-od-az-1a-dy-c7i-48xlarge-1 {61}
[0] MPI startup(): 110 15456 efa-od-az-1a-dy-c7i-48xlarge-1 {62}
[0] MPI startup(): 111 15457 efa-od-az-1a-dy-c7i-48xlarge-1 {63}
[0] MPI startup(): 112 15458 efa-od-az-1a-dy-c7i-48xlarge-1 {64}
[0] MPI startup(): 113 15459 efa-od-az-1a-dy-c7i-48xlarge-1 {65}
[0] MPI startup(): 114 15460 efa-od-az-1a-dy-c7i-48xlarge-1 {66}
[0] MPI startup(): 115 15461 efa-od-az-1a-dy-c7i-48xlarge-1 {67}
[0] MPI startup(): 116 15462 efa-od-az-1a-dy-c7i-48xlarge-1 {68}
[0] MPI startup(): 117 15463 efa-od-az-1a-dy-c7i-48xlarge-1 {69}
[0] MPI startup(): 118 15464 efa-od-az-1a-dy-c7i-48xlarge-1 {70}
[0] MPI startup(): 119 15465 efa-od-az-1a-dy-c7i-48xlarge-1 {71}
[0] MPI startup(): 120 15466 efa-od-az-1a-dy-c7i-48xlarge-1 {72}
[0] MPI startup(): 121 15467 efa-od-az-1a-dy-c7i-48xlarge-1 {73}
[0] MPI startup(): 122 15468 efa-od-az-1a-dy-c7i-48xlarge-1 {74}
[0] MPI startup(): 123 15469 efa-od-az-1a-dy-c7i-48xlarge-1 {75}
[0] MPI startup(): 124 15470 efa-od-az-1a-dy-c7i-48xlarge-1 {76}
[0] MPI startup(): 125 15471 efa-od-az-1a-dy-c7i-48xlarge-1 {77}
[0] MPI startup(): 126 15472 efa-od-az-1a-dy-c7i-48xlarge-1 {78}
[0] MPI startup(): 127 15473 efa-od-az-1a-dy-c7i-48xlarge-1 {79}
[0] MPI startup(): 128 15474 efa-od-az-1a-dy-c7i-48xlarge-1 {80}
[0] MPI startup(): 129 15475 efa-od-az-1a-dy-c7i-48xlarge-1 {81}
[0] MPI startup(): 130 15478 efa-od-az-1a-dy-c7i-48xlarge-1 {82}
[0] MPI startup(): 131 15479 efa-od-az-1a-dy-c7i-48xlarge-1 {83}
[0] MPI startup(): 132 15484 efa-od-az-1a-dy-c7i-48xlarge-1 {84}
[0] MPI startup(): 133 15487 efa-od-az-1a-dy-c7i-48xlarge-1 {85}
[0] MPI startup(): 134 15490 efa-od-az-1a-dy-c7i-48xlarge-1 {86}
[0] MPI startup(): 135 15495 efa-od-az-1a-dy-c7i-48xlarge-1 {87}
[0] MPI startup(): 136 15497 efa-od-az-1a-dy-c7i-48xlarge-1 {88}
[0] MPI startup(): 137 15501 efa-od-az-1a-dy-c7i-48xlarge-1 {89}
[0] MPI startup(): 138 15509 efa-od-az-1a-dy-c7i-48xlarge-1 {90}
[0] MPI startup(): 139 15515 efa-od-az-1a-dy-c7i-48xlarge-1 {91}
[0] MPI startup(): 140 15516 efa-od-az-1a-dy-c7i-48xlarge-1 {92}
[0] MPI startup(): 141 15521 efa-od-az-1a-dy-c7i-48xlarge-1 {93}
[0] MPI startup(): 142 15529 efa-od-az-1a-dy-c7i-48xlarge-1 {94}
[0] MPI startup(): 143 15535 efa-od-az-1a-dy-c7i-48xlarge-1 {95}
[0] MPI startup(): 144 15547 efa-od-az-1a-dy-c7i-48xlarge-1 {0}
[0] MPI startup(): 145 15548 efa-od-az-1a-dy-c7i-48xlarge-1 {1}
[0] MPI startup(): 146 15549 efa-od-az-1a-dy-c7i-48xlarge-1 {2}
[0] MPI startup(): 147 15550 efa-od-az-1a-dy-c7i-48xlarge-1 {3}
[0] MPI startup(): 148 15551 efa-od-az-1a-dy-c7i-48xlarge-1 {4}
[0] MPI startup(): 149 15552 efa-od-az-1a-dy-c7i-48xlarge-1 {5}
[0] MPI startup(): 150 15553 efa-od-az-1a-dy-c7i-48xlarge-1 {6}
[0] MPI startup(): 151 15554 efa-od-az-1a-dy-c7i-48xlarge-1 {7}
[0] MPI startup(): 152 15555 efa-od-az-1a-dy-c7i-48xlarge-1 {8}
[0] MPI startup(): 153 15556 efa-od-az-1a-dy-c7i-48xlarge-1 {9}
[0] MPI startup(): 154 15557 efa-od-az-1a-dy-c7i-48xlarge-1 {10}
[0] MPI startup(): 155 15558 efa-od-az-1a-dy-c7i-48xlarge-1 {11}
[0] MPI startup(): 156 15559 efa-od-az-1a-dy-c7i-48xlarge-1 {12}
[0] MPI startup(): 157 15560 efa-od-az-1a-dy-c7i-48xlarge-1 {13}
[0] MPI startup(): 158 15561 efa-od-az-1a-dy-c7i-48xlarge-1 {14}
[0] MPI startup(): 159 15562 efa-od-az-1a-dy-c7i-48xlarge-1 {15}
[0] MPI startup(): 160 15563 efa-od-az-1a-dy-c7i-48xlarge-1 {16}
[0] MPI startup(): 161 15564 efa-od-az-1a-dy-c7i-48xlarge-1 {17}
[0] MPI startup(): 162 15565 efa-od-az-1a-dy-c7i-48xlarge-1 {18}
[0] MPI startup(): 163 15566 efa-od-az-1a-dy-c7i-48xlarge-1 {19}
[0] MPI startup(): 164 15567 efa-od-az-1a-dy-c7i-48xlarge-1 {20}
[0] MPI startup(): 165 15568 efa-od-az-1a-dy-c7i-48xlarge-1 {21}
[0] MPI startup(): 166 15569 efa-od-az-1a-dy-c7i-48xlarge-1 {22}
[0] MPI startup(): 167 15570 efa-od-az-1a-dy-c7i-48xlarge-1 {23}
[0] MPI startup(): 168 15572 efa-od-az-1a-dy-c7i-48xlarge-1 {24}
[0] MPI startup(): 169 15573 efa-od-az-1a-dy-c7i-48xlarge-1 {25}
[0] MPI startup(): 170 15574 efa-od-az-1a-dy-c7i-48xlarge-1 {26}
[0] MPI startup(): 171 15575 efa-od-az-1a-dy-c7i-48xlarge-1 {27}
[0] MPI startup(): 172 15576 efa-od-az-1a-dy-c7i-48xlarge-1 {28}
[0] MPI startup(): 173 15577 efa-od-az-1a-dy-c7i-48xlarge-1 {29}
[0] MPI startup(): 174 15578 efa-od-az-1a-dy-c7i-48xlarge-1 {30}
[0] MPI startup(): 175 15579 efa-od-az-1a-dy-c7i-48xlarge-1 {31}
[0] MPI startup(): 176 15580 efa-od-az-1a-dy-c7i-48xlarge-1 {32}
[0] MPI startup(): 177 15581 efa-od-az-1a-dy-c7i-48xlarge-1 {33}
[0] MPI startup(): 178 15582 efa-od-az-1a-dy-c7i-48xlarge-1 {34}
[0] MPI startup(): 179 15583 efa-od-az-1a-dy-c7i-48xlarge-1 {35}
[0] MPI startup(): 180 15584 efa-od-az-1a-dy-c7i-48xlarge-1 {36}
[0] MPI startup(): 181 15585 efa-od-az-1a-dy-c7i-48xlarge-1 {37}
[0] MPI startup(): 182 15586 efa-od-az-1a-dy-c7i-48xlarge-1 {38}
[0] MPI startup(): 183 15587 efa-od-az-1a-dy-c7i-48xlarge-1 {39}
[0] MPI startup(): 184 15588 efa-od-az-1a-dy-c7i-48xlarge-1 {40}
[0] MPI startup(): 185 15589 efa-od-az-1a-dy-c7i-48xlarge-1 {41}
[0] MPI startup(): 186 15590 efa-od-az-1a-dy-c7i-48xlarge-1 {42}
[0] MPI startup(): 187 15591 efa-od-az-1a-dy-c7i-48xlarge-1 {43}
[0] MPI startup(): 188 15592 efa-od-az-1a-dy-c7i-48xlarge-1 {44}
[0] MPI startup(): 189 15593 efa-od-az-1a-dy-c7i-48xlarge-1 {45}
[0] MPI startup(): 190 15594 efa-od-az-1a-dy-c7i-48xlarge-1 {46}
[0] MPI startup(): 191 15595 efa-od-az-1a-dy-c7i-48xlarge-1 {47}
[0] MPI startup(): I_MPI_OFI_LIBRARY_INTERNAL=1
[0] MPI startup(): I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.14
[0] MPI startup(): I_MPI_MPIRUN=mpirun
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_FABRICS=shm:ofi
[0] MPI startup(): I_MPI_OFI_PROVIDER=efa
[0] MPI startup(): I_MPI_DEBUG=10
#----------------------------------------------------------------
# Intel(R) MPI Benchmarks 2021.8, MPI-1 part
#----------------------------------------------------------------
# Date : Mon Feb 17 11:30:57 2025
# Machine : x86_64
# System : Linux
# Release : 5.14.0-427.26.1.el9_4.x86_64
# Version : #1 SMP PREEMPT_DYNAMIC Fri Jul 5 11:34:54 EDT 2024
# MPI Version : 3.1
# MPI Thread Environment:

# Calling sequence was:

# IMB-MPI1 barrier

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# Barrier

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 2
# ( 190 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.37 0.37 0.37

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 4
# ( 188 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.60 0.60 0.60

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 8
# ( 184 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
376 0.68 0.68 0.68

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 16
# ( 176 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
283 25936.42 26010.59 25971.98

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 32
# ( 160 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
194 31180.34 31360.85 31291.58

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 64
# ( 128 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
196 43785.67 43984.83 43892.37

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 128
# ( 64 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
140 57157.02 57571.38 57406.40

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 192
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
109 52944.94 53403.62 53220.60

# All processes entering MPI_Finalize

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Mon, 17 Feb 2025 12:05:46 GMT

@TobiasK if you see the pinning information, 192 mpi ranks has been assigned to cpu 0-95. My expectation is that 192 mpi ranks should be assigned to cpu 0-191. Thanks!

# Calling sequence was:

# IMB-MPI1 barrier

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# Barrier

# All processes entering MPI_Finalize

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Mon, 17 Feb 2025 13:56:08 GMT

@TobiasK If you see the pinning information, 192 mpi ranks has been assigned to cpu 0-95 (two ranks per cpu). My expectation is that 192 mpi ranks should be assigned to cpu 0-191.

Please find the output of 'mpirun IMB-MPI1 barrier' in the attachment. Thanks!

Re: IntelMPI cpuinfo showing one thread per core

TobiasK — Mon, 17 Feb 2025 17:32:54 GMT

Thank you for the output. I see slurm is used, how is slurm configured? Can you provide all Slurm environment variables, e.g.

export | grep SLURM

do you see the same pinning issue if you use srun instead of mpirun?

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Mon, 17 Feb 2025 18:46:10 GMT

@TobiasK Thanks!

Yes, I see the same behavior when i use srun to stress all the 192 cpus, but from the htop i see only 50 percent of total cpu's (96 cpu's) went to 100 percent utilization. Is this due to slurm configuration issue?

command used: srun -n 192 -p efa-od-az-1a -w efa-od-az-1a-dy-c7i-48xlarge-1 --pty /usr/bin/stress --cpu 1 --timeout 10

Re: IntelMPI cpuinfo showing one thread per core

TobiasK — Mon, 17 Feb 2025 20:09:09 GMT

@mgHarish most likely yes.

Can you try with:

I_MPI_PIN_RESPECT_CPUSET=0 I_MPI_DEBUG=10 mpirun -np 192 IMB-MPI1 -npmin 192 barrier

What does

scontrol show node efa-od-az-1a-dy-c7i-48xlarge-1

show?

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Tue, 18 Feb 2025 11:05:21 GMT

@TobiasK I get the same behavior with the command

I_MPI_PIN_RESPECT_CPUSET=0 I_MPI_DEBUG=10 mpirun -np 192 IMB-MPI1 -npmin 192 barrier

But when I tried ssh into the node and ran the same command, I get the desired output (both will be in the attachments).

Please find the output of scontrol show node command while the below command is running

srun -n 192 -p efa-od-az-1a -w efa-od-az-1a-dy-c7i-48xlarge-1 --pty /usr/bin/stress --cpu 1 --timeout 300

Thanks

Re: IntelMPI cpuinfo showing one thread per core

mgHarish — Wed, 26 Feb 2025 12:46:25 GMT

@TobiasK I found the cause of the issue. By, default slurm was configured with 1 thread per core, when I configured with 2 threads per core as suggested in (3.9.0‐3.9.1) Default ThreadsPerCore Slurm setting causes reduced CPU utilization · aws/aws-parallelcluster Wiki · GitHub, Intel MPI also shows threads per core equals 2.

Help me to understand why the change in slurm configuration has resolved my issue. Exactly how slurm and intelmpi is linked?