Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2248 Discussions

IntelMPI cpuinfo showing one thread per core

mgHarish
Beginner
‎02-11-2025 01:45 PM
1,211 Views

My machine has 96 Physical Cores (192 logical cpu), but the cpuinfo comes with intelmpi show threads per core = 1.

lscpu

mgHarish_0-1739310233625.png

 

cpuinfo (intelmpi)

mgHarish_1-1739310285182.png

 

Because of this I was not able to pin processors to the logical cpus.

0 Kudos

Link Copied

11 Replies
TobiasK
Moderator
‎02-17-2025 02:36 AM
1,099 Views

@mgHarish thanks for reporting that, however, it would be really helpful if you also provide your OS version and the version of Intel MPI used to generate this output.

0 Kudos
Copy link
mgHarish
Beginner
‎02-17-2025 02:54 AM
1,090 Views
In response to TobiasK

Hi @TobiasK, Thank for the response!

OS Version

mgHarish_0-1739789453216.png

IntelMPI Version

mgHarish_1-1739789616973.png

 

 

In response to TobiasK
0 Kudos
Copy link
TobiasK
Moderator
‎02-17-2025 03:15 AM
1,084 Views

@mgHarish thanks, can you please provide the output of?

export I_MPI_DEBUG=10
mpirun IMB-MPI1 barrier

 

0 Kudos
Copy link
mgHarish
Beginner
‎02-17-2025 03:38 AM
879 Views
In response to TobiasK

@TobiasK If you see the pinning information, each pin cpu {0-95} has two mpi ranks. My expectation is pin cpu should be 0-191 with one mpi ranks on each cpu. Thanks!

[0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20241121 (id: e7829d6)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric loaded: libfabric.so.1
[0] MPI startup(): libfabric version: 1.21.0-impi
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): libfabric provider: efa
[0] MPI startup(): shm segment size (150 MB per rank) * (192 local ranks) = 28870 MB total
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa_100_x1.dat" not found
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa_100.dat" not found
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa.dat" not found
[0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: async_progress coll split: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): tag bits available: 30 (TAG_UB value: 1073741823)
[0] MPI startup(): source bits available: 30 (Maximal number of rank: 1073741823)
[0] MPI startup(): Number of NICs: 1
[0] MPI startup(): ===== NIC pinning on efa-od-az-1a-dy-c7i-48xlarge-1 =====
[0] MPI startup(): Rank Thread id Pin nic Nic id
[0] MPI startup(): 0 0 rdmap150s0-rdm 0
[0] MPI startup(): 1 0 rdmap150s0-rdm 0
[0] MPI startup(): 2 0 rdmap150s0-rdm 0
[0] MPI startup(): 3 0 rdmap150s0-rdm 0
[0] MPI startup(): 4 0 rdmap150s0-rdm 0
[0] MPI startup(): 5 0 rdmap150s0-rdm 0
[0] MPI startup(): 6 0 rdmap150s0-rdm 0
[0] MPI startup(): 7 0 rdmap150s0-rdm 0
[0] MPI startup(): 8 0 rdmap150s0-rdm 0
[0] MPI startup(): 9 0 rdmap150s0-rdm 0
[0] MPI startup(): 10 0 rdmap150s0-rdm 0
[0] MPI startup(): 11 0 rdmap150s0-rdm 0
[0] MPI startup(): 12 0 rdmap150s0-rdm 0
[0] MPI startup(): 13 0 rdmap150s0-rdm 0
[0] MPI startup(): 14 0 rdmap150s0-rdm 0
[0] MPI startup(): 15 0 rdmap150s0-rdm 0
[0] MPI startup(): 16 0 rdmap150s0-rdm 0
[0] MPI startup(): 17 0 rdmap150s0-rdm 0
[0] MPI startup(): 18 0 rdmap150s0-rdm 0
[0] MPI startup(): 19 0 rdmap150s0-rdm 0
[0] MPI startup(): 20 0 rdmap150s0-rdm 0
[0] MPI startup(): 21 0 rdmap150s0-rdm 0
[0] MPI startup(): 22 0 rdmap150s0-rdm 0
[0] MPI startup(): 23 0 rdmap150s0-rdm 0
[0] MPI startup(): 24 0 rdmap150s0-rdm 0
[0] MPI startup(): 25 0 rdmap150s0-rdm 0
[0] MPI startup(): 26 0 rdmap150s0-rdm 0
[0] MPI startup(): 27 0 rdmap150s0-rdm 0
[0] MPI startup(): 28 0 rdmap150s0-rdm 0
[0] MPI startup(): 29 0 rdmap150s0-rdm 0
[0] MPI startup(): 30 0 rdmap150s0-rdm 0
[0] MPI startup(): 31 0 rdmap150s0-rdm 0
[0] MPI startup(): 32 0 rdmap150s0-rdm 0
[0] MPI startup(): 33 0 rdmap150s0-rdm 0
[0] MPI startup(): 34 0 rdmap150s0-rdm 0
[0] MPI startup(): 35 0 rdmap150s0-rdm 0
[0] MPI startup(): 36 0 rdmap150s0-rdm 0
[0] MPI startup(): 37 0 rdmap150s0-rdm 0
[0] MPI startup(): 38 0 rdmap150s0-rdm 0
[0] MPI startup(): 39 0 rdmap150s0-rdm 0
[0] MPI startup(): 40 0 rdmap150s0-rdm 0
[0] MPI startup(): 41 0 rdmap150s0-rdm 0
[0] MPI startup(): 42 0 rdmap150s0-rdm 0
[0] MPI startup(): 43 0 rdmap150s0-rdm 0
[0] MPI startup(): 44 0 rdmap150s0-rdm 0
[0] MPI startup(): 45 0 rdmap150s0-rdm 0
[0] MPI startup(): 46 0 rdmap150s0-rdm 0
[0] MPI startup(): 47 0 rdmap150s0-rdm 0
[0] MPI startup(): 48 0 rdmap150s0-rdm 0
[0] MPI startup(): 49 0 rdmap150s0-rdm 0
[0] MPI startup(): 50 0 rdmap150s0-rdm 0
[0] MPI startup(): 51 0 rdmap150s0-rdm 0
[0] MPI startup(): 52 0 rdmap150s0-rdm 0
[0] MPI startup(): 53 0 rdmap150s0-rdm 0
[0] MPI startup(): 54 0 rdmap150s0-rdm 0
[0] MPI startup(): 55 0 rdmap150s0-rdm 0
[0] MPI startup(): 56 0 rdmap150s0-rdm 0
[0] MPI startup(): 57 0 rdmap150s0-rdm 0
[0] MPI startup(): 58 0 rdmap150s0-rdm 0
[0] MPI startup(): 59 0 rdmap150s0-rdm 0
[0] MPI startup(): 60 0 rdmap150s0-rdm 0
[0] MPI startup(): 61 0 rdmap150s0-rdm 0
[0] MPI startup(): 62 0 rdmap150s0-rdm 0
[0] MPI startup(): 63 0 rdmap150s0-rdm 0
[0] MPI startup(): 64 0 rdmap150s0-rdm 0
[0] MPI startup(): 65 0 rdmap150s0-rdm 0
[0] MPI startup(): 66 0 rdmap150s0-rdm 0
[0] MPI startup(): 67 0 rdmap150s0-rdm 0
[0] MPI startup(): 68 0 rdmap150s0-rdm 0
[0] MPI startup(): 69 0 rdmap150s0-rdm 0
[0] MPI startup(): 70 0 rdmap150s0-rdm 0
[0] MPI startup(): 71 0 rdmap150s0-rdm 0
[0] MPI startup(): 72 0 rdmap150s0-rdm 0
[0] MPI startup(): 73 0 rdmap150s0-rdm 0
[0] MPI startup(): 74 0 rdmap150s0-rdm 0
[0] MPI startup(): 75 0 rdmap150s0-rdm 0
[0] MPI startup(): 76 0 rdmap150s0-rdm 0
[0] MPI startup(): 77 0 rdmap150s0-rdm 0
[0] MPI startup(): 78 0 rdmap150s0-rdm 0
[0] MPI startup(): 79 0 rdmap150s0-rdm 0
[0] MPI startup(): 80 0 rdmap150s0-rdm 0
[0] MPI startup(): 81 0 rdmap150s0-rdm 0
[0] MPI startup(): 82 0 rdmap150s0-rdm 0
[0] MPI startup(): 83 0 rdmap150s0-rdm 0
[0] MPI startup(): 84 0 rdmap150s0-rdm 0
[0] MPI startup(): 85 0 rdmap150s0-rdm 0
[0] MPI startup(): 86 0 rdmap150s0-rdm 0
[0] MPI startup(): 87 0 rdmap150s0-rdm 0
[0] MPI startup(): 88 0 rdmap150s0-rdm 0
[0] MPI startup(): 89 0 rdmap150s0-rdm 0
[0] MPI startup(): 90 0 rdmap150s0-rdm 0
[0] MPI startup(): 91 0 rdmap150s0-rdm 0
[0] MPI startup(): 92 0 rdmap150s0-rdm 0
[0] MPI startup(): 93 0 rdmap150s0-rdm 0
[0] MPI startup(): 94 0 rdmap150s0-rdm 0
[0] MPI startup(): 95 0 rdmap150s0-rdm 0
[0] MPI startup(): 96 0 rdmap150s0-rdm 0
[0] MPI startup(): 97 0 rdmap150s0-rdm 0
[0] MPI startup(): 98 0 rdmap150s0-rdm 0
[0] MPI startup(): 99 0 rdmap150s0-rdm 0
[0] MPI startup(): 100 0 rdmap150s0-rdm 0
[0] MPI startup(): 101 0 rdmap150s0-rdm 0
[0] MPI startup(): 102 0 rdmap150s0-rdm 0
[0] MPI startup(): 103 0 rdmap150s0-rdm 0
[0] MPI startup(): 104 0 rdmap150s0-rdm 0
[0] MPI startup(): 105 0 rdmap150s0-rdm 0
[0] MPI startup(): 106 0 rdmap150s0-rdm 0
[0] MPI startup(): 107 0 rdmap150s0-rdm 0
[0] MPI startup(): 108 0 rdmap150s0-rdm 0
[0] MPI startup(): 109 0 rdmap150s0-rdm 0
[0] MPI startup(): 110 0 rdmap150s0-rdm 0
[0] MPI startup(): 111 0 rdmap150s0-rdm 0
[0] MPI startup(): 112 0 rdmap150s0-rdm 0
[0] MPI startup(): 113 0 rdmap150s0-rdm 0
[0] MPI startup(): 114 0 rdmap150s0-rdm 0
[0] MPI startup(): 115 0 rdmap150s0-rdm 0
[0] MPI startup(): 116 0 rdmap150s0-rdm 0
[0] MPI startup(): 117 0 rdmap150s0-rdm 0
[0] MPI startup(): 118 0 rdmap150s0-rdm 0
[0] MPI startup(): 119 0 rdmap150s0-rdm 0
[0] MPI startup(): 120 0 rdmap150s0-rdm 0
[0] MPI startup(): 121 0 rdmap150s0-rdm 0
[0] MPI startup(): 122 0 rdmap150s0-rdm 0
[0] MPI startup(): 123 0 rdmap150s0-rdm 0
[0] MPI startup(): 124 0 rdmap150s0-rdm 0
[0] MPI startup(): 125 0 rdmap150s0-rdm 0
[0] MPI startup(): 126 0 rdmap150s0-rdm 0
[0] MPI startup(): 127 0 rdmap150s0-rdm 0
[0] MPI startup(): 128 0 rdmap150s0-rdm 0
[0] MPI startup(): 129 0 rdmap150s0-rdm 0
[0] MPI startup(): 130 0 rdmap150s0-rdm 0
[0] MPI startup(): 131 0 rdmap150s0-rdm 0
[0] MPI startup(): 132 0 rdmap150s0-rdm 0
[0] MPI startup(): 133 0 rdmap150s0-rdm 0
[0] MPI startup(): 134 0 rdmap150s0-rdm 0
[0] MPI startup(): 135 0 rdmap150s0-rdm 0
[0] MPI startup(): 136 0 rdmap150s0-rdm 0
[0] MPI startup(): 137 0 rdmap150s0-rdm 0
[0] MPI startup(): 138 0 rdmap150s0-rdm 0
[0] MPI startup(): 139 0 rdmap150s0-rdm 0
[0] MPI startup(): 140 0 rdmap150s0-rdm 0
[0] MPI startup(): 141 0 rdmap150s0-rdm 0
[0] MPI startup(): 142 0 rdmap150s0-rdm 0
[0] MPI startup(): 143 0 rdmap150s0-rdm 0
[0] MPI startup(): 144 0 rdmap150s0-rdm 0
[0] MPI startup(): 145 0 rdmap150s0-rdm 0
[0] MPI startup(): 146 0 rdmap150s0-rdm 0
[0] MPI startup(): 147 0 rdmap150s0-rdm 0
[0] MPI startup(): 148 0 rdmap150s0-rdm 0
[0] MPI startup(): 149 0 rdmap150s0-rdm 0
[0] MPI startup(): 150 0 rdmap150s0-rdm 0
[0] MPI startup(): 151 0 rdmap150s0-rdm 0
[0] MPI startup(): 152 0 rdmap150s0-rdm 0
[0] MPI startup(): 153 0 rdmap150s0-rdm 0
[0] MPI startup(): 154 0 rdmap150s0-rdm 0
[0] MPI startup(): 155 0 rdmap150s0-rdm 0
[0] MPI startup(): 156 0 rdmap150s0-rdm 0
[0] MPI startup(): 157 0 rdmap150s0-rdm 0
[0] MPI startup(): 158 0 rdmap150s0-rdm 0
[0] MPI startup(): 159 0 rdmap150s0-rdm 0
[0] MPI startup(): 160 0 rdmap150s0-rdm 0
[0] MPI startup(): 161 0 rdmap150s0-rdm 0
[0] MPI startup(): 162 0 rdmap150s0-rdm 0
[0] MPI startup(): 163 0 rdmap150s0-rdm 0
[0] MPI startup(): 164 0 rdmap150s0-rdm 0
[0] MPI startup(): 165 0 rdmap150s0-rdm 0
[0] MPI startup(): 166 0 rdmap150s0-rdm 0
[0] MPI startup(): 167 0 rdmap150s0-rdm 0
[0] MPI startup(): 168 0 rdmap150s0-rdm 0
[0] MPI startup(): 169 0 rdmap150s0-rdm 0
[0] MPI startup(): 170 0 rdmap150s0-rdm 0
[0] MPI startup(): 171 0 rdmap150s0-rdm 0
[0] MPI startup(): 172 0 rdmap150s0-rdm 0
[0] MPI startup(): 173 0 rdmap150s0-rdm 0
[0] MPI startup(): 174 0 rdmap150s0-rdm 0
[0] MPI startup(): 175 0 rdmap150s0-rdm 0
[0] MPI startup(): 176 0 rdmap150s0-rdm 0
[0] MPI startup(): 177 0 rdmap150s0-rdm 0
[0] MPI startup(): 178 0 rdmap150s0-rdm 0
[0] MPI startup(): 179 0 rdmap150s0-rdm 0
[0] MPI startup(): 180 0 rdmap150s0-rdm 0
[0] MPI startup(): 181 0 rdmap150s0-rdm 0
[0] MPI startup(): 182 0 rdmap150s0-rdm 0
[0] MPI startup(): 183 0 rdmap150s0-rdm 0
[0] MPI startup(): 184 0 rdmap150s0-rdm 0
[0] MPI startup(): 185 0 rdmap150s0-rdm 0
[0] MPI startup(): 186 0 rdmap150s0-rdm 0
[0] MPI startup(): 187 0 rdmap150s0-rdm 0
[0] MPI startup(): 188 0 rdmap150s0-rdm 0
[0] MPI startup(): 189 0 rdmap150s0-rdm 0
[0] MPI startup(): 190 0 rdmap150s0-rdm 0
[0] MPI startup(): 191 0 rdmap150s0-rdm 0
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 15233 efa-od-az-1a-dy-c7i-48xlarge-1 {48}
[0] MPI startup(): 1 15234 efa-od-az-1a-dy-c7i-48xlarge-1 {49}
[0] MPI startup(): 2 15235 efa-od-az-1a-dy-c7i-48xlarge-1 {50}
[0] MPI startup(): 3 15236 efa-od-az-1a-dy-c7i-48xlarge-1 {51}
[0] MPI startup(): 4 15237 efa-od-az-1a-dy-c7i-48xlarge-1 {52}
[0] MPI startup(): 5 15238 efa-od-az-1a-dy-c7i-48xlarge-1 {53}
[0] MPI startup(): 6 15239 efa-od-az-1a-dy-c7i-48xlarge-1 {54}
[0] MPI startup(): 7 15240 efa-od-az-1a-dy-c7i-48xlarge-1 {55}
[0] MPI startup(): 8 15241 efa-od-az-1a-dy-c7i-48xlarge-1 {56}
[0] MPI startup(): 9 15242 efa-od-az-1a-dy-c7i-48xlarge-1 {57}
[0] MPI startup(): 10 15243 efa-od-az-1a-dy-c7i-48xlarge-1 {58}
[0] MPI startup(): 11 15244 efa-od-az-1a-dy-c7i-48xlarge-1 {59}
[0] MPI startup(): 12 15245 efa-od-az-1a-dy-c7i-48xlarge-1 {60}
[0] MPI startup(): 13 15250 efa-od-az-1a-dy-c7i-48xlarge-1 {61}
[0] MPI startup(): 14 15253 efa-od-az-1a-dy-c7i-48xlarge-1 {62}
[0] MPI startup(): 15 15254 efa-od-az-1a-dy-c7i-48xlarge-1 {63}
[0] MPI startup(): 16 15258 efa-od-az-1a-dy-c7i-48xlarge-1 {64}
[0] MPI startup(): 17 15265 efa-od-az-1a-dy-c7i-48xlarge-1 {65}
[0] MPI startup(): 18 15269 efa-od-az-1a-dy-c7i-48xlarge-1 {66}
[0] MPI startup(): 19 15270 efa-od-az-1a-dy-c7i-48xlarge-1 {67}
[0] MPI startup(): 20 15273 efa-od-az-1a-dy-c7i-48xlarge-1 {68}
[0] MPI startup(): 21 15276 efa-od-az-1a-dy-c7i-48xlarge-1 {69}
[0] MPI startup(): 22 15279 efa-od-az-1a-dy-c7i-48xlarge-1 {70}
[0] MPI startup(): 23 15285 efa-od-az-1a-dy-c7i-48xlarge-1 {71}
[0] MPI startup(): 24 15287 efa-od-az-1a-dy-c7i-48xlarge-1 {72}
[0] MPI startup(): 25 15293 efa-od-az-1a-dy-c7i-48xlarge-1 {73}
[0] MPI startup(): 26 15294 efa-od-az-1a-dy-c7i-48xlarge-1 {74}
[0] MPI startup(): 27 15297 efa-od-az-1a-dy-c7i-48xlarge-1 {75}
[0] MPI startup(): 28 15298 efa-od-az-1a-dy-c7i-48xlarge-1 {76}
[0] MPI startup(): 29 15299 efa-od-az-1a-dy-c7i-48xlarge-1 {77}
[0] MPI startup(): 30 15303 efa-od-az-1a-dy-c7i-48xlarge-1 {78}
[0] MPI startup(): 31 15304 efa-od-az-1a-dy-c7i-48xlarge-1 {79}
[0] MPI startup(): 32 15307 efa-od-az-1a-dy-c7i-48xlarge-1 {80}
[0] MPI startup(): 33 15311 efa-od-az-1a-dy-c7i-48xlarge-1 {81}
[0] MPI startup(): 34 15312 efa-od-az-1a-dy-c7i-48xlarge-1 {82}
[0] MPI startup(): 35 15314 efa-od-az-1a-dy-c7i-48xlarge-1 {83}
[0] MPI startup(): 36 15315 efa-od-az-1a-dy-c7i-48xlarge-1 {84}
[0] MPI startup(): 37 15316 efa-od-az-1a-dy-c7i-48xlarge-1 {85}
[0] MPI startup(): 38 15321 efa-od-az-1a-dy-c7i-48xlarge-1 {86}
[0] MPI startup(): 39 15325 efa-od-az-1a-dy-c7i-48xlarge-1 {87}
[0] MPI startup(): 40 15326 efa-od-az-1a-dy-c7i-48xlarge-1 {88}
[0] MPI startup(): 41 15328 efa-od-az-1a-dy-c7i-48xlarge-1 {89}
[0] MPI startup(): 42 15329 efa-od-az-1a-dy-c7i-48xlarge-1 {90}
[0] MPI startup(): 43 15330 efa-od-az-1a-dy-c7i-48xlarge-1 {91}
[0] MPI startup(): 44 15336 efa-od-az-1a-dy-c7i-48xlarge-1 {92}
[0] MPI startup(): 45 15337 efa-od-az-1a-dy-c7i-48xlarge-1 {93}
[0] MPI startup(): 46 15340 efa-od-az-1a-dy-c7i-48xlarge-1 {94}
[0] MPI startup(): 47 15344 efa-od-az-1a-dy-c7i-48xlarge-1 {95}
[0] MPI startup(): 48 15348 efa-od-az-1a-dy-c7i-48xlarge-1 {0}
[0] MPI startup(): 49 15349 efa-od-az-1a-dy-c7i-48xlarge-1 {1}
[0] MPI startup(): 50 15352 efa-od-az-1a-dy-c7i-48xlarge-1 {2}
[0] MPI startup(): 51 15356 efa-od-az-1a-dy-c7i-48xlarge-1 {3}
[0] MPI startup(): 52 15362 efa-od-az-1a-dy-c7i-48xlarge-1 {4}
[0] MPI startup(): 53 15364 efa-od-az-1a-dy-c7i-48xlarge-1 {5}
[0] MPI startup(): 54 15367 efa-od-az-1a-dy-c7i-48xlarge-1 {6}
[0] MPI startup(): 55 15372 efa-od-az-1a-dy-c7i-48xlarge-1 {7}
[0] MPI startup(): 56 15373 efa-od-az-1a-dy-c7i-48xlarge-1 {8}
[0] MPI startup(): 57 15377 efa-od-az-1a-dy-c7i-48xlarge-1 {9}
[0] MPI startup(): 58 15378 efa-od-az-1a-dy-c7i-48xlarge-1 {10}
[0] MPI startup(): 59 15381 efa-od-az-1a-dy-c7i-48xlarge-1 {11}
[0] MPI startup(): 60 15386 efa-od-az-1a-dy-c7i-48xlarge-1 {12}
[0] MPI startup(): 61 15387 efa-od-az-1a-dy-c7i-48xlarge-1 {13}
[0] MPI startup(): 62 15389 efa-od-az-1a-dy-c7i-48xlarge-1 {14}
[0] MPI startup(): 63 15393 efa-od-az-1a-dy-c7i-48xlarge-1 {15}
[0] MPI startup(): 64 15395 efa-od-az-1a-dy-c7i-48xlarge-1 {16}
[0] MPI startup(): 65 15396 efa-od-az-1a-dy-c7i-48xlarge-1 {17}
[0] MPI startup(): 66 15402 efa-od-az-1a-dy-c7i-48xlarge-1 {18}
[0] MPI startup(): 67 15403 efa-od-az-1a-dy-c7i-48xlarge-1 {19}
[0] MPI startup(): 68 15404 efa-od-az-1a-dy-c7i-48xlarge-1 {20}
[0] MPI startup(): 69 15405 efa-od-az-1a-dy-c7i-48xlarge-1 {21}
[0] MPI startup(): 70 15406 efa-od-az-1a-dy-c7i-48xlarge-1 {22}
[0] MPI startup(): 71 15410 efa-od-az-1a-dy-c7i-48xlarge-1 {23}
[0] MPI startup(): 72 15413 efa-od-az-1a-dy-c7i-48xlarge-1 {24}
[0] MPI startup(): 73 15414 efa-od-az-1a-dy-c7i-48xlarge-1 {25}
[0] MPI startup(): 74 15417 efa-od-az-1a-dy-c7i-48xlarge-1 {26}
[0] MPI startup(): 75 15418 efa-od-az-1a-dy-c7i-48xlarge-1 {27}
[0] MPI startup(): 76 15420 efa-od-az-1a-dy-c7i-48xlarge-1 {28}
[0] MPI startup(): 77 15421 efa-od-az-1a-dy-c7i-48xlarge-1 {29}
[0] MPI startup(): 78 15422 efa-od-az-1a-dy-c7i-48xlarge-1 {30}
[0] MPI startup(): 79 15423 efa-od-az-1a-dy-c7i-48xlarge-1 {31}
[0] MPI startup(): 80 15424 efa-od-az-1a-dy-c7i-48xlarge-1 {32}
[0] MPI startup(): 81 15425 efa-od-az-1a-dy-c7i-48xlarge-1 {33}
[0] MPI startup(): 82 15426 efa-od-az-1a-dy-c7i-48xlarge-1 {34}
[0] MPI startup(): 83 15427 efa-od-az-1a-dy-c7i-48xlarge-1 {35}
[0] MPI startup(): 84 15428 efa-od-az-1a-dy-c7i-48xlarge-1 {36}
[0] MPI startup(): 85 15429 efa-od-az-1a-dy-c7i-48xlarge-1 {37}
[0] MPI startup(): 86 15430 efa-od-az-1a-dy-c7i-48xlarge-1 {38}
[0] MPI startup(): 87 15431 efa-od-az-1a-dy-c7i-48xlarge-1 {39}
[0] MPI startup(): 88 15432 efa-od-az-1a-dy-c7i-48xlarge-1 {40}
[0] MPI startup(): 89 15433 efa-od-az-1a-dy-c7i-48xlarge-1 {41}
[0] MPI startup(): 90 15434 efa-od-az-1a-dy-c7i-48xlarge-1 {42}
[0] MPI startup(): 91 15435 efa-od-az-1a-dy-c7i-48xlarge-1 {43}
[0] MPI startup(): 92 15436 efa-od-az-1a-dy-c7i-48xlarge-1 {44}
[0] MPI startup(): 93 15437 efa-od-az-1a-dy-c7i-48xlarge-1 {45}
[0] MPI startup(): 94 15438 efa-od-az-1a-dy-c7i-48xlarge-1 {46}
[0] MPI startup(): 95 15439 efa-od-az-1a-dy-c7i-48xlarge-1 {47}
[0] MPI startup(): 96 15440 efa-od-az-1a-dy-c7i-48xlarge-1 {48}
[0] MPI startup(): 97 15441 efa-od-az-1a-dy-c7i-48xlarge-1 {49}
[0] MPI startup(): 98 15442 efa-od-az-1a-dy-c7i-48xlarge-1 {50}
[0] MPI startup(): 99 15443 efa-od-az-1a-dy-c7i-48xlarge-1 {51}
[0] MPI startup(): 100 15444 efa-od-az-1a-dy-c7i-48xlarge-1 {52}
[0] MPI startup(): 101 15445 efa-od-az-1a-dy-c7i-48xlarge-1 {53}
[0] MPI startup(): 102 15446 efa-od-az-1a-dy-c7i-48xlarge-1 {54}
[0] MPI startup(): 103 15447 efa-od-az-1a-dy-c7i-48xlarge-1 {55}
[0] MPI startup(): 104 15448 efa-od-az-1a-dy-c7i-48xlarge-1 {56}
[0] MPI startup(): 105 15450 efa-od-az-1a-dy-c7i-48xlarge-1 {57}
[0] MPI startup(): 106 15451 efa-od-az-1a-dy-c7i-48xlarge-1 {58}
[0] MPI startup(): 107 15452 efa-od-az-1a-dy-c7i-48xlarge-1 {59}
[0] MPI startup(): 108 15453 efa-od-az-1a-dy-c7i-48xlarge-1 {60}
[0] MPI startup(): 109 15455 efa-od-az-1a-dy-c7i-48xlarge-1 {61}
[0] MPI startup(): 110 15456 efa-od-az-1a-dy-c7i-48xlarge-1 {62}
[0] MPI startup(): 111 15457 efa-od-az-1a-dy-c7i-48xlarge-1 {63}
[0] MPI startup(): 112 15458 efa-od-az-1a-dy-c7i-48xlarge-1 {64}
[0] MPI startup(): 113 15459 efa-od-az-1a-dy-c7i-48xlarge-1 {65}
[0] MPI startup(): 114 15460 efa-od-az-1a-dy-c7i-48xlarge-1 {66}
[0] MPI startup(): 115 15461 efa-od-az-1a-dy-c7i-48xlarge-1 {67}
[0] MPI startup(): 116 15462 efa-od-az-1a-dy-c7i-48xlarge-1 {68}
[0] MPI startup(): 117 15463 efa-od-az-1a-dy-c7i-48xlarge-1 {69}
[0] MPI startup(): 118 15464 efa-od-az-1a-dy-c7i-48xlarge-1 {70}
[0] MPI startup(): 119 15465 efa-od-az-1a-dy-c7i-48xlarge-1 {71}
[0] MPI startup(): 120 15466 efa-od-az-1a-dy-c7i-48xlarge-1 {72}
[0] MPI startup(): 121 15467 efa-od-az-1a-dy-c7i-48xlarge-1 {73}
[0] MPI startup(): 122 15468 efa-od-az-1a-dy-c7i-48xlarge-1 {74}
[0] MPI startup(): 123 15469 efa-od-az-1a-dy-c7i-48xlarge-1 {75}
[0] MPI startup(): 124 15470 efa-od-az-1a-dy-c7i-48xlarge-1 {76}
[0] MPI startup(): 125 15471 efa-od-az-1a-dy-c7i-48xlarge-1 {77}
[0] MPI startup(): 126 15472 efa-od-az-1a-dy-c7i-48xlarge-1 {78}
[0] MPI startup(): 127 15473 efa-od-az-1a-dy-c7i-48xlarge-1 {79}
[0] MPI startup(): 128 15474 efa-od-az-1a-dy-c7i-48xlarge-1 {80}
[0] MPI startup(): 129 15475 efa-od-az-1a-dy-c7i-48xlarge-1 {81}
[0] MPI startup(): 130 15478 efa-od-az-1a-dy-c7i-48xlarge-1 {82}
[0] MPI startup(): 131 15479 efa-od-az-1a-dy-c7i-48xlarge-1 {83}
[0] MPI startup(): 132 15484 efa-od-az-1a-dy-c7i-48xlarge-1 {84}
[0] MPI startup(): 133 15487 efa-od-az-1a-dy-c7i-48xlarge-1 {85}
[0] MPI startup(): 134 15490 efa-od-az-1a-dy-c7i-48xlarge-1 {86}
[0] MPI startup(): 135 15495 efa-od-az-1a-dy-c7i-48xlarge-1 {87}
[0] MPI startup(): 136 15497 efa-od-az-1a-dy-c7i-48xlarge-1 {88}
[0] MPI startup(): 137 15501 efa-od-az-1a-dy-c7i-48xlarge-1 {89}
[0] MPI startup(): 138 15509 efa-od-az-1a-dy-c7i-48xlarge-1 {90}
[0] MPI startup(): 139 15515 efa-od-az-1a-dy-c7i-48xlarge-1 {91}
[0] MPI startup(): 140 15516 efa-od-az-1a-dy-c7i-48xlarge-1 {92}
[0] MPI startup(): 141 15521 efa-od-az-1a-dy-c7i-48xlarge-1 {93}
[0] MPI startup(): 142 15529 efa-od-az-1a-dy-c7i-48xlarge-1 {94}
[0] MPI startup(): 143 15535 efa-od-az-1a-dy-c7i-48xlarge-1 {95}
[0] MPI startup(): 144 15547 efa-od-az-1a-dy-c7i-48xlarge-1 {0}
[0] MPI startup(): 145 15548 efa-od-az-1a-dy-c7i-48xlarge-1 {1}
[0] MPI startup(): 146 15549 efa-od-az-1a-dy-c7i-48xlarge-1 {2}
[0] MPI startup(): 147 15550 efa-od-az-1a-dy-c7i-48xlarge-1 {3}
[0] MPI startup(): 148 15551 efa-od-az-1a-dy-c7i-48xlarge-1 {4}
[0] MPI startup(): 149 15552 efa-od-az-1a-dy-c7i-48xlarge-1 {5}
[0] MPI startup(): 150 15553 efa-od-az-1a-dy-c7i-48xlarge-1 {6}
[0] MPI startup(): 151 15554 efa-od-az-1a-dy-c7i-48xlarge-1 {7}
[0] MPI startup(): 152 15555 efa-od-az-1a-dy-c7i-48xlarge-1 {8}
[0] MPI startup(): 153 15556 efa-od-az-1a-dy-c7i-48xlarge-1 {9}
[0] MPI startup(): 154 15557 efa-od-az-1a-dy-c7i-48xlarge-1 {10}
[0] MPI startup(): 155 15558 efa-od-az-1a-dy-c7i-48xlarge-1 {11}
[0] MPI startup(): 156 15559 efa-od-az-1a-dy-c7i-48xlarge-1 {12}
[0] MPI startup(): 157 15560 efa-od-az-1a-dy-c7i-48xlarge-1 {13}
[0] MPI startup(): 158 15561 efa-od-az-1a-dy-c7i-48xlarge-1 {14}
[0] MPI startup(): 159 15562 efa-od-az-1a-dy-c7i-48xlarge-1 {15}
[0] MPI startup(): 160 15563 efa-od-az-1a-dy-c7i-48xlarge-1 {16}
[0] MPI startup(): 161 15564 efa-od-az-1a-dy-c7i-48xlarge-1 {17}
[0] MPI startup(): 162 15565 efa-od-az-1a-dy-c7i-48xlarge-1 {18}
[0] MPI startup(): 163 15566 efa-od-az-1a-dy-c7i-48xlarge-1 {19}
[0] MPI startup(): 164 15567 efa-od-az-1a-dy-c7i-48xlarge-1 {20}
[0] MPI startup(): 165 15568 efa-od-az-1a-dy-c7i-48xlarge-1 {21}
[0] MPI startup(): 166 15569 efa-od-az-1a-dy-c7i-48xlarge-1 {22}
[0] MPI startup(): 167 15570 efa-od-az-1a-dy-c7i-48xlarge-1 {23}
[0] MPI startup(): 168 15572 efa-od-az-1a-dy-c7i-48xlarge-1 {24}
[0] MPI startup(): 169 15573 efa-od-az-1a-dy-c7i-48xlarge-1 {25}
[0] MPI startup(): 170 15574 efa-od-az-1a-dy-c7i-48xlarge-1 {26}
[0] MPI startup(): 171 15575 efa-od-az-1a-dy-c7i-48xlarge-1 {27}
[0] MPI startup(): 172 15576 efa-od-az-1a-dy-c7i-48xlarge-1 {28}
[0] MPI startup(): 173 15577 efa-od-az-1a-dy-c7i-48xlarge-1 {29}
[0] MPI startup(): 174 15578 efa-od-az-1a-dy-c7i-48xlarge-1 {30}
[0] MPI startup(): 175 15579 efa-od-az-1a-dy-c7i-48xlarge-1 {31}
[0] MPI startup(): 176 15580 efa-od-az-1a-dy-c7i-48xlarge-1 {32}
[0] MPI startup(): 177 15581 efa-od-az-1a-dy-c7i-48xlarge-1 {33}
[0] MPI startup(): 178 15582 efa-od-az-1a-dy-c7i-48xlarge-1 {34}
[0] MPI startup(): 179 15583 efa-od-az-1a-dy-c7i-48xlarge-1 {35}
[0] MPI startup(): 180 15584 efa-od-az-1a-dy-c7i-48xlarge-1 {36}
[0] MPI startup(): 181 15585 efa-od-az-1a-dy-c7i-48xlarge-1 {37}
[0] MPI startup(): 182 15586 efa-od-az-1a-dy-c7i-48xlarge-1 {38}
[0] MPI startup(): 183 15587 efa-od-az-1a-dy-c7i-48xlarge-1 {39}
[0] MPI startup(): 184 15588 efa-od-az-1a-dy-c7i-48xlarge-1 {40}
[0] MPI startup(): 185 15589 efa-od-az-1a-dy-c7i-48xlarge-1 {41}
[0] MPI startup(): 186 15590 efa-od-az-1a-dy-c7i-48xlarge-1 {42}
[0] MPI startup(): 187 15591 efa-od-az-1a-dy-c7i-48xlarge-1 {43}
[0] MPI startup(): 188 15592 efa-od-az-1a-dy-c7i-48xlarge-1 {44}
[0] MPI startup(): 189 15593 efa-od-az-1a-dy-c7i-48xlarge-1 {45}
[0] MPI startup(): 190 15594 efa-od-az-1a-dy-c7i-48xlarge-1 {46}
[0] MPI startup(): 191 15595 efa-od-az-1a-dy-c7i-48xlarge-1 {47}
[0] MPI startup(): I_MPI_OFI_LIBRARY_INTERNAL=1
[0] MPI startup(): I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.14
[0] MPI startup(): I_MPI_MPIRUN=mpirun
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_FABRICS=shm:ofi
[0] MPI startup(): I_MPI_OFI_PROVIDER=efa
[0] MPI startup(): I_MPI_DEBUG=10
#----------------------------------------------------------------
# Intel(R) MPI Benchmarks 2021.8, MPI-1 part
#----------------------------------------------------------------
# Date : Mon Feb 17 11:30:57 2025
# Machine : x86_64
# System : Linux
# Release : 5.14.0-427.26.1.el9_4.x86_64
# Version : #1 SMP PREEMPT_DYNAMIC Fri Jul 5 11:34:54 EDT 2024
# MPI Version : 3.1
# MPI Thread Environment:


# Calling sequence was:

# IMB-MPI1 barrier

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# Barrier

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 2
# ( 190 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.37 0.37 0.37

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 4
# ( 188 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.60 0.60 0.60

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 8
# ( 184 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
376 0.68 0.68 0.68

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 16
# ( 176 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
283 25936.42 26010.59 25971.98

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 32
# ( 160 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
194 31180.34 31360.85 31291.58

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 64
# ( 128 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
196 43785.67 43984.83 43892.37

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 128
# ( 64 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
140 57157.02 57571.38 57406.40

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 192
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
109 52944.94 53403.62 53220.60


# All processes entering MPI_Finalize

 

In response to TobiasK
0 Kudos
Copy link
mgHarish
Beginner
‎02-17-2025 04:05 AM
879 Views
In response to TobiasK

@TobiasK if you see the pinning information, 192 mpi ranks has been assigned to cpu 0-95. My expectation is that 192 mpi ranks should be assigned to cpu 0-191. Thanks!

[0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20241121 (id: e7829d6)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric loaded: libfabric.so.1
[0] MPI startup(): libfabric version: 1.21.0-impi
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): libfabric provider: efa
[0] MPI startup(): shm segment size (150 MB per rank) * (192 local ranks) = 28870 MB total
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa_100_x1.dat" not found
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa_100.dat" not found
[0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi_efa.dat" not found
[0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: async_progress coll split: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): tag bits available: 30 (TAG_UB value: 1073741823)
[0] MPI startup(): source bits available: 30 (Maximal number of rank: 1073741823)
[0] MPI startup(): Number of NICs: 1
[0] MPI startup(): ===== NIC pinning on efa-od-az-1a-dy-c7i-48xlarge-1 =====
[0] MPI startup(): Rank Thread id Pin nic Nic id
[0] MPI startup(): 0 0 rdmap150s0-rdm 0
[0] MPI startup(): 1 0 rdmap150s0-rdm 0
[0] MPI startup(): 2 0 rdmap150s0-rdm 0
[0] MPI startup(): 3 0 rdmap150s0-rdm 0
[0] MPI startup(): 4 0 rdmap150s0-rdm 0
[0] MPI startup(): 5 0 rdmap150s0-rdm 0
[0] MPI startup(): 6 0 rdmap150s0-rdm 0
[0] MPI startup(): 7 0 rdmap150s0-rdm 0
[0] MPI startup(): 8 0 rdmap150s0-rdm 0
[0] MPI startup(): 9 0 rdmap150s0-rdm 0
[0] MPI startup(): 10 0 rdmap150s0-rdm 0
[0] MPI startup(): 11 0 rdmap150s0-rdm 0
[0] MPI startup(): 12 0 rdmap150s0-rdm 0
[0] MPI startup(): 13 0 rdmap150s0-rdm 0
[0] MPI startup(): 14 0 rdmap150s0-rdm 0
[0] MPI startup(): 15 0 rdmap150s0-rdm 0
[0] MPI startup(): 16 0 rdmap150s0-rdm 0
[0] MPI startup(): 17 0 rdmap150s0-rdm 0
[0] MPI startup(): 18 0 rdmap150s0-rdm 0
[0] MPI startup(): 19 0 rdmap150s0-rdm 0
[0] MPI startup(): 20 0 rdmap150s0-rdm 0
[0] MPI startup(): 21 0 rdmap150s0-rdm 0
[0] MPI startup(): 22 0 rdmap150s0-rdm 0
[0] MPI startup(): 23 0 rdmap150s0-rdm 0
[0] MPI startup(): 24 0 rdmap150s0-rdm 0
[0] MPI startup(): 25 0 rdmap150s0-rdm 0
[0] MPI startup(): 26 0 rdmap150s0-rdm 0
[0] MPI startup(): 27 0 rdmap150s0-rdm 0
[0] MPI startup(): 28 0 rdmap150s0-rdm 0
[0] MPI startup(): 29 0 rdmap150s0-rdm 0
[0] MPI startup(): 30 0 rdmap150s0-rdm 0
[0] MPI startup(): 31 0 rdmap150s0-rdm 0
[0] MPI startup(): 32 0 rdmap150s0-rdm 0
[0] MPI startup(): 33 0 rdmap150s0-rdm 0
[0] MPI startup(): 34 0 rdmap150s0-rdm 0
[0] MPI startup(): 35 0 rdmap150s0-rdm 0
[0] MPI startup(): 36 0 rdmap150s0-rdm 0
[0] MPI startup(): 37 0 rdmap150s0-rdm 0
[0] MPI startup(): 38 0 rdmap150s0-rdm 0
[0] MPI startup(): 39 0 rdmap150s0-rdm 0
[0] MPI startup(): 40 0 rdmap150s0-rdm 0
[0] MPI startup(): 41 0 rdmap150s0-rdm 0
[0] MPI startup(): 42 0 rdmap150s0-rdm 0
[0] MPI startup(): 43 0 rdmap150s0-rdm 0
[0] MPI startup(): 44 0 rdmap150s0-rdm 0
[0] MPI startup(): 45 0 rdmap150s0-rdm 0
[0] MPI startup(): 46 0 rdmap150s0-rdm 0
[0] MPI startup(): 47 0 rdmap150s0-rdm 0
[0] MPI startup(): 48 0 rdmap150s0-rdm 0
[0] MPI startup(): 49 0 rdmap150s0-rdm 0
[0] MPI startup(): 50 0 rdmap150s0-rdm 0
[0] MPI startup(): 51 0 rdmap150s0-rdm 0
[0] MPI startup(): 52 0 rdmap150s0-rdm 0
[0] MPI startup(): 53 0 rdmap150s0-rdm 0
[0] MPI startup(): 54 0 rdmap150s0-rdm 0
[0] MPI startup(): 55 0 rdmap150s0-rdm 0
[0] MPI startup(): 56 0 rdmap150s0-rdm 0
[0] MPI startup(): 57 0 rdmap150s0-rdm 0
[0] MPI startup(): 58 0 rdmap150s0-rdm 0
[0] MPI startup(): 59 0 rdmap150s0-rdm 0
[0] MPI startup(): 60 0 rdmap150s0-rdm 0
[0] MPI startup(): 61 0 rdmap150s0-rdm 0
[0] MPI startup(): 62 0 rdmap150s0-rdm 0
[0] MPI startup(): 63 0 rdmap150s0-rdm 0
[0] MPI startup(): 64 0 rdmap150s0-rdm 0
[0] MPI startup(): 65 0 rdmap150s0-rdm 0
[0] MPI startup(): 66 0 rdmap150s0-rdm 0
[0] MPI startup(): 67 0 rdmap150s0-rdm 0
[0] MPI startup(): 68 0 rdmap150s0-rdm 0
[0] MPI startup(): 69 0 rdmap150s0-rdm 0
[0] MPI startup(): 70 0 rdmap150s0-rdm 0
[0] MPI startup(): 71 0 rdmap150s0-rdm 0
[0] MPI startup(): 72 0 rdmap150s0-rdm 0
[0] MPI startup(): 73 0 rdmap150s0-rdm 0
[0] MPI startup(): 74 0 rdmap150s0-rdm 0
[0] MPI startup(): 75 0 rdmap150s0-rdm 0
[0] MPI startup(): 76 0 rdmap150s0-rdm 0
[0] MPI startup(): 77 0 rdmap150s0-rdm 0
[0] MPI startup(): 78 0 rdmap150s0-rdm 0
[0] MPI startup(): 79 0 rdmap150s0-rdm 0
[0] MPI startup(): 80 0 rdmap150s0-rdm 0
[0] MPI startup(): 81 0 rdmap150s0-rdm 0
[0] MPI startup(): 82 0 rdmap150s0-rdm 0
[0] MPI startup(): 83 0 rdmap150s0-rdm 0
[0] MPI startup(): 84 0 rdmap150s0-rdm 0
[0] MPI startup(): 85 0 rdmap150s0-rdm 0
[0] MPI startup(): 86 0 rdmap150s0-rdm 0
[0] MPI startup(): 87 0 rdmap150s0-rdm 0
[0] MPI startup(): 88 0 rdmap150s0-rdm 0
[0] MPI startup(): 89 0 rdmap150s0-rdm 0
[0] MPI startup(): 90 0 rdmap150s0-rdm 0
[0] MPI startup(): 91 0 rdmap150s0-rdm 0
[0] MPI startup(): 92 0 rdmap150s0-rdm 0
[0] MPI startup(): 93 0 rdmap150s0-rdm 0
[0] MPI startup(): 94 0 rdmap150s0-rdm 0
[0] MPI startup(): 95 0 rdmap150s0-rdm 0
[0] MPI startup(): 96 0 rdmap150s0-rdm 0
[0] MPI startup(): 97 0 rdmap150s0-rdm 0
[0] MPI startup(): 98 0 rdmap150s0-rdm 0
[0] MPI startup(): 99 0 rdmap150s0-rdm 0
[0] MPI startup(): 100 0 rdmap150s0-rdm 0
[0] MPI startup(): 101 0 rdmap150s0-rdm 0
[0] MPI startup(): 102 0 rdmap150s0-rdm 0
[0] MPI startup(): 103 0 rdmap150s0-rdm 0
[0] MPI startup(): 104 0 rdmap150s0-rdm 0
[0] MPI startup(): 105 0 rdmap150s0-rdm 0
[0] MPI startup(): 106 0 rdmap150s0-rdm 0
[0] MPI startup(): 107 0 rdmap150s0-rdm 0
[0] MPI startup(): 108 0 rdmap150s0-rdm 0
[0] MPI startup(): 109 0 rdmap150s0-rdm 0
[0] MPI startup(): 110 0 rdmap150s0-rdm 0
[0] MPI startup(): 111 0 rdmap150s0-rdm 0
[0] MPI startup(): 112 0 rdmap150s0-rdm 0
[0] MPI startup(): 113 0 rdmap150s0-rdm 0
[0] MPI startup(): 114 0 rdmap150s0-rdm 0
[0] MPI startup(): 115 0 rdmap150s0-rdm 0
[0] MPI startup(): 116 0 rdmap150s0-rdm 0
[0] MPI startup(): 117 0 rdmap150s0-rdm 0
[0] MPI startup(): 118 0 rdmap150s0-rdm 0
[0] MPI startup(): 119 0 rdmap150s0-rdm 0
[0] MPI startup(): 120 0 rdmap150s0-rdm 0
[0] MPI startup(): 121 0 rdmap150s0-rdm 0
[0] MPI startup(): 122 0 rdmap150s0-rdm 0
[0] MPI startup(): 123 0 rdmap150s0-rdm 0
[0] MPI startup(): 124 0 rdmap150s0-rdm 0
[0] MPI startup(): 125 0 rdmap150s0-rdm 0
[0] MPI startup(): 126 0 rdmap150s0-rdm 0
[0] MPI startup(): 127 0 rdmap150s0-rdm 0
[0] MPI startup(): 128 0 rdmap150s0-rdm 0
[0] MPI startup(): 129 0 rdmap150s0-rdm 0
[0] MPI startup(): 130 0 rdmap150s0-rdm 0
[0] MPI startup(): 131 0 rdmap150s0-rdm 0
[0] MPI startup(): 132 0 rdmap150s0-rdm 0
[0] MPI startup(): 133 0 rdmap150s0-rdm 0
[0] MPI startup(): 134 0 rdmap150s0-rdm 0
[0] MPI startup(): 135 0 rdmap150s0-rdm 0
[0] MPI startup(): 136 0 rdmap150s0-rdm 0
[0] MPI startup(): 137 0 rdmap150s0-rdm 0
[0] MPI startup(): 138 0 rdmap150s0-rdm 0
[0] MPI startup(): 139 0 rdmap150s0-rdm 0
[0] MPI startup(): 140 0 rdmap150s0-rdm 0
[0] MPI startup(): 141 0 rdmap150s0-rdm 0
[0] MPI startup(): 142 0 rdmap150s0-rdm 0
[0] MPI startup(): 143 0 rdmap150s0-rdm 0
[0] MPI startup(): 144 0 rdmap150s0-rdm 0
[0] MPI startup(): 145 0 rdmap150s0-rdm 0
[0] MPI startup(): 146 0 rdmap150s0-rdm 0
[0] MPI startup(): 147 0 rdmap150s0-rdm 0
[0] MPI startup(): 148 0 rdmap150s0-rdm 0
[0] MPI startup(): 149 0 rdmap150s0-rdm 0
[0] MPI startup(): 150 0 rdmap150s0-rdm 0
[0] MPI startup(): 151 0 rdmap150s0-rdm 0
[0] MPI startup(): 152 0 rdmap150s0-rdm 0
[0] MPI startup(): 153 0 rdmap150s0-rdm 0
[0] MPI startup(): 154 0 rdmap150s0-rdm 0
[0] MPI startup(): 155 0 rdmap150s0-rdm 0
[0] MPI startup(): 156 0 rdmap150s0-rdm 0
[0] MPI startup(): 157 0 rdmap150s0-rdm 0
[0] MPI startup(): 158 0 rdmap150s0-rdm 0
[0] MPI startup(): 159 0 rdmap150s0-rdm 0
[0] MPI startup(): 160 0 rdmap150s0-rdm 0
[0] MPI startup(): 161 0 rdmap150s0-rdm 0
[0] MPI startup(): 162 0 rdmap150s0-rdm 0
[0] MPI startup(): 163 0 rdmap150s0-rdm 0
[0] MPI startup(): 164 0 rdmap150s0-rdm 0
[0] MPI startup(): 165 0 rdmap150s0-rdm 0
[0] MPI startup(): 166 0 rdmap150s0-rdm 0
[0] MPI startup(): 167 0 rdmap150s0-rdm 0
[0] MPI startup(): 168 0 rdmap150s0-rdm 0
[0] MPI startup(): 169 0 rdmap150s0-rdm 0
[0] MPI startup(): 170 0 rdmap150s0-rdm 0
[0] MPI startup(): 171 0 rdmap150s0-rdm 0
[0] MPI startup(): 172 0 rdmap150s0-rdm 0
[0] MPI startup(): 173 0 rdmap150s0-rdm 0
[0] MPI startup(): 174 0 rdmap150s0-rdm 0
[0] MPI startup(): 175 0 rdmap150s0-rdm 0
[0] MPI startup(): 176 0 rdmap150s0-rdm 0
[0] MPI startup(): 177 0 rdmap150s0-rdm 0
[0] MPI startup(): 178 0 rdmap150s0-rdm 0
[0] MPI startup(): 179 0 rdmap150s0-rdm 0
[0] MPI startup(): 180 0 rdmap150s0-rdm 0
[0] MPI startup(): 181 0 rdmap150s0-rdm 0
[0] MPI startup(): 182 0 rdmap150s0-rdm 0
[0] MPI startup(): 183 0 rdmap150s0-rdm 0
[0] MPI startup(): 184 0 rdmap150s0-rdm 0
[0] MPI startup(): 185 0 rdmap150s0-rdm 0
[0] MPI startup(): 186 0 rdmap150s0-rdm 0
[0] MPI startup(): 187 0 rdmap150s0-rdm 0
[0] MPI startup(): 188 0 rdmap150s0-rdm 0
[0] MPI startup(): 189 0 rdmap150s0-rdm 0
[0] MPI startup(): 190 0 rdmap150s0-rdm 0
[0] MPI startup(): 191 0 rdmap150s0-rdm 0
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 15233 efa-od-az-1a-dy-c7i-48xlarge-1 {48}
[0] MPI startup(): 1 15234 efa-od-az-1a-dy-c7i-48xlarge-1 {49}
[0] MPI startup(): 2 15235 efa-od-az-1a-dy-c7i-48xlarge-1 {50}
[0] MPI startup(): 3 15236 efa-od-az-1a-dy-c7i-48xlarge-1 {51}
[0] MPI startup(): 4 15237 efa-od-az-1a-dy-c7i-48xlarge-1 {52}
[0] MPI startup(): 5 15238 efa-od-az-1a-dy-c7i-48xlarge-1 {53}
[0] MPI startup(): 6 15239 efa-od-az-1a-dy-c7i-48xlarge-1 {54}
[0] MPI startup(): 7 15240 efa-od-az-1a-dy-c7i-48xlarge-1 {55}
[0] MPI startup(): 8 15241 efa-od-az-1a-dy-c7i-48xlarge-1 {56}
[0] MPI startup(): 9 15242 efa-od-az-1a-dy-c7i-48xlarge-1 {57}
[0] MPI startup(): 10 15243 efa-od-az-1a-dy-c7i-48xlarge-1 {58}
[0] MPI startup(): 11 15244 efa-od-az-1a-dy-c7i-48xlarge-1 {59}
[0] MPI startup(): 12 15245 efa-od-az-1a-dy-c7i-48xlarge-1 {60}
[0] MPI startup(): 13 15250 efa-od-az-1a-dy-c7i-48xlarge-1 {61}
[0] MPI startup(): 14 15253 efa-od-az-1a-dy-c7i-48xlarge-1 {62}
[0] MPI startup(): 15 15254 efa-od-az-1a-dy-c7i-48xlarge-1 {63}
[0] MPI startup(): 16 15258 efa-od-az-1a-dy-c7i-48xlarge-1 {64}
[0] MPI startup(): 17 15265 efa-od-az-1a-dy-c7i-48xlarge-1 {65}
[0] MPI startup(): 18 15269 efa-od-az-1a-dy-c7i-48xlarge-1 {66}
[0] MPI startup(): 19 15270 efa-od-az-1a-dy-c7i-48xlarge-1 {67}
[0] MPI startup(): 20 15273 efa-od-az-1a-dy-c7i-48xlarge-1 {68}
[0] MPI startup(): 21 15276 efa-od-az-1a-dy-c7i-48xlarge-1 {69}
[0] MPI startup(): 22 15279 efa-od-az-1a-dy-c7i-48xlarge-1 {70}
[0] MPI startup(): 23 15285 efa-od-az-1a-dy-c7i-48xlarge-1 {71}
[0] MPI startup(): 24 15287 efa-od-az-1a-dy-c7i-48xlarge-1 {72}
[0] MPI startup(): 25 15293 efa-od-az-1a-dy-c7i-48xlarge-1 {73}
[0] MPI startup(): 26 15294 efa-od-az-1a-dy-c7i-48xlarge-1 {74}
[0] MPI startup(): 27 15297 efa-od-az-1a-dy-c7i-48xlarge-1 {75}
[0] MPI startup(): 28 15298 efa-od-az-1a-dy-c7i-48xlarge-1 {76}
[0] MPI startup(): 29 15299 efa-od-az-1a-dy-c7i-48xlarge-1 {77}
[0] MPI startup(): 30 15303 efa-od-az-1a-dy-c7i-48xlarge-1 {78}
[0] MPI startup(): 31 15304 efa-od-az-1a-dy-c7i-48xlarge-1 {79}
[0] MPI startup(): 32 15307 efa-od-az-1a-dy-c7i-48xlarge-1 {80}
[0] MPI startup(): 33 15311 efa-od-az-1a-dy-c7i-48xlarge-1 {81}
[0] MPI startup(): 34 15312 efa-od-az-1a-dy-c7i-48xlarge-1 {82}
[0] MPI startup(): 35 15314 efa-od-az-1a-dy-c7i-48xlarge-1 {83}
[0] MPI startup(): 36 15315 efa-od-az-1a-dy-c7i-48xlarge-1 {84}
[0] MPI startup(): 37 15316 efa-od-az-1a-dy-c7i-48xlarge-1 {85}
[0] MPI startup(): 38 15321 efa-od-az-1a-dy-c7i-48xlarge-1 {86}
[0] MPI startup(): 39 15325 efa-od-az-1a-dy-c7i-48xlarge-1 {87}
[0] MPI startup(): 40 15326 efa-od-az-1a-dy-c7i-48xlarge-1 {88}
[0] MPI startup(): 41 15328 efa-od-az-1a-dy-c7i-48xlarge-1 {89}
[0] MPI startup(): 42 15329 efa-od-az-1a-dy-c7i-48xlarge-1 {90}
[0] MPI startup(): 43 15330 efa-od-az-1a-dy-c7i-48xlarge-1 {91}
[0] MPI startup(): 44 15336 efa-od-az-1a-dy-c7i-48xlarge-1 {92}
[0] MPI startup(): 45 15337 efa-od-az-1a-dy-c7i-48xlarge-1 {93}
[0] MPI startup(): 46 15340 efa-od-az-1a-dy-c7i-48xlarge-1 {94}
[0] MPI startup(): 47 15344 efa-od-az-1a-dy-c7i-48xlarge-1 {95}
[0] MPI startup(): 48 15348 efa-od-az-1a-dy-c7i-48xlarge-1 {0}
[0] MPI startup(): 49 15349 efa-od-az-1a-dy-c7i-48xlarge-1 {1}
[0] MPI startup(): 50 15352 efa-od-az-1a-dy-c7i-48xlarge-1 {2}
[0] MPI startup(): 51 15356 efa-od-az-1a-dy-c7i-48xlarge-1 {3}
[0] MPI startup(): 52 15362 efa-od-az-1a-dy-c7i-48xlarge-1 {4}
[0] MPI startup(): 53 15364 efa-od-az-1a-dy-c7i-48xlarge-1 {5}
[0] MPI startup(): 54 15367 efa-od-az-1a-dy-c7i-48xlarge-1 {6}
[0] MPI startup(): 55 15372 efa-od-az-1a-dy-c7i-48xlarge-1 {7}
[0] MPI startup(): 56 15373 efa-od-az-1a-dy-c7i-48xlarge-1 {8}
[0] MPI startup(): 57 15377 efa-od-az-1a-dy-c7i-48xlarge-1 {9}
[0] MPI startup(): 58 15378 efa-od-az-1a-dy-c7i-48xlarge-1 {10}
[0] MPI startup(): 59 15381 efa-od-az-1a-dy-c7i-48xlarge-1 {11}
[0] MPI startup(): 60 15386 efa-od-az-1a-dy-c7i-48xlarge-1 {12}
[0] MPI startup(): 61 15387 efa-od-az-1a-dy-c7i-48xlarge-1 {13}
[0] MPI startup(): 62 15389 efa-od-az-1a-dy-c7i-48xlarge-1 {14}
[0] MPI startup(): 63 15393 efa-od-az-1a-dy-c7i-48xlarge-1 {15}
[0] MPI startup(): 64 15395 efa-od-az-1a-dy-c7i-48xlarge-1 {16}
[0] MPI startup(): 65 15396 efa-od-az-1a-dy-c7i-48xlarge-1 {17}
[0] MPI startup(): 66 15402 efa-od-az-1a-dy-c7i-48xlarge-1 {18}
[0] MPI startup(): 67 15403 efa-od-az-1a-dy-c7i-48xlarge-1 {19}
[0] MPI startup(): 68 15404 efa-od-az-1a-dy-c7i-48xlarge-1 {20}
[0] MPI startup(): 69 15405 efa-od-az-1a-dy-c7i-48xlarge-1 {21}
[0] MPI startup(): 70 15406 efa-od-az-1a-dy-c7i-48xlarge-1 {22}
[0] MPI startup(): 71 15410 efa-od-az-1a-dy-c7i-48xlarge-1 {23}
[0] MPI startup(): 72 15413 efa-od-az-1a-dy-c7i-48xlarge-1 {24}
[0] MPI startup(): 73 15414 efa-od-az-1a-dy-c7i-48xlarge-1 {25}
[0] MPI startup(): 74 15417 efa-od-az-1a-dy-c7i-48xlarge-1 {26}
[0] MPI startup(): 75 15418 efa-od-az-1a-dy-c7i-48xlarge-1 {27}
[0] MPI startup(): 76 15420 efa-od-az-1a-dy-c7i-48xlarge-1 {28}
[0] MPI startup(): 77 15421 efa-od-az-1a-dy-c7i-48xlarge-1 {29}
[0] MPI startup(): 78 15422 efa-od-az-1a-dy-c7i-48xlarge-1 {30}
[0] MPI startup(): 79 15423 efa-od-az-1a-dy-c7i-48xlarge-1 {31}
[0] MPI startup(): 80 15424 efa-od-az-1a-dy-c7i-48xlarge-1 {32}
[0] MPI startup(): 81 15425 efa-od-az-1a-dy-c7i-48xlarge-1 {33}
[0] MPI startup(): 82 15426 efa-od-az-1a-dy-c7i-48xlarge-1 {34}
[0] MPI startup(): 83 15427 efa-od-az-1a-dy-c7i-48xlarge-1 {35}
[0] MPI startup(): 84 15428 efa-od-az-1a-dy-c7i-48xlarge-1 {36}
[0] MPI startup(): 85 15429 efa-od-az-1a-dy-c7i-48xlarge-1 {37}
[0] MPI startup(): 86 15430 efa-od-az-1a-dy-c7i-48xlarge-1 {38}
[0] MPI startup(): 87 15431 efa-od-az-1a-dy-c7i-48xlarge-1 {39}
[0] MPI startup(): 88 15432 efa-od-az-1a-dy-c7i-48xlarge-1 {40}
[0] MPI startup(): 89 15433 efa-od-az-1a-dy-c7i-48xlarge-1 {41}
[0] MPI startup(): 90 15434 efa-od-az-1a-dy-c7i-48xlarge-1 {42}
[0] MPI startup(): 91 15435 efa-od-az-1a-dy-c7i-48xlarge-1 {43}
[0] MPI startup(): 92 15436 efa-od-az-1a-dy-c7i-48xlarge-1 {44}
[0] MPI startup(): 93 15437 efa-od-az-1a-dy-c7i-48xlarge-1 {45}
[0] MPI startup(): 94 15438 efa-od-az-1a-dy-c7i-48xlarge-1 {46}
[0] MPI startup(): 95 15439 efa-od-az-1a-dy-c7i-48xlarge-1 {47}
[0] MPI startup(): 96 15440 efa-od-az-1a-dy-c7i-48xlarge-1 {48}
[0] MPI startup(): 97 15441 efa-od-az-1a-dy-c7i-48xlarge-1 {49}
[0] MPI startup(): 98 15442 efa-od-az-1a-dy-c7i-48xlarge-1 {50}
[0] MPI startup(): 99 15443 efa-od-az-1a-dy-c7i-48xlarge-1 {51}
[0] MPI startup(): 100 15444 efa-od-az-1a-dy-c7i-48xlarge-1 {52}
[0] MPI startup(): 101 15445 efa-od-az-1a-dy-c7i-48xlarge-1 {53}
[0] MPI startup(): 102 15446 efa-od-az-1a-dy-c7i-48xlarge-1 {54}
[0] MPI startup(): 103 15447 efa-od-az-1a-dy-c7i-48xlarge-1 {55}
[0] MPI startup(): 104 15448 efa-od-az-1a-dy-c7i-48xlarge-1 {56}
[0] MPI startup(): 105 15450 efa-od-az-1a-dy-c7i-48xlarge-1 {57}
[0] MPI startup(): 106 15451 efa-od-az-1a-dy-c7i-48xlarge-1 {58}
[0] MPI startup(): 107 15452 efa-od-az-1a-dy-c7i-48xlarge-1 {59}
[0] MPI startup(): 108 15453 efa-od-az-1a-dy-c7i-48xlarge-1 {60}
[0] MPI startup(): 109 15455 efa-od-az-1a-dy-c7i-48xlarge-1 {61}
[0] MPI startup(): 110 15456 efa-od-az-1a-dy-c7i-48xlarge-1 {62}
[0] MPI startup(): 111 15457 efa-od-az-1a-dy-c7i-48xlarge-1 {63}
[0] MPI startup(): 112 15458 efa-od-az-1a-dy-c7i-48xlarge-1 {64}
[0] MPI startup(): 113 15459 efa-od-az-1a-dy-c7i-48xlarge-1 {65}
[0] MPI startup(): 114 15460 efa-od-az-1a-dy-c7i-48xlarge-1 {66}
[0] MPI startup(): 115 15461 efa-od-az-1a-dy-c7i-48xlarge-1 {67}
[0] MPI startup(): 116 15462 efa-od-az-1a-dy-c7i-48xlarge-1 {68}
[0] MPI startup(): 117 15463 efa-od-az-1a-dy-c7i-48xlarge-1 {69}
[0] MPI startup(): 118 15464 efa-od-az-1a-dy-c7i-48xlarge-1 {70}
[0] MPI startup(): 119 15465 efa-od-az-1a-dy-c7i-48xlarge-1 {71}
[0] MPI startup(): 120 15466 efa-od-az-1a-dy-c7i-48xlarge-1 {72}
[0] MPI startup(): 121 15467 efa-od-az-1a-dy-c7i-48xlarge-1 {73}
[0] MPI startup(): 122 15468 efa-od-az-1a-dy-c7i-48xlarge-1 {74}
[0] MPI startup(): 123 15469 efa-od-az-1a-dy-c7i-48xlarge-1 {75}
[0] MPI startup(): 124 15470 efa-od-az-1a-dy-c7i-48xlarge-1 {76}
[0] MPI startup(): 125 15471 efa-od-az-1a-dy-c7i-48xlarge-1 {77}
[0] MPI startup(): 126 15472 efa-od-az-1a-dy-c7i-48xlarge-1 {78}
[0] MPI startup(): 127 15473 efa-od-az-1a-dy-c7i-48xlarge-1 {79}
[0] MPI startup(): 128 15474 efa-od-az-1a-dy-c7i-48xlarge-1 {80}
[0] MPI startup(): 129 15475 efa-od-az-1a-dy-c7i-48xlarge-1 {81}
[0] MPI startup(): 130 15478 efa-od-az-1a-dy-c7i-48xlarge-1 {82}
[0] MPI startup(): 131 15479 efa-od-az-1a-dy-c7i-48xlarge-1 {83}
[0] MPI startup(): 132 15484 efa-od-az-1a-dy-c7i-48xlarge-1 {84}
[0] MPI startup(): 133 15487 efa-od-az-1a-dy-c7i-48xlarge-1 {85}
[0] MPI startup(): 134 15490 efa-od-az-1a-dy-c7i-48xlarge-1 {86}
[0] MPI startup(): 135 15495 efa-od-az-1a-dy-c7i-48xlarge-1 {87}
[0] MPI startup(): 136 15497 efa-od-az-1a-dy-c7i-48xlarge-1 {88}
[0] MPI startup(): 137 15501 efa-od-az-1a-dy-c7i-48xlarge-1 {89}
[0] MPI startup(): 138 15509 efa-od-az-1a-dy-c7i-48xlarge-1 {90}
[0] MPI startup(): 139 15515 efa-od-az-1a-dy-c7i-48xlarge-1 {91}
[0] MPI startup(): 140 15516 efa-od-az-1a-dy-c7i-48xlarge-1 {92}
[0] MPI startup(): 141 15521 efa-od-az-1a-dy-c7i-48xlarge-1 {93}
[0] MPI startup(): 142 15529 efa-od-az-1a-dy-c7i-48xlarge-1 {94}
[0] MPI startup(): 143 15535 efa-od-az-1a-dy-c7i-48xlarge-1 {95}
[0] MPI startup(): 144 15547 efa-od-az-1a-dy-c7i-48xlarge-1 {0}
[0] MPI startup(): 145 15548 efa-od-az-1a-dy-c7i-48xlarge-1 {1}
[0] MPI startup(): 146 15549 efa-od-az-1a-dy-c7i-48xlarge-1 {2}
[0] MPI startup(): 147 15550 efa-od-az-1a-dy-c7i-48xlarge-1 {3}
[0] MPI startup(): 148 15551 efa-od-az-1a-dy-c7i-48xlarge-1 {4}
[0] MPI startup(): 149 15552 efa-od-az-1a-dy-c7i-48xlarge-1 {5}
[0] MPI startup(): 150 15553 efa-od-az-1a-dy-c7i-48xlarge-1 {6}
[0] MPI startup(): 151 15554 efa-od-az-1a-dy-c7i-48xlarge-1 {7}
[0] MPI startup(): 152 15555 efa-od-az-1a-dy-c7i-48xlarge-1 {8}
[0] MPI startup(): 153 15556 efa-od-az-1a-dy-c7i-48xlarge-1 {9}
[0] MPI startup(): 154 15557 efa-od-az-1a-dy-c7i-48xlarge-1 {10}
[0] MPI startup(): 155 15558 efa-od-az-1a-dy-c7i-48xlarge-1 {11}
[0] MPI startup(): 156 15559 efa-od-az-1a-dy-c7i-48xlarge-1 {12}
[0] MPI startup(): 157 15560 efa-od-az-1a-dy-c7i-48xlarge-1 {13}
[0] MPI startup(): 158 15561 efa-od-az-1a-dy-c7i-48xlarge-1 {14}
[0] MPI startup(): 159 15562 efa-od-az-1a-dy-c7i-48xlarge-1 {15}
[0] MPI startup(): 160 15563 efa-od-az-1a-dy-c7i-48xlarge-1 {16}
[0] MPI startup(): 161 15564 efa-od-az-1a-dy-c7i-48xlarge-1 {17}
[0] MPI startup(): 162 15565 efa-od-az-1a-dy-c7i-48xlarge-1 {18}
[0] MPI startup(): 163 15566 efa-od-az-1a-dy-c7i-48xlarge-1 {19}
[0] MPI startup(): 164 15567 efa-od-az-1a-dy-c7i-48xlarge-1 {20}
[0] MPI startup(): 165 15568 efa-od-az-1a-dy-c7i-48xlarge-1 {21}
[0] MPI startup(): 166 15569 efa-od-az-1a-dy-c7i-48xlarge-1 {22}
[0] MPI startup(): 167 15570 efa-od-az-1a-dy-c7i-48xlarge-1 {23}
[0] MPI startup(): 168 15572 efa-od-az-1a-dy-c7i-48xlarge-1 {24}
[0] MPI startup(): 169 15573 efa-od-az-1a-dy-c7i-48xlarge-1 {25}
[0] MPI startup(): 170 15574 efa-od-az-1a-dy-c7i-48xlarge-1 {26}
[0] MPI startup(): 171 15575 efa-od-az-1a-dy-c7i-48xlarge-1 {27}
[0] MPI startup(): 172 15576 efa-od-az-1a-dy-c7i-48xlarge-1 {28}
[0] MPI startup(): 173 15577 efa-od-az-1a-dy-c7i-48xlarge-1 {29}
[0] MPI startup(): 174 15578 efa-od-az-1a-dy-c7i-48xlarge-1 {30}
[0] MPI startup(): 175 15579 efa-od-az-1a-dy-c7i-48xlarge-1 {31}
[0] MPI startup(): 176 15580 efa-od-az-1a-dy-c7i-48xlarge-1 {32}
[0] MPI startup(): 177 15581 efa-od-az-1a-dy-c7i-48xlarge-1 {33}
[0] MPI startup(): 178 15582 efa-od-az-1a-dy-c7i-48xlarge-1 {34}
[0] MPI startup(): 179 15583 efa-od-az-1a-dy-c7i-48xlarge-1 {35}
[0] MPI startup(): 180 15584 efa-od-az-1a-dy-c7i-48xlarge-1 {36}
[0] MPI startup(): 181 15585 efa-od-az-1a-dy-c7i-48xlarge-1 {37}
[0] MPI startup(): 182 15586 efa-od-az-1a-dy-c7i-48xlarge-1 {38}
[0] MPI startup(): 183 15587 efa-od-az-1a-dy-c7i-48xlarge-1 {39}
[0] MPI startup(): 184 15588 efa-od-az-1a-dy-c7i-48xlarge-1 {40}
[0] MPI startup(): 185 15589 efa-od-az-1a-dy-c7i-48xlarge-1 {41}
[0] MPI startup(): 186 15590 efa-od-az-1a-dy-c7i-48xlarge-1 {42}
[0] MPI startup(): 187 15591 efa-od-az-1a-dy-c7i-48xlarge-1 {43}
[0] MPI startup(): 188 15592 efa-od-az-1a-dy-c7i-48xlarge-1 {44}
[0] MPI startup(): 189 15593 efa-od-az-1a-dy-c7i-48xlarge-1 {45}
[0] MPI startup(): 190 15594 efa-od-az-1a-dy-c7i-48xlarge-1 {46}
[0] MPI startup(): 191 15595 efa-od-az-1a-dy-c7i-48xlarge-1 {47}
[0] MPI startup(): I_MPI_OFI_LIBRARY_INTERNAL=1
[0] MPI startup(): I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.14
[0] MPI startup(): I_MPI_MPIRUN=mpirun
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_FABRICS=shm:ofi
[0] MPI startup(): I_MPI_OFI_PROVIDER=efa
[0] MPI startup(): I_MPI_DEBUG=10
#----------------------------------------------------------------
# Intel(R) MPI Benchmarks 2021.8, MPI-1 part
#----------------------------------------------------------------
# Date : Mon Feb 17 11:30:57 2025
# Machine : x86_64
# System : Linux
# Release : 5.14.0-427.26.1.el9_4.x86_64
# Version : #1 SMP PREEMPT_DYNAMIC Fri Jul 5 11:34:54 EDT 2024
# MPI Version : 3.1
# MPI Thread Environment:


# Calling sequence was:

# IMB-MPI1 barrier

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# Barrier

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 2
# ( 190 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.37 0.37 0.37

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 4
# ( 188 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.60 0.60 0.60

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 8
# ( 184 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
376 0.68 0.68 0.68

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 16
# ( 176 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
283 25936.42 26010.59 25971.98

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 32
# ( 160 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
194 31180.34 31360.85 31291.58

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 64
# ( 128 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
196 43785.67 43984.83 43892.37

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 128
# ( 64 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
140 57157.02 57571.38 57406.40

#---------------------------------------------------
# Benchmarking Barrier
# #processes = 192
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
109 52944.94 53403.62 53220.60


# All processes entering MPI_Finalize

 

In response to TobiasK
0 Kudos
Copy link
mgHarish
Beginner
‎02-17-2025 04:11 AM
1,049 Views

@TobiasK  If you see the pinning information, 192 mpi ranks has been assigned to cpu 0-95 (two ranks per cpu). My expectation is that 192 mpi ranks should be assigned to cpu 0-191.

Please find the output of 'mpirun IMB-MPI1 barrier' in the attachment. Thanks!

IMB-MPI1.txt
(Virus scan in progress ...)
0 Kudos
Copy link
TobiasK
Moderator
‎02-17-2025 09:32 AM
1,026 Views

Thank you for the output. I see slurm is used, how is slurm configured? Can you provide all Slurm environment variables, e.g.

 

export | grep SLURM

do you see the same pinning issue if you use srun instead of mpirun? 

0 Kudos
Copy link
mgHarish
Beginner
‎02-17-2025 10:46 AM
1,019 Views
In response to TobiasK

@TobiasK Thanks!

mgHarish_0-1739815914325.png

 

Yes, I see the same behavior when i use srun to stress all the 192 cpus, but from the htop i see only 50 percent of total cpu's (96 cpu's) went to 100 percent utilization. Is this due to slurm configuration issue?

mgHarish_1-1739817841377.png


command used: srun -n 192 -p efa-od-az-1a -w efa-od-az-1a-dy-c7i-48xlarge-1 --pty /usr/bin/stress --cpu 1 --timeout 10

 

In response to TobiasK
0 Kudos
Copy link
TobiasK
Moderator
‎02-17-2025 12:09 PM
1,009 Views

@mgHarish most likely yes.

Can you try with:

I_MPI_PIN_RESPECT_CPUSET=0 I_MPI_DEBUG=10 mpirun -np 192 IMB-MPI1 -npmin 192 barrier

What does 

scontrol show node efa-od-az-1a-dy-c7i-48xlarge-1

show?

0 Kudos
Copy link
mgHarish
Beginner
‎02-18-2025 03:05 AM
954 Views

@TobiasK I get the same behavior with the command

I_MPI_PIN_RESPECT_CPUSET=0 I_MPI_DEBUG=10 mpirun -np 192 IMB-MPI1 -npmin 192 barrier

But when I tried ssh into the node and ran the same command, I get the desired output (both will be in the attachments).

Please find the output of scontrol show node command while the below command is running

srun -n 192 -p efa-od-az-1a -w efa-od-az-1a-dy-c7i-48xlarge-1 --pty /usr/bin/stress --cpu 1 --timeout 300

mgHarish_1-1739876251359.png

 

Thanks

IMB-MPI1-ssh.txt
(Virus scan in progress ...)
IMB-MPI1-srun.txt
(Virus scan in progress ...)
0 Kudos
Copy link
mgHarish
Beginner
‎02-26-2025 04:46 AM
719 Views
In response to mgHarish

@TobiasK  I found the cause of the issue. By, default slurm was configured with 1 thread per core, when I configured with 2 threads per core as suggested in (3.9.0‐3.9.1) Default ThreadsPerCore Slurm setting causes reduced CPU utilization · aws/aws-parallelcluster Wiki · GitHub, Intel MPI also shows threads per core equals 2.

 

Help me to understand why the change in slurm configuration has resolved my issue. Exactly how slurm and intelmpi is linked?

In response to mgHarish
0 Kudos
Copy link
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.