# IntelMPI cpuinfo showing one thread per core **URL:** \ **Category:** Intel® MPI Library\ **Created:** [February 11, 2025, 9:45pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221 "2025-02-11T21:45:48Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 11, 2025, 9:45pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/1 "2025-02-11T21:45:48Z")
My machine has 96 Physical Cores (192 logical cpu), but the cpuinfo comes with intelmpi show threads per core = 1. lscpu ![mgHarish_0-1739310233625.png](https://us1.discourse-cdn.com/intel/original/2X/4/4b039cb2493a3de03b71c4314cb05784377514cd.png) cpuinfo (intelmpi) ![mgHarish_1-1739310285182.png](https://us1.discourse-cdn.com/intel/original/3X/f/b/fb499c297f5b4f025312540c9684e6f3b3b00c6e.png) Because of this I was not able to pin processors to the logical cpus. ---
**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [February 17, 2025, 10:36am UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/2 "2025-02-17T10:36:12Z")
@mgHarish  thanks for reporting that, however, it would be really helpful if you also provide your OS version and the version of Intel MPI used to generate this output. ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 17, 2025, 10:54am UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/3 "2025-02-17T10:54:32Z")
Hi  @TobiasK , Thank for the response! OS Version ![mgHarish_0-1739789453216.png](https://us1.discourse-cdn.com/intel/original/3X/f/3/f3f083dbb50f03459f514bdc82a3d3e9442002d3.png) IntelMPI Version ![mgHarish_1-1739789616973.png](https://us1.discourse-cdn.com/intel/original/3X/8/1/8106ecdeed0eb47e023f63f195a0f50fa0fccf5c.png) ---
**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [February 17, 2025, 11:15am UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/4 "2025-02-17T11:15:20Z")
@mgHarish  thanks, can you please provide the output of? ```bash export I_MPI_DEBUG=10 mpirun IMB-MPI1 barrier ``` ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 17, 2025, 11:38am UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/5 "2025-02-17T11:38:54Z")
@TobiasK  If you see the pinning information, each pin cpu {0-95} has two mpi ranks. My expectation is pin cpu should be 0-191 with one mpi ranks on each cpu. Thanks! [0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20241121 (id: e7829d6) [0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric loaded: libfabric.so.1 [0] MPI startup(): libfabric version: 1.21.0-impi [0] MPI startup(): max number of MPI\_Request per vci: 67108864 (pools: 1) [0] MPI startup(): libfabric provider: efa [0] MPI startup(): shm segment size (150 MB per rank) \* (192 local ranks) = 28870 MB total [0] MPI startup(): File “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi\_efa\_100\_x1.dat” not found [0] MPI startup(): File “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi\_efa\_100.dat” not found [0] MPI startup(): File “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi\_efa.dat” not found [0] MPI startup(): Load tuning file: “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi.dat” [0] MPI startup(): threading: mode: direct [0] MPI startup(): threading: vcis: 1 [0] MPI startup(): threading: app\_threads: -1 [0] MPI startup(): threading: runtime: generic [0] MPI startup(): threading: progress\_threads: 0 [0] MPI startup(): threading: async\_progress: 0 [0] MPI startup(): threading: async\_progress coll split: 0 [0] MPI startup(): threading: lock\_level: global [0] MPI startup(): tag bits available: 30 (TAG\_UB value: 1073741823) [0] MPI startup(): source bits available: 30 (Maximal number of rank: 1073741823) [0] MPI startup(): Number of NICs: 1 [0] MPI startup(): ===== NIC pinning on efa-od-az-1a-dy-c7i-48xlarge-1 ===== [0] MPI startup(): Rank Thread id Pin nic Nic id [0] MPI startup(): 0 0 rdmap150s0-rdm 0 [0] MPI startup(): 1 0 rdmap150s0-rdm 0 [0] MPI startup(): 2 0 rdmap150s0-rdm 0 [0] MPI startup(): 3 0 rdmap150s0-rdm 0 [0] MPI startup(): 4 0 rdmap150s0-rdm 0 [0] MPI startup(): 5 0 rdmap150s0-rdm 0 [0] MPI startup(): 6 0 rdmap150s0-rdm 0 [0] MPI startup(): 7 0 rdmap150s0-rdm 0 [0] MPI startup(): 8 0 rdmap150s0-rdm 0 [0] MPI startup(): 9 0 rdmap150s0-rdm 0 [0] MPI startup(): 10 0 rdmap150s0-rdm 0 [0] MPI startup(): 11 0 rdmap150s0-rdm 0 [0] MPI startup(): 12 0 rdmap150s0-rdm 0 [0] MPI startup(): 13 0 rdmap150s0-rdm 0 [0] MPI startup(): 14 0 rdmap150s0-rdm 0 [0] MPI startup(): 15 0 rdmap150s0-rdm 0 [0] MPI startup(): 16 0 rdmap150s0-rdm 0 [0] MPI startup(): 17 0 rdmap150s0-rdm 0 [0] MPI startup(): 18 0 rdmap150s0-rdm 0 [0] MPI startup(): 19 0 rdmap150s0-rdm 0 [0] MPI startup(): 20 0 rdmap150s0-rdm 0 [0] MPI startup(): 21 0 rdmap150s0-rdm 0 [0] MPI startup(): 22 0 rdmap150s0-rdm 0 [0] MPI startup(): 23 0 rdmap150s0-rdm 0 [0] MPI startup(): 24 0 rdmap150s0-rdm 0 [0] MPI startup(): 25 0 rdmap150s0-rdm 0 [0] MPI startup(): 26 0 rdmap150s0-rdm 0 [0] MPI startup(): 27 0 rdmap150s0-rdm 0 [0] MPI startup(): 28 0 rdmap150s0-rdm 0 [0] MPI startup(): 29 0 rdmap150s0-rdm 0 [0] MPI startup(): 30 0 rdmap150s0-rdm 0 [0] MPI startup(): 31 0 rdmap150s0-rdm 0 [0] MPI startup(): 32 0 rdmap150s0-rdm 0 [0] MPI startup(): 33 0 rdmap150s0-rdm 0 [0] MPI startup(): 34 0 rdmap150s0-rdm 0 [0] MPI startup(): 35 0 rdmap150s0-rdm 0 [0] MPI startup(): 36 0 rdmap150s0-rdm 0 [0] MPI startup(): 37 0 rdmap150s0-rdm 0 [0] MPI startup(): 38 0 rdmap150s0-rdm 0 [0] MPI startup(): 39 0 rdmap150s0-rdm 0 [0] MPI startup(): 40 0 rdmap150s0-rdm 0 [0] MPI startup(): 41 0 rdmap150s0-rdm 0 [0] MPI startup(): 42 0 rdmap150s0-rdm 0 [0] MPI startup(): 43 0 rdmap150s0-rdm 0 [0] MPI startup(): 44 0 rdmap150s0-rdm 0 [0] MPI startup(): 45 0 rdmap150s0-rdm 0 [0] MPI startup(): 46 0 rdmap150s0-rdm 0 [0] MPI startup(): 47 0 rdmap150s0-rdm 0 [0] MPI startup(): 48 0 rdmap150s0-rdm 0 [0] MPI startup(): 49 0 rdmap150s0-rdm 0 [0] MPI startup(): 50 0 rdmap150s0-rdm 0 [0] MPI startup(): 51 0 rdmap150s0-rdm 0 [0] MPI startup(): 52 0 rdmap150s0-rdm 0 [0] MPI startup(): 53 0 rdmap150s0-rdm 0 [0] MPI startup(): 54 0 rdmap150s0-rdm 0 [0] MPI startup(): 55 0 rdmap150s0-rdm 0 [0] MPI startup(): 56 0 rdmap150s0-rdm 0 [0] MPI startup(): 57 0 rdmap150s0-rdm 0 [0] MPI startup(): 58 0 rdmap150s0-rdm 0 [0] MPI startup(): 59 0 rdmap150s0-rdm 0 [0] MPI startup(): 60 0 rdmap150s0-rdm 0 [0] MPI startup(): 61 0 rdmap150s0-rdm 0 [0] MPI startup(): 62 0 rdmap150s0-rdm 0 [0] MPI startup(): 63 0 rdmap150s0-rdm 0 [0] MPI startup(): 64 0 rdmap150s0-rdm 0 [0] MPI startup(): 65 0 rdmap150s0-rdm 0 [0] MPI startup(): 66 0 rdmap150s0-rdm 0 [0] MPI startup(): 67 0 rdmap150s0-rdm 0 [0] MPI startup(): 68 0 rdmap150s0-rdm 0 [0] MPI startup(): 69 0 rdmap150s0-rdm 0 [0] MPI startup(): 70 0 rdmap150s0-rdm 0 [0] MPI startup(): 71 0 rdmap150s0-rdm 0 [0] MPI startup(): 72 0 rdmap150s0-rdm 0 [0] MPI startup(): 73 0 rdmap150s0-rdm 0 [0] MPI startup(): 74 0 rdmap150s0-rdm 0 [0] MPI startup(): 75 0 rdmap150s0-rdm 0 [0] MPI startup(): 76 0 rdmap150s0-rdm 0 [0] MPI startup(): 77 0 rdmap150s0-rdm 0 [0] MPI startup(): 78 0 rdmap150s0-rdm 0 [0] MPI startup(): 79 0 rdmap150s0-rdm 0 [0] MPI startup(): 80 0 rdmap150s0-rdm 0 [0] MPI startup(): 81 0 rdmap150s0-rdm 0 [0] MPI startup(): 82 0 rdmap150s0-rdm 0 [0] MPI startup(): 83 0 rdmap150s0-rdm 0 [0] MPI startup(): 84 0 rdmap150s0-rdm 0 [0] MPI startup(): 85 0 rdmap150s0-rdm 0 [0] MPI startup(): 86 0 rdmap150s0-rdm 0 [0] MPI startup(): 87 0 rdmap150s0-rdm 0 [0] MPI startup(): 88 0 rdmap150s0-rdm 0 [0] MPI startup(): 89 0 rdmap150s0-rdm 0 [0] MPI startup(): 90 0 rdmap150s0-rdm 0 [0] MPI startup(): 91 0 rdmap150s0-rdm 0 [0] MPI startup(): 92 0 rdmap150s0-rdm 0 [0] MPI startup(): 93 0 rdmap150s0-rdm 0 [0] MPI startup(): 94 0 rdmap150s0-rdm 0 [0] MPI startup(): 95 0 rdmap150s0-rdm 0 [0] MPI startup(): 96 0 rdmap150s0-rdm 0 [0] MPI startup(): 97 0 rdmap150s0-rdm 0 [0] MPI startup(): 98 0 rdmap150s0-rdm 0 [0] MPI startup(): 99 0 rdmap150s0-rdm 0 [0] MPI startup(): 100 0 rdmap150s0-rdm 0 [0] MPI startup(): 101 0 rdmap150s0-rdm 0 [0] MPI startup(): 102 0 rdmap150s0-rdm 0 [0] MPI startup(): 103 0 rdmap150s0-rdm 0 [0] MPI startup(): 104 0 rdmap150s0-rdm 0 [0] MPI startup(): 105 0 rdmap150s0-rdm 0 [0] MPI startup(): 106 0 rdmap150s0-rdm 0 [0] MPI startup(): 107 0 rdmap150s0-rdm 0 [0] MPI startup(): 108 0 rdmap150s0-rdm 0 [0] MPI startup(): 109 0 rdmap150s0-rdm 0 [0] MPI startup(): 110 0 rdmap150s0-rdm 0 [0] MPI startup(): 111 0 rdmap150s0-rdm 0 [0] MPI startup(): 112 0 rdmap150s0-rdm 0 [0] MPI startup(): 113 0 rdmap150s0-rdm 0 [0] MPI startup(): 114 0 rdmap150s0-rdm 0 [0] MPI startup(): 115 0 rdmap150s0-rdm 0 [0] MPI startup(): 116 0 rdmap150s0-rdm 0 [0] MPI startup(): 117 0 rdmap150s0-rdm 0 [0] MPI startup(): 118 0 rdmap150s0-rdm 0 [0] MPI startup(): 119 0 rdmap150s0-rdm 0 [0] MPI startup(): 120 0 rdmap150s0-rdm 0 [0] MPI startup(): 121 0 rdmap150s0-rdm 0 [0] MPI startup(): 122 0 rdmap150s0-rdm 0 [0] MPI startup(): 123 0 rdmap150s0-rdm 0 [0] MPI startup(): 124 0 rdmap150s0-rdm 0 [0] MPI startup(): 125 0 rdmap150s0-rdm 0 [0] MPI startup(): 126 0 rdmap150s0-rdm 0 [0] MPI startup(): 127 0 rdmap150s0-rdm 0 [0] MPI startup(): 128 0 rdmap150s0-rdm 0 [0] MPI startup(): 129 0 rdmap150s0-rdm 0 [0] MPI startup(): 130 0 rdmap150s0-rdm 0 [0] MPI startup(): 131 0 rdmap150s0-rdm 0 [0] MPI startup(): 132 0 rdmap150s0-rdm 0 [0] MPI startup(): 133 0 rdmap150s0-rdm 0 [0] MPI startup(): 134 0 rdmap150s0-rdm 0 [0] MPI startup(): 135 0 rdmap150s0-rdm 0 [0] MPI startup(): 136 0 rdmap150s0-rdm 0 [0] MPI startup(): 137 0 rdmap150s0-rdm 0 [0] MPI startup(): 138 0 rdmap150s0-rdm 0 [0] MPI startup(): 139 0 rdmap150s0-rdm 0 [0] MPI startup(): 140 0 rdmap150s0-rdm 0 [0] MPI startup(): 141 0 rdmap150s0-rdm 0 [0] MPI startup(): 142 0 rdmap150s0-rdm 0 [0] MPI startup(): 143 0 rdmap150s0-rdm 0 [0] MPI startup(): 144 0 rdmap150s0-rdm 0 [0] MPI startup(): 145 0 rdmap150s0-rdm 0 [0] MPI startup(): 146 0 rdmap150s0-rdm 0 [0] MPI startup(): 147 0 rdmap150s0-rdm 0 [0] MPI startup(): 148 0 rdmap150s0-rdm 0 [0] MPI startup(): 149 0 rdmap150s0-rdm 0 [0] MPI startup(): 150 0 rdmap150s0-rdm 0 [0] MPI startup(): 151 0 rdmap150s0-rdm 0 [0] MPI startup(): 152 0 rdmap150s0-rdm 0 [0] MPI startup(): 153 0 rdmap150s0-rdm 0 [0] MPI startup(): 154 0 rdmap150s0-rdm 0 [0] MPI startup(): 155 0 rdmap150s0-rdm 0 [0] MPI startup(): 156 0 rdmap150s0-rdm 0 [0] MPI startup(): 157 0 rdmap150s0-rdm 0 [0] MPI startup(): 158 0 rdmap150s0-rdm 0 [0] MPI startup(): 159 0 rdmap150s0-rdm 0 [0] MPI startup(): 160 0 rdmap150s0-rdm 0 [0] MPI startup(): 161 0 rdmap150s0-rdm 0 [0] MPI startup(): 162 0 rdmap150s0-rdm 0 [0] MPI startup(): 163 0 rdmap150s0-rdm 0 [0] MPI startup(): 164 0 rdmap150s0-rdm 0 [0] MPI startup(): 165 0 rdmap150s0-rdm 0 [0] MPI startup(): 166 0 rdmap150s0-rdm 0 [0] MPI startup(): 167 0 rdmap150s0-rdm 0 [0] MPI startup(): 168 0 rdmap150s0-rdm 0 [0] MPI startup(): 169 0 rdmap150s0-rdm 0 [0] MPI startup(): 170 0 rdmap150s0-rdm 0 [0] MPI startup(): 171 0 rdmap150s0-rdm 0 [0] MPI startup(): 172 0 rdmap150s0-rdm 0 [0] MPI startup(): 173 0 rdmap150s0-rdm 0 [0] MPI startup(): 174 0 rdmap150s0-rdm 0 [0] MPI startup(): 175 0 rdmap150s0-rdm 0 [0] MPI startup(): 176 0 rdmap150s0-rdm 0 [0] MPI startup(): 177 0 rdmap150s0-rdm 0 [0] MPI startup(): 178 0 rdmap150s0-rdm 0 [0] MPI startup(): 179 0 rdmap150s0-rdm 0 [0] MPI startup(): 180 0 rdmap150s0-rdm 0 [0] MPI startup(): 181 0 rdmap150s0-rdm 0 [0] MPI startup(): 182 0 rdmap150s0-rdm 0 [0] MPI startup(): 183 0 rdmap150s0-rdm 0 [0] MPI startup(): 184 0 rdmap150s0-rdm 0 [0] MPI startup(): 185 0 rdmap150s0-rdm 0 [0] MPI startup(): 186 0 rdmap150s0-rdm 0 [0] MPI startup(): 187 0 rdmap150s0-rdm 0 [0] MPI startup(): 188 0 rdmap150s0-rdm 0 [0] MPI startup(): 189 0 rdmap150s0-rdm 0 [0] MPI startup(): 190 0 rdmap150s0-rdm 0 [0] MPI startup(): 191 0 rdmap150s0-rdm 0 [0] MPI startup(): ===== CPU pinning ===== [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 15233 efa-od-az-1a-dy-c7i-48xlarge-1 {48} [0] MPI startup(): 1 15234 efa-od-az-1a-dy-c7i-48xlarge-1 {49} [0] MPI startup(): 2 15235 efa-od-az-1a-dy-c7i-48xlarge-1 {50} [0] MPI startup(): 3 15236 efa-od-az-1a-dy-c7i-48xlarge-1 {51} [0] MPI startup(): 4 15237 efa-od-az-1a-dy-c7i-48xlarge-1 {52} [0] MPI startup(): 5 15238 efa-od-az-1a-dy-c7i-48xlarge-1 {53} [0] MPI startup(): 6 15239 efa-od-az-1a-dy-c7i-48xlarge-1 {54} [0] MPI startup(): 7 15240 efa-od-az-1a-dy-c7i-48xlarge-1 {55} [0] MPI startup(): 8 15241 efa-od-az-1a-dy-c7i-48xlarge-1 {56} [0] MPI startup(): 9 15242 efa-od-az-1a-dy-c7i-48xlarge-1 {57} [0] MPI startup(): 10 15243 efa-od-az-1a-dy-c7i-48xlarge-1 {58} [0] MPI startup(): 11 15244 efa-od-az-1a-dy-c7i-48xlarge-1 {59} [0] MPI startup(): 12 15245 efa-od-az-1a-dy-c7i-48xlarge-1 {60} [0] MPI startup(): 13 15250 efa-od-az-1a-dy-c7i-48xlarge-1 {61} [0] MPI startup(): 14 15253 efa-od-az-1a-dy-c7i-48xlarge-1 {62} [0] MPI startup(): 15 15254 efa-od-az-1a-dy-c7i-48xlarge-1 {63} [0] MPI startup(): 16 15258 efa-od-az-1a-dy-c7i-48xlarge-1 {64} [0] MPI startup(): 17 15265 efa-od-az-1a-dy-c7i-48xlarge-1 {65} [0] MPI startup(): 18 15269 efa-od-az-1a-dy-c7i-48xlarge-1 {66} [0] MPI startup(): 19 15270 efa-od-az-1a-dy-c7i-48xlarge-1 {67} [0] MPI startup(): 20 15273 efa-od-az-1a-dy-c7i-48xlarge-1 {68} [0] MPI startup(): 21 15276 efa-od-az-1a-dy-c7i-48xlarge-1 {69} [0] MPI startup(): 22 15279 efa-od-az-1a-dy-c7i-48xlarge-1 {70} [0] MPI startup(): 23 15285 efa-od-az-1a-dy-c7i-48xlarge-1 {71} [0] MPI startup(): 24 15287 efa-od-az-1a-dy-c7i-48xlarge-1 {72} [0] MPI startup(): 25 15293 efa-od-az-1a-dy-c7i-48xlarge-1 {73} [0] MPI startup(): 26 15294 efa-od-az-1a-dy-c7i-48xlarge-1 {74} [0] MPI startup(): 27 15297 efa-od-az-1a-dy-c7i-48xlarge-1 {75} [0] MPI startup(): 28 15298 efa-od-az-1a-dy-c7i-48xlarge-1 {76} [0] MPI startup(): 29 15299 efa-od-az-1a-dy-c7i-48xlarge-1 {77} [0] MPI startup(): 30 15303 efa-od-az-1a-dy-c7i-48xlarge-1 {78} [0] MPI startup(): 31 15304 efa-od-az-1a-dy-c7i-48xlarge-1 {79} [0] MPI startup(): 32 15307 efa-od-az-1a-dy-c7i-48xlarge-1 {80} [0] MPI startup(): 33 15311 efa-od-az-1a-dy-c7i-48xlarge-1 {81} [0] MPI startup(): 34 15312 efa-od-az-1a-dy-c7i-48xlarge-1 {82} [0] MPI startup(): 35 15314 efa-od-az-1a-dy-c7i-48xlarge-1 {83} [0] MPI startup(): 36 15315 efa-od-az-1a-dy-c7i-48xlarge-1 {84} [0] MPI startup(): 37 15316 efa-od-az-1a-dy-c7i-48xlarge-1 {85} [0] MPI startup(): 38 15321 efa-od-az-1a-dy-c7i-48xlarge-1 {86} [0] MPI startup(): 39 15325 efa-od-az-1a-dy-c7i-48xlarge-1 {87} [0] MPI startup(): 40 15326 efa-od-az-1a-dy-c7i-48xlarge-1 {88} [0] MPI startup(): 41 15328 efa-od-az-1a-dy-c7i-48xlarge-1 {89} [0] MPI startup(): 42 15329 efa-od-az-1a-dy-c7i-48xlarge-1 {90} [0] MPI startup(): 43 15330 efa-od-az-1a-dy-c7i-48xlarge-1 {91} [0] MPI startup(): 44 15336 efa-od-az-1a-dy-c7i-48xlarge-1 {92} [0] MPI startup(): 45 15337 efa-od-az-1a-dy-c7i-48xlarge-1 {93} [0] MPI startup(): 46 15340 efa-od-az-1a-dy-c7i-48xlarge-1 {94} [0] MPI startup(): 47 15344 efa-od-az-1a-dy-c7i-48xlarge-1 {95} [0] MPI startup(): 48 15348 efa-od-az-1a-dy-c7i-48xlarge-1 {0} [0] MPI startup(): 49 15349 efa-od-az-1a-dy-c7i-48xlarge-1 {1} [0] MPI startup(): 50 15352 efa-od-az-1a-dy-c7i-48xlarge-1 {2} [0] MPI startup(): 51 15356 efa-od-az-1a-dy-c7i-48xlarge-1 {3} [0] MPI startup(): 52 15362 efa-od-az-1a-dy-c7i-48xlarge-1 {4} [0] MPI startup(): 53 15364 efa-od-az-1a-dy-c7i-48xlarge-1 {5} [0] MPI startup(): 54 15367 efa-od-az-1a-dy-c7i-48xlarge-1 {6} [0] MPI startup(): 55 15372 efa-od-az-1a-dy-c7i-48xlarge-1 {7} [0] MPI startup(): 56 15373 efa-od-az-1a-dy-c7i-48xlarge-1 {8} [0] MPI startup(): 57 15377 efa-od-az-1a-dy-c7i-48xlarge-1 {9} [0] MPI startup(): 58 15378 efa-od-az-1a-dy-c7i-48xlarge-1 {10} [0] MPI startup(): 59 15381 efa-od-az-1a-dy-c7i-48xlarge-1 {11} [0] MPI startup(): 60 15386 efa-od-az-1a-dy-c7i-48xlarge-1 {12} [0] MPI startup(): 61 15387 efa-od-az-1a-dy-c7i-48xlarge-1 {13} [0] MPI startup(): 62 15389 efa-od-az-1a-dy-c7i-48xlarge-1 {14} [0] MPI startup(): 63 15393 efa-od-az-1a-dy-c7i-48xlarge-1 {15} [0] MPI startup(): 64 15395 efa-od-az-1a-dy-c7i-48xlarge-1 {16} [0] MPI startup(): 65 15396 efa-od-az-1a-dy-c7i-48xlarge-1 {17} [0] MPI startup(): 66 15402 efa-od-az-1a-dy-c7i-48xlarge-1 {18} [0] MPI startup(): 67 15403 efa-od-az-1a-dy-c7i-48xlarge-1 {19} [0] MPI startup(): 68 15404 efa-od-az-1a-dy-c7i-48xlarge-1 {20} [0] MPI startup(): 69 15405 efa-od-az-1a-dy-c7i-48xlarge-1 {21} [0] MPI startup(): 70 15406 efa-od-az-1a-dy-c7i-48xlarge-1 {22} [0] MPI startup(): 71 15410 efa-od-az-1a-dy-c7i-48xlarge-1 {23} [0] MPI startup(): 72 15413 efa-od-az-1a-dy-c7i-48xlarge-1 {24} [0] MPI startup(): 73 15414 efa-od-az-1a-dy-c7i-48xlarge-1 {25} [0] MPI startup(): 74 15417 efa-od-az-1a-dy-c7i-48xlarge-1 {26} [0] MPI startup(): 75 15418 efa-od-az-1a-dy-c7i-48xlarge-1 {27} [0] MPI startup(): 76 15420 efa-od-az-1a-dy-c7i-48xlarge-1 {28} [0] MPI startup(): 77 15421 efa-od-az-1a-dy-c7i-48xlarge-1 {29} [0] MPI startup(): 78 15422 efa-od-az-1a-dy-c7i-48xlarge-1 {30} [0] MPI startup(): 79 15423 efa-od-az-1a-dy-c7i-48xlarge-1 {31} [0] MPI startup(): 80 15424 efa-od-az-1a-dy-c7i-48xlarge-1 {32} [0] MPI startup(): 81 15425 efa-od-az-1a-dy-c7i-48xlarge-1 {33} [0] MPI startup(): 82 15426 efa-od-az-1a-dy-c7i-48xlarge-1 {34} [0] MPI startup(): 83 15427 efa-od-az-1a-dy-c7i-48xlarge-1 {35} [0] MPI startup(): 84 15428 efa-od-az-1a-dy-c7i-48xlarge-1 {36} [0] MPI startup(): 85 15429 efa-od-az-1a-dy-c7i-48xlarge-1 {37} [0] MPI startup(): 86 15430 efa-od-az-1a-dy-c7i-48xlarge-1 {38} [0] MPI startup(): 87 15431 efa-od-az-1a-dy-c7i-48xlarge-1 {39} [0] MPI startup(): 88 15432 efa-od-az-1a-dy-c7i-48xlarge-1 {40} [0] MPI startup(): 89 15433 efa-od-az-1a-dy-c7i-48xlarge-1 {41} [0] MPI startup(): 90 15434 efa-od-az-1a-dy-c7i-48xlarge-1 {42} [0] MPI startup(): 91 15435 efa-od-az-1a-dy-c7i-48xlarge-1 {43} [0] MPI startup(): 92 15436 efa-od-az-1a-dy-c7i-48xlarge-1 {44} [0] MPI startup(): 93 15437 efa-od-az-1a-dy-c7i-48xlarge-1 {45} [0] MPI startup(): 94 15438 efa-od-az-1a-dy-c7i-48xlarge-1 {46} [0] MPI startup(): 95 15439 efa-od-az-1a-dy-c7i-48xlarge-1 {47} [0] MPI startup(): 96 15440 efa-od-az-1a-dy-c7i-48xlarge-1 {48} [0] MPI startup(): 97 15441 efa-od-az-1a-dy-c7i-48xlarge-1 {49} [0] MPI startup(): 98 15442 efa-od-az-1a-dy-c7i-48xlarge-1 {50} [0] MPI startup(): 99 15443 efa-od-az-1a-dy-c7i-48xlarge-1 {51} [0] MPI startup(): 100 15444 efa-od-az-1a-dy-c7i-48xlarge-1 {52} [0] MPI startup(): 101 15445 efa-od-az-1a-dy-c7i-48xlarge-1 {53} [0] MPI startup(): 102 15446 efa-od-az-1a-dy-c7i-48xlarge-1 {54} [0] MPI startup(): 103 15447 efa-od-az-1a-dy-c7i-48xlarge-1 {55} [0] MPI startup(): 104 15448 efa-od-az-1a-dy-c7i-48xlarge-1 {56} [0] MPI startup(): 105 15450 efa-od-az-1a-dy-c7i-48xlarge-1 {57} [0] MPI startup(): 106 15451 efa-od-az-1a-dy-c7i-48xlarge-1 {58} [0] MPI startup(): 107 15452 efa-od-az-1a-dy-c7i-48xlarge-1 {59} [0] MPI startup(): 108 15453 efa-od-az-1a-dy-c7i-48xlarge-1 {60} [0] MPI startup(): 109 15455 efa-od-az-1a-dy-c7i-48xlarge-1 {61} [0] MPI startup(): 110 15456 efa-od-az-1a-dy-c7i-48xlarge-1 {62} [0] MPI startup(): 111 15457 efa-od-az-1a-dy-c7i-48xlarge-1 {63} [0] MPI startup(): 112 15458 efa-od-az-1a-dy-c7i-48xlarge-1 {64} [0] MPI startup(): 113 15459 efa-od-az-1a-dy-c7i-48xlarge-1 {65} [0] MPI startup(): 114 15460 efa-od-az-1a-dy-c7i-48xlarge-1 {66} [0] MPI startup(): 115 15461 efa-od-az-1a-dy-c7i-48xlarge-1 {67} [0] MPI startup(): 116 15462 efa-od-az-1a-dy-c7i-48xlarge-1 {68} [0] MPI startup(): 117 15463 efa-od-az-1a-dy-c7i-48xlarge-1 {69} [0] MPI startup(): 118 15464 efa-od-az-1a-dy-c7i-48xlarge-1 {70} [0] MPI startup(): 119 15465 efa-od-az-1a-dy-c7i-48xlarge-1 {71} [0] MPI startup(): 120 15466 efa-od-az-1a-dy-c7i-48xlarge-1 {72} [0] MPI startup(): 121 15467 efa-od-az-1a-dy-c7i-48xlarge-1 {73} [0] MPI startup(): 122 15468 efa-od-az-1a-dy-c7i-48xlarge-1 {74} [0] MPI startup(): 123 15469 efa-od-az-1a-dy-c7i-48xlarge-1 {75} [0] MPI startup(): 124 15470 efa-od-az-1a-dy-c7i-48xlarge-1 {76} [0] MPI startup(): 125 15471 efa-od-az-1a-dy-c7i-48xlarge-1 {77} [0] MPI startup(): 126 15472 efa-od-az-1a-dy-c7i-48xlarge-1 {78} [0] MPI startup(): 127 15473 efa-od-az-1a-dy-c7i-48xlarge-1 {79} [0] MPI startup(): 128 15474 efa-od-az-1a-dy-c7i-48xlarge-1 {80} [0] MPI startup(): 129 15475 efa-od-az-1a-dy-c7i-48xlarge-1 {81} [0] MPI startup(): 130 15478 efa-od-az-1a-dy-c7i-48xlarge-1 {82} [0] MPI startup(): 131 15479 efa-od-az-1a-dy-c7i-48xlarge-1 {83} [0] MPI startup(): 132 15484 efa-od-az-1a-dy-c7i-48xlarge-1 {84} [0] MPI startup(): 133 15487 efa-od-az-1a-dy-c7i-48xlarge-1 {85} [0] MPI startup(): 134 15490 efa-od-az-1a-dy-c7i-48xlarge-1 {86} [0] MPI startup(): 135 15495 efa-od-az-1a-dy-c7i-48xlarge-1 {87} [0] MPI startup(): 136 15497 efa-od-az-1a-dy-c7i-48xlarge-1 {88} [0] MPI startup(): 137 15501 efa-od-az-1a-dy-c7i-48xlarge-1 {89} [0] MPI startup(): 138 15509 efa-od-az-1a-dy-c7i-48xlarge-1 {90} [0] MPI startup(): 139 15515 efa-od-az-1a-dy-c7i-48xlarge-1 {91} [0] MPI startup(): 140 15516 efa-od-az-1a-dy-c7i-48xlarge-1 {92} [0] MPI startup(): 141 15521 efa-od-az-1a-dy-c7i-48xlarge-1 {93} [0] MPI startup(): 142 15529 efa-od-az-1a-dy-c7i-48xlarge-1 {94} [0] MPI startup(): 143 15535 efa-od-az-1a-dy-c7i-48xlarge-1 {95} [0] MPI startup(): 144 15547 efa-od-az-1a-dy-c7i-48xlarge-1 {0} [0] MPI startup(): 145 15548 efa-od-az-1a-dy-c7i-48xlarge-1 {1} [0] MPI startup(): 146 15549 efa-od-az-1a-dy-c7i-48xlarge-1 {2} [0] MPI startup(): 147 15550 efa-od-az-1a-dy-c7i-48xlarge-1 {3} [0] MPI startup(): 148 15551 efa-od-az-1a-dy-c7i-48xlarge-1 {4} [0] MPI startup(): 149 15552 efa-od-az-1a-dy-c7i-48xlarge-1 {5} [0] MPI startup(): 150 15553 efa-od-az-1a-dy-c7i-48xlarge-1 {6} [0] MPI startup(): 151 15554 efa-od-az-1a-dy-c7i-48xlarge-1 {7} [0] MPI startup(): 152 15555 efa-od-az-1a-dy-c7i-48xlarge-1 {8} [0] MPI startup(): 153 15556 efa-od-az-1a-dy-c7i-48xlarge-1 {9} [0] MPI startup(): 154 15557 efa-od-az-1a-dy-c7i-48xlarge-1 {10} [0] MPI startup(): 155 15558 efa-od-az-1a-dy-c7i-48xlarge-1 {11} [0] MPI startup(): 156 15559 efa-od-az-1a-dy-c7i-48xlarge-1 {12} [0] MPI startup(): 157 15560 efa-od-az-1a-dy-c7i-48xlarge-1 {13} [0] MPI startup(): 158 15561 efa-od-az-1a-dy-c7i-48xlarge-1 {14} [0] MPI startup(): 159 15562 efa-od-az-1a-dy-c7i-48xlarge-1 {15} [0] MPI startup(): 160 15563 efa-od-az-1a-dy-c7i-48xlarge-1 {16} [0] MPI startup(): 161 15564 efa-od-az-1a-dy-c7i-48xlarge-1 {17} [0] MPI startup(): 162 15565 efa-od-az-1a-dy-c7i-48xlarge-1 {18} [0] MPI startup(): 163 15566 efa-od-az-1a-dy-c7i-48xlarge-1 {19} [0] MPI startup(): 164 15567 efa-od-az-1a-dy-c7i-48xlarge-1 {20} [0] MPI startup(): 165 15568 efa-od-az-1a-dy-c7i-48xlarge-1 {21} [0] MPI startup(): 166 15569 efa-od-az-1a-dy-c7i-48xlarge-1 {22} [0] MPI startup(): 167 15570 efa-od-az-1a-dy-c7i-48xlarge-1 {23} [0] MPI startup(): 168 15572 efa-od-az-1a-dy-c7i-48xlarge-1 {24} [0] MPI startup(): 169 15573 efa-od-az-1a-dy-c7i-48xlarge-1 {25} [0] MPI startup(): 170 15574 efa-od-az-1a-dy-c7i-48xlarge-1 {26} [0] MPI startup(): 171 15575 efa-od-az-1a-dy-c7i-48xlarge-1 {27} [0] MPI startup(): 172 15576 efa-od-az-1a-dy-c7i-48xlarge-1 {28} [0] MPI startup(): 173 15577 efa-od-az-1a-dy-c7i-48xlarge-1 {29} [0] MPI startup(): 174 15578 efa-od-az-1a-dy-c7i-48xlarge-1 {30} [0] MPI startup(): 175 15579 efa-od-az-1a-dy-c7i-48xlarge-1 {31} [0] MPI startup(): 176 15580 efa-od-az-1a-dy-c7i-48xlarge-1 {32} [0] MPI startup(): 177 15581 efa-od-az-1a-dy-c7i-48xlarge-1 {33} [0] MPI startup(): 178 15582 efa-od-az-1a-dy-c7i-48xlarge-1 {34} [0] MPI startup(): 179 15583 efa-od-az-1a-dy-c7i-48xlarge-1 {35} [0] MPI startup(): 180 15584 efa-od-az-1a-dy-c7i-48xlarge-1 {36} [0] MPI startup(): 181 15585 efa-od-az-1a-dy-c7i-48xlarge-1 {37} [0] MPI startup(): 182 15586 efa-od-az-1a-dy-c7i-48xlarge-1 {38} [0] MPI startup(): 183 15587 efa-od-az-1a-dy-c7i-48xlarge-1 {39} [0] MPI startup(): 184 15588 efa-od-az-1a-dy-c7i-48xlarge-1 {40} [0] MPI startup(): 185 15589 efa-od-az-1a-dy-c7i-48xlarge-1 {41} [0] MPI startup(): 186 15590 efa-od-az-1a-dy-c7i-48xlarge-1 {42} [0] MPI startup(): 187 15591 efa-od-az-1a-dy-c7i-48xlarge-1 {43} [0] MPI startup(): 188 15592 efa-od-az-1a-dy-c7i-48xlarge-1 {44} [0] MPI startup(): 189 15593 efa-od-az-1a-dy-c7i-48xlarge-1 {45} [0] MPI startup(): 190 15594 efa-od-az-1a-dy-c7i-48xlarge-1 {46} [0] MPI startup(): 191 15595 efa-od-az-1a-dy-c7i-48xlarge-1 {47} [0] MPI startup(): I\_MPI\_OFI\_LIBRARY\_INTERNAL=1 [0] MPI startup(): I\_MPI\_ROOT=/opt/intel/oneapi/mpi/2021.14 [0] MPI startup(): I\_MPI\_MPIRUN=mpirun [0] MPI startup(): I\_MPI\_BIND\_WIN\_ALLOCATE=localalloc [0] MPI startup(): I\_MPI\_HYDRA\_BOOTSTRAP\_EXEC\_EXTRA\_ARGS=--external-launcher [0] MPI startup(): I\_MPI\_HYDRA\_TOPOLIB=hwloc [0] MPI startup(): I\_MPI\_HYDRA\_BOOTSTRAP=slurm [0] MPI startup(): I\_MPI\_RETURN\_WIN\_MEM\_NUMA=1 [0] MPI startup(): I\_MPI\_INTERNAL\_MEM\_POLICY=default [0] MPI startup(): I\_MPI\_FABRICS=shm:ofi [0] MPI startup(): I\_MPI\_OFI\_PROVIDER=efa [0] MPI startup(): I\_MPI\_DEBUG=10 #---------------------------------------------------------------- # Intel(R) MPI Benchmarks 2021.8, MPI-1 part #---------------------------------------------------------------- # Date : Mon Feb 17 11:30:57 2025 # Machine : x86\_64 # System : Linux # Release : 5.14.0-427.26.1.el9\_4.x86\_64 # Version : #1 SMP PREEMPT\_DYNAMIC Fri Jul 5 11:34:54 EDT 2024 # MPI Version : 3.1 # MPI Thread Environment: # Calling sequence was: # IMB-MPI1 barrier # Minimum message length in bytes: 0 # Maximum message length in bytes: 4194304 # # MPI\_Datatype : MPI\_BYTE # MPI\_Datatype for reductions : MPI\_FLOAT # MPI\_Op : MPI\_SUM # # # List of Benchmarks to run: # Barrier #--------------------------------------------------- # Benchmarking Barrier # #processes = 2 # ( 190 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 1000 0.37 0.37 0.37 #--------------------------------------------------- # Benchmarking Barrier # #processes = 4 # ( 188 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 1000 0.60 0.60 0.60 #--------------------------------------------------- # Benchmarking Barrier # #processes = 8 # ( 184 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 376 0.68 0.68 0.68 #--------------------------------------------------- # Benchmarking Barrier # #processes = 16 # ( 176 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 283 25936.42 26010.59 25971.98 #--------------------------------------------------- # Benchmarking Barrier # #processes = 32 # ( 160 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 194 31180.34 31360.85 31291.58 #--------------------------------------------------- # Benchmarking Barrier # #processes = 64 # ( 128 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 196 43785.67 43984.83 43892.37 #--------------------------------------------------- # Benchmarking Barrier # #processes = 128 # ( 64 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 140 57157.02 57571.38 57406.40 #--------------------------------------------------- # Benchmarking Barrier # #processes = 192 #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 109 52944.94 53403.62 53220.60 # All processes entering MPI\_Finalize ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 17, 2025, 12:05pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/6 "2025-02-17T12:05:46Z")
@TobiasK  if you see the pinning information, 192 mpi ranks has been assigned to cpu 0-95. My expectation is that 192 mpi ranks should be assigned to cpu 0-191. Thanks! [0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20241121 (id: e7829d6) [0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric loaded: libfabric.so.1 [0] MPI startup(): libfabric version: 1.21.0-impi [0] MPI startup(): max number of MPI\_Request per vci: 67108864 (pools: 1) [0] MPI startup(): libfabric provider: efa [0] MPI startup(): shm segment size (150 MB per rank) \* (192 local ranks) = 28870 MB total [0] MPI startup(): File “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi\_efa\_100\_x1.dat” not found [0] MPI startup(): File “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi\_efa\_100.dat” not found [0] MPI startup(): File “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi\_efa.dat” not found [0] MPI startup(): Load tuning file: “/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning\_spr\_shm-ofi.dat” [0] MPI startup(): threading: mode: direct [0] MPI startup(): threading: vcis: 1 [0] MPI startup(): threading: app\_threads: -1 [0] MPI startup(): threading: runtime: generic [0] MPI startup(): threading: progress\_threads: 0 [0] MPI startup(): threading: async\_progress: 0 [0] MPI startup(): threading: async\_progress coll split: 0 [0] MPI startup(): threading: lock\_level: global [0] MPI startup(): tag bits available: 30 (TAG\_UB value: 1073741823) [0] MPI startup(): source bits available: 30 (Maximal number of rank: 1073741823) [0] MPI startup(): Number of NICs: 1 [0] MPI startup(): ===== NIC pinning on efa-od-az-1a-dy-c7i-48xlarge-1 ===== [0] MPI startup(): Rank Thread id Pin nic Nic id [0] MPI startup(): 0 0 rdmap150s0-rdm 0 [0] MPI startup(): 1 0 rdmap150s0-rdm 0 [0] MPI startup(): 2 0 rdmap150s0-rdm 0 [0] MPI startup(): 3 0 rdmap150s0-rdm 0 [0] MPI startup(): 4 0 rdmap150s0-rdm 0 [0] MPI startup(): 5 0 rdmap150s0-rdm 0 [0] MPI startup(): 6 0 rdmap150s0-rdm 0 [0] MPI startup(): 7 0 rdmap150s0-rdm 0 [0] MPI startup(): 8 0 rdmap150s0-rdm 0 [0] MPI startup(): 9 0 rdmap150s0-rdm 0 [0] MPI startup(): 10 0 rdmap150s0-rdm 0 [0] MPI startup(): 11 0 rdmap150s0-rdm 0 [0] MPI startup(): 12 0 rdmap150s0-rdm 0 [0] MPI startup(): 13 0 rdmap150s0-rdm 0 [0] MPI startup(): 14 0 rdmap150s0-rdm 0 [0] MPI startup(): 15 0 rdmap150s0-rdm 0 [0] MPI startup(): 16 0 rdmap150s0-rdm 0 [0] MPI startup(): 17 0 rdmap150s0-rdm 0 [0] MPI startup(): 18 0 rdmap150s0-rdm 0 [0] MPI startup(): 19 0 rdmap150s0-rdm 0 [0] MPI startup(): 20 0 rdmap150s0-rdm 0 [0] MPI startup(): 21 0 rdmap150s0-rdm 0 [0] MPI startup(): 22 0 rdmap150s0-rdm 0 [0] MPI startup(): 23 0 rdmap150s0-rdm 0 [0] MPI startup(): 24 0 rdmap150s0-rdm 0 [0] MPI startup(): 25 0 rdmap150s0-rdm 0 [0] MPI startup(): 26 0 rdmap150s0-rdm 0 [0] MPI startup(): 27 0 rdmap150s0-rdm 0 [0] MPI startup(): 28 0 rdmap150s0-rdm 0 [0] MPI startup(): 29 0 rdmap150s0-rdm 0 [0] MPI startup(): 30 0 rdmap150s0-rdm 0 [0] MPI startup(): 31 0 rdmap150s0-rdm 0 [0] MPI startup(): 32 0 rdmap150s0-rdm 0 [0] MPI startup(): 33 0 rdmap150s0-rdm 0 [0] MPI startup(): 34 0 rdmap150s0-rdm 0 [0] MPI startup(): 35 0 rdmap150s0-rdm 0 [0] MPI startup(): 36 0 rdmap150s0-rdm 0 [0] MPI startup(): 37 0 rdmap150s0-rdm 0 [0] MPI startup(): 38 0 rdmap150s0-rdm 0 [0] MPI startup(): 39 0 rdmap150s0-rdm 0 [0] MPI startup(): 40 0 rdmap150s0-rdm 0 [0] MPI startup(): 41 0 rdmap150s0-rdm 0 [0] MPI startup(): 42 0 rdmap150s0-rdm 0 [0] MPI startup(): 43 0 rdmap150s0-rdm 0 [0] MPI startup(): 44 0 rdmap150s0-rdm 0 [0] MPI startup(): 45 0 rdmap150s0-rdm 0 [0] MPI startup(): 46 0 rdmap150s0-rdm 0 [0] MPI startup(): 47 0 rdmap150s0-rdm 0 [0] MPI startup(): 48 0 rdmap150s0-rdm 0 [0] MPI startup(): 49 0 rdmap150s0-rdm 0 [0] MPI startup(): 50 0 rdmap150s0-rdm 0 [0] MPI startup(): 51 0 rdmap150s0-rdm 0 [0] MPI startup(): 52 0 rdmap150s0-rdm 0 [0] MPI startup(): 53 0 rdmap150s0-rdm 0 [0] MPI startup(): 54 0 rdmap150s0-rdm 0 [0] MPI startup(): 55 0 rdmap150s0-rdm 0 [0] MPI startup(): 56 0 rdmap150s0-rdm 0 [0] MPI startup(): 57 0 rdmap150s0-rdm 0 [0] MPI startup(): 58 0 rdmap150s0-rdm 0 [0] MPI startup(): 59 0 rdmap150s0-rdm 0 [0] MPI startup(): 60 0 rdmap150s0-rdm 0 [0] MPI startup(): 61 0 rdmap150s0-rdm 0 [0] MPI startup(): 62 0 rdmap150s0-rdm 0 [0] MPI startup(): 63 0 rdmap150s0-rdm 0 [0] MPI startup(): 64 0 rdmap150s0-rdm 0 [0] MPI startup(): 65 0 rdmap150s0-rdm 0 [0] MPI startup(): 66 0 rdmap150s0-rdm 0 [0] MPI startup(): 67 0 rdmap150s0-rdm 0 [0] MPI startup(): 68 0 rdmap150s0-rdm 0 [0] MPI startup(): 69 0 rdmap150s0-rdm 0 [0] MPI startup(): 70 0 rdmap150s0-rdm 0 [0] MPI startup(): 71 0 rdmap150s0-rdm 0 [0] MPI startup(): 72 0 rdmap150s0-rdm 0 [0] MPI startup(): 73 0 rdmap150s0-rdm 0 [0] MPI startup(): 74 0 rdmap150s0-rdm 0 [0] MPI startup(): 75 0 rdmap150s0-rdm 0 [0] MPI startup(): 76 0 rdmap150s0-rdm 0 [0] MPI startup(): 77 0 rdmap150s0-rdm 0 [0] MPI startup(): 78 0 rdmap150s0-rdm 0 [0] MPI startup(): 79 0 rdmap150s0-rdm 0 [0] MPI startup(): 80 0 rdmap150s0-rdm 0 [0] MPI startup(): 81 0 rdmap150s0-rdm 0 [0] MPI startup(): 82 0 rdmap150s0-rdm 0 [0] MPI startup(): 83 0 rdmap150s0-rdm 0 [0] MPI startup(): 84 0 rdmap150s0-rdm 0 [0] MPI startup(): 85 0 rdmap150s0-rdm 0 [0] MPI startup(): 86 0 rdmap150s0-rdm 0 [0] MPI startup(): 87 0 rdmap150s0-rdm 0 [0] MPI startup(): 88 0 rdmap150s0-rdm 0 [0] MPI startup(): 89 0 rdmap150s0-rdm 0 [0] MPI startup(): 90 0 rdmap150s0-rdm 0 [0] MPI startup(): 91 0 rdmap150s0-rdm 0 [0] MPI startup(): 92 0 rdmap150s0-rdm 0 [0] MPI startup(): 93 0 rdmap150s0-rdm 0 [0] MPI startup(): 94 0 rdmap150s0-rdm 0 [0] MPI startup(): 95 0 rdmap150s0-rdm 0 [0] MPI startup(): 96 0 rdmap150s0-rdm 0 [0] MPI startup(): 97 0 rdmap150s0-rdm 0 [0] MPI startup(): 98 0 rdmap150s0-rdm 0 [0] MPI startup(): 99 0 rdmap150s0-rdm 0 [0] MPI startup(): 100 0 rdmap150s0-rdm 0 [0] MPI startup(): 101 0 rdmap150s0-rdm 0 [0] MPI startup(): 102 0 rdmap150s0-rdm 0 [0] MPI startup(): 103 0 rdmap150s0-rdm 0 [0] MPI startup(): 104 0 rdmap150s0-rdm 0 [0] MPI startup(): 105 0 rdmap150s0-rdm 0 [0] MPI startup(): 106 0 rdmap150s0-rdm 0 [0] MPI startup(): 107 0 rdmap150s0-rdm 0 [0] MPI startup(): 108 0 rdmap150s0-rdm 0 [0] MPI startup(): 109 0 rdmap150s0-rdm 0 [0] MPI startup(): 110 0 rdmap150s0-rdm 0 [0] MPI startup(): 111 0 rdmap150s0-rdm 0 [0] MPI startup(): 112 0 rdmap150s0-rdm 0 [0] MPI startup(): 113 0 rdmap150s0-rdm 0 [0] MPI startup(): 114 0 rdmap150s0-rdm 0 [0] MPI startup(): 115 0 rdmap150s0-rdm 0 [0] MPI startup(): 116 0 rdmap150s0-rdm 0 [0] MPI startup(): 117 0 rdmap150s0-rdm 0 [0] MPI startup(): 118 0 rdmap150s0-rdm 0 [0] MPI startup(): 119 0 rdmap150s0-rdm 0 [0] MPI startup(): 120 0 rdmap150s0-rdm 0 [0] MPI startup(): 121 0 rdmap150s0-rdm 0 [0] MPI startup(): 122 0 rdmap150s0-rdm 0 [0] MPI startup(): 123 0 rdmap150s0-rdm 0 [0] MPI startup(): 124 0 rdmap150s0-rdm 0 [0] MPI startup(): 125 0 rdmap150s0-rdm 0 [0] MPI startup(): 126 0 rdmap150s0-rdm 0 [0] MPI startup(): 127 0 rdmap150s0-rdm 0 [0] MPI startup(): 128 0 rdmap150s0-rdm 0 [0] MPI startup(): 129 0 rdmap150s0-rdm 0 [0] MPI startup(): 130 0 rdmap150s0-rdm 0 [0] MPI startup(): 131 0 rdmap150s0-rdm 0 [0] MPI startup(): 132 0 rdmap150s0-rdm 0 [0] MPI startup(): 133 0 rdmap150s0-rdm 0 [0] MPI startup(): 134 0 rdmap150s0-rdm 0 [0] MPI startup(): 135 0 rdmap150s0-rdm 0 [0] MPI startup(): 136 0 rdmap150s0-rdm 0 [0] MPI startup(): 137 0 rdmap150s0-rdm 0 [0] MPI startup(): 138 0 rdmap150s0-rdm 0 [0] MPI startup(): 139 0 rdmap150s0-rdm 0 [0] MPI startup(): 140 0 rdmap150s0-rdm 0 [0] MPI startup(): 141 0 rdmap150s0-rdm 0 [0] MPI startup(): 142 0 rdmap150s0-rdm 0 [0] MPI startup(): 143 0 rdmap150s0-rdm 0 [0] MPI startup(): 144 0 rdmap150s0-rdm 0 [0] MPI startup(): 145 0 rdmap150s0-rdm 0 [0] MPI startup(): 146 0 rdmap150s0-rdm 0 [0] MPI startup(): 147 0 rdmap150s0-rdm 0 [0] MPI startup(): 148 0 rdmap150s0-rdm 0 [0] MPI startup(): 149 0 rdmap150s0-rdm 0 [0] MPI startup(): 150 0 rdmap150s0-rdm 0 [0] MPI startup(): 151 0 rdmap150s0-rdm 0 [0] MPI startup(): 152 0 rdmap150s0-rdm 0 [0] MPI startup(): 153 0 rdmap150s0-rdm 0 [0] MPI startup(): 154 0 rdmap150s0-rdm 0 [0] MPI startup(): 155 0 rdmap150s0-rdm 0 [0] MPI startup(): 156 0 rdmap150s0-rdm 0 [0] MPI startup(): 157 0 rdmap150s0-rdm 0 [0] MPI startup(): 158 0 rdmap150s0-rdm 0 [0] MPI startup(): 159 0 rdmap150s0-rdm 0 [0] MPI startup(): 160 0 rdmap150s0-rdm 0 [0] MPI startup(): 161 0 rdmap150s0-rdm 0 [0] MPI startup(): 162 0 rdmap150s0-rdm 0 [0] MPI startup(): 163 0 rdmap150s0-rdm 0 [0] MPI startup(): 164 0 rdmap150s0-rdm 0 [0] MPI startup(): 165 0 rdmap150s0-rdm 0 [0] MPI startup(): 166 0 rdmap150s0-rdm 0 [0] MPI startup(): 167 0 rdmap150s0-rdm 0 [0] MPI startup(): 168 0 rdmap150s0-rdm 0 [0] MPI startup(): 169 0 rdmap150s0-rdm 0 [0] MPI startup(): 170 0 rdmap150s0-rdm 0 [0] MPI startup(): 171 0 rdmap150s0-rdm 0 [0] MPI startup(): 172 0 rdmap150s0-rdm 0 [0] MPI startup(): 173 0 rdmap150s0-rdm 0 [0] MPI startup(): 174 0 rdmap150s0-rdm 0 [0] MPI startup(): 175 0 rdmap150s0-rdm 0 [0] MPI startup(): 176 0 rdmap150s0-rdm 0 [0] MPI startup(): 177 0 rdmap150s0-rdm 0 [0] MPI startup(): 178 0 rdmap150s0-rdm 0 [0] MPI startup(): 179 0 rdmap150s0-rdm 0 [0] MPI startup(): 180 0 rdmap150s0-rdm 0 [0] MPI startup(): 181 0 rdmap150s0-rdm 0 [0] MPI startup(): 182 0 rdmap150s0-rdm 0 [0] MPI startup(): 183 0 rdmap150s0-rdm 0 [0] MPI startup(): 184 0 rdmap150s0-rdm 0 [0] MPI startup(): 185 0 rdmap150s0-rdm 0 [0] MPI startup(): 186 0 rdmap150s0-rdm 0 [0] MPI startup(): 187 0 rdmap150s0-rdm 0 [0] MPI startup(): 188 0 rdmap150s0-rdm 0 [0] MPI startup(): 189 0 rdmap150s0-rdm 0 [0] MPI startup(): 190 0 rdmap150s0-rdm 0 [0] MPI startup(): 191 0 rdmap150s0-rdm 0 [0] MPI startup(): ===== CPU pinning ===== [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 15233 efa-od-az-1a-dy-c7i-48xlarge-1 {48} [0] MPI startup(): 1 15234 efa-od-az-1a-dy-c7i-48xlarge-1 {49} [0] MPI startup(): 2 15235 efa-od-az-1a-dy-c7i-48xlarge-1 {50} [0] MPI startup(): 3 15236 efa-od-az-1a-dy-c7i-48xlarge-1 {51} [0] MPI startup(): 4 15237 efa-od-az-1a-dy-c7i-48xlarge-1 {52} [0] MPI startup(): 5 15238 efa-od-az-1a-dy-c7i-48xlarge-1 {53} [0] MPI startup(): 6 15239 efa-od-az-1a-dy-c7i-48xlarge-1 {54} [0] MPI startup(): 7 15240 efa-od-az-1a-dy-c7i-48xlarge-1 {55} [0] MPI startup(): 8 15241 efa-od-az-1a-dy-c7i-48xlarge-1 {56} [0] MPI startup(): 9 15242 efa-od-az-1a-dy-c7i-48xlarge-1 {57} [0] MPI startup(): 10 15243 efa-od-az-1a-dy-c7i-48xlarge-1 {58} [0] MPI startup(): 11 15244 efa-od-az-1a-dy-c7i-48xlarge-1 {59} [0] MPI startup(): 12 15245 efa-od-az-1a-dy-c7i-48xlarge-1 {60} [0] MPI startup(): 13 15250 efa-od-az-1a-dy-c7i-48xlarge-1 {61} [0] MPI startup(): 14 15253 efa-od-az-1a-dy-c7i-48xlarge-1 {62} [0] MPI startup(): 15 15254 efa-od-az-1a-dy-c7i-48xlarge-1 {63} [0] MPI startup(): 16 15258 efa-od-az-1a-dy-c7i-48xlarge-1 {64} [0] MPI startup(): 17 15265 efa-od-az-1a-dy-c7i-48xlarge-1 {65} [0] MPI startup(): 18 15269 efa-od-az-1a-dy-c7i-48xlarge-1 {66} [0] MPI startup(): 19 15270 efa-od-az-1a-dy-c7i-48xlarge-1 {67} [0] MPI startup(): 20 15273 efa-od-az-1a-dy-c7i-48xlarge-1 {68} [0] MPI startup(): 21 15276 efa-od-az-1a-dy-c7i-48xlarge-1 {69} [0] MPI startup(): 22 15279 efa-od-az-1a-dy-c7i-48xlarge-1 {70} [0] MPI startup(): 23 15285 efa-od-az-1a-dy-c7i-48xlarge-1 {71} [0] MPI startup(): 24 15287 efa-od-az-1a-dy-c7i-48xlarge-1 {72} [0] MPI startup(): 25 15293 efa-od-az-1a-dy-c7i-48xlarge-1 {73} [0] MPI startup(): 26 15294 efa-od-az-1a-dy-c7i-48xlarge-1 {74} [0] MPI startup(): 27 15297 efa-od-az-1a-dy-c7i-48xlarge-1 {75} [0] MPI startup(): 28 15298 efa-od-az-1a-dy-c7i-48xlarge-1 {76} [0] MPI startup(): 29 15299 efa-od-az-1a-dy-c7i-48xlarge-1 {77} [0] MPI startup(): 30 15303 efa-od-az-1a-dy-c7i-48xlarge-1 {78} [0] MPI startup(): 31 15304 efa-od-az-1a-dy-c7i-48xlarge-1 {79} [0] MPI startup(): 32 15307 efa-od-az-1a-dy-c7i-48xlarge-1 {80} [0] MPI startup(): 33 15311 efa-od-az-1a-dy-c7i-48xlarge-1 {81} [0] MPI startup(): 34 15312 efa-od-az-1a-dy-c7i-48xlarge-1 {82} [0] MPI startup(): 35 15314 efa-od-az-1a-dy-c7i-48xlarge-1 {83} [0] MPI startup(): 36 15315 efa-od-az-1a-dy-c7i-48xlarge-1 {84} [0] MPI startup(): 37 15316 efa-od-az-1a-dy-c7i-48xlarge-1 {85} [0] MPI startup(): 38 15321 efa-od-az-1a-dy-c7i-48xlarge-1 {86} [0] MPI startup(): 39 15325 efa-od-az-1a-dy-c7i-48xlarge-1 {87} [0] MPI startup(): 40 15326 efa-od-az-1a-dy-c7i-48xlarge-1 {88} [0] MPI startup(): 41 15328 efa-od-az-1a-dy-c7i-48xlarge-1 {89} [0] MPI startup(): 42 15329 efa-od-az-1a-dy-c7i-48xlarge-1 {90} [0] MPI startup(): 43 15330 efa-od-az-1a-dy-c7i-48xlarge-1 {91} [0] MPI startup(): 44 15336 efa-od-az-1a-dy-c7i-48xlarge-1 {92} [0] MPI startup(): 45 15337 efa-od-az-1a-dy-c7i-48xlarge-1 {93} [0] MPI startup(): 46 15340 efa-od-az-1a-dy-c7i-48xlarge-1 {94} [0] MPI startup(): 47 15344 efa-od-az-1a-dy-c7i-48xlarge-1 {95} [0] MPI startup(): 48 15348 efa-od-az-1a-dy-c7i-48xlarge-1 {0} [0] MPI startup(): 49 15349 efa-od-az-1a-dy-c7i-48xlarge-1 {1} [0] MPI startup(): 50 15352 efa-od-az-1a-dy-c7i-48xlarge-1 {2} [0] MPI startup(): 51 15356 efa-od-az-1a-dy-c7i-48xlarge-1 {3} [0] MPI startup(): 52 15362 efa-od-az-1a-dy-c7i-48xlarge-1 {4} [0] MPI startup(): 53 15364 efa-od-az-1a-dy-c7i-48xlarge-1 {5} [0] MPI startup(): 54 15367 efa-od-az-1a-dy-c7i-48xlarge-1 {6} [0] MPI startup(): 55 15372 efa-od-az-1a-dy-c7i-48xlarge-1 {7} [0] MPI startup(): 56 15373 efa-od-az-1a-dy-c7i-48xlarge-1 {8} [0] MPI startup(): 57 15377 efa-od-az-1a-dy-c7i-48xlarge-1 {9} [0] MPI startup(): 58 15378 efa-od-az-1a-dy-c7i-48xlarge-1 {10} [0] MPI startup(): 59 15381 efa-od-az-1a-dy-c7i-48xlarge-1 {11} [0] MPI startup(): 60 15386 efa-od-az-1a-dy-c7i-48xlarge-1 {12} [0] MPI startup(): 61 15387 efa-od-az-1a-dy-c7i-48xlarge-1 {13} [0] MPI startup(): 62 15389 efa-od-az-1a-dy-c7i-48xlarge-1 {14} [0] MPI startup(): 63 15393 efa-od-az-1a-dy-c7i-48xlarge-1 {15} [0] MPI startup(): 64 15395 efa-od-az-1a-dy-c7i-48xlarge-1 {16} [0] MPI startup(): 65 15396 efa-od-az-1a-dy-c7i-48xlarge-1 {17} [0] MPI startup(): 66 15402 efa-od-az-1a-dy-c7i-48xlarge-1 {18} [0] MPI startup(): 67 15403 efa-od-az-1a-dy-c7i-48xlarge-1 {19} [0] MPI startup(): 68 15404 efa-od-az-1a-dy-c7i-48xlarge-1 {20} [0] MPI startup(): 69 15405 efa-od-az-1a-dy-c7i-48xlarge-1 {21} [0] MPI startup(): 70 15406 efa-od-az-1a-dy-c7i-48xlarge-1 {22} [0] MPI startup(): 71 15410 efa-od-az-1a-dy-c7i-48xlarge-1 {23} [0] MPI startup(): 72 15413 efa-od-az-1a-dy-c7i-48xlarge-1 {24} [0] MPI startup(): 73 15414 efa-od-az-1a-dy-c7i-48xlarge-1 {25} [0] MPI startup(): 74 15417 efa-od-az-1a-dy-c7i-48xlarge-1 {26} [0] MPI startup(): 75 15418 efa-od-az-1a-dy-c7i-48xlarge-1 {27} [0] MPI startup(): 76 15420 efa-od-az-1a-dy-c7i-48xlarge-1 {28} [0] MPI startup(): 77 15421 efa-od-az-1a-dy-c7i-48xlarge-1 {29} [0] MPI startup(): 78 15422 efa-od-az-1a-dy-c7i-48xlarge-1 {30} [0] MPI startup(): 79 15423 efa-od-az-1a-dy-c7i-48xlarge-1 {31} [0] MPI startup(): 80 15424 efa-od-az-1a-dy-c7i-48xlarge-1 {32} [0] MPI startup(): 81 15425 efa-od-az-1a-dy-c7i-48xlarge-1 {33} [0] MPI startup(): 82 15426 efa-od-az-1a-dy-c7i-48xlarge-1 {34} [0] MPI startup(): 83 15427 efa-od-az-1a-dy-c7i-48xlarge-1 {35} [0] MPI startup(): 84 15428 efa-od-az-1a-dy-c7i-48xlarge-1 {36} [0] MPI startup(): 85 15429 efa-od-az-1a-dy-c7i-48xlarge-1 {37} [0] MPI startup(): 86 15430 efa-od-az-1a-dy-c7i-48xlarge-1 {38} [0] MPI startup(): 87 15431 efa-od-az-1a-dy-c7i-48xlarge-1 {39} [0] MPI startup(): 88 15432 efa-od-az-1a-dy-c7i-48xlarge-1 {40} [0] MPI startup(): 89 15433 efa-od-az-1a-dy-c7i-48xlarge-1 {41} [0] MPI startup(): 90 15434 efa-od-az-1a-dy-c7i-48xlarge-1 {42} [0] MPI startup(): 91 15435 efa-od-az-1a-dy-c7i-48xlarge-1 {43} [0] MPI startup(): 92 15436 efa-od-az-1a-dy-c7i-48xlarge-1 {44} [0] MPI startup(): 93 15437 efa-od-az-1a-dy-c7i-48xlarge-1 {45} [0] MPI startup(): 94 15438 efa-od-az-1a-dy-c7i-48xlarge-1 {46} [0] MPI startup(): 95 15439 efa-od-az-1a-dy-c7i-48xlarge-1 {47} [0] MPI startup(): 96 15440 efa-od-az-1a-dy-c7i-48xlarge-1 {48} [0] MPI startup(): 97 15441 efa-od-az-1a-dy-c7i-48xlarge-1 {49} [0] MPI startup(): 98 15442 efa-od-az-1a-dy-c7i-48xlarge-1 {50} [0] MPI startup(): 99 15443 efa-od-az-1a-dy-c7i-48xlarge-1 {51} [0] MPI startup(): 100 15444 efa-od-az-1a-dy-c7i-48xlarge-1 {52} [0] MPI startup(): 101 15445 efa-od-az-1a-dy-c7i-48xlarge-1 {53} [0] MPI startup(): 102 15446 efa-od-az-1a-dy-c7i-48xlarge-1 {54} [0] MPI startup(): 103 15447 efa-od-az-1a-dy-c7i-48xlarge-1 {55} [0] MPI startup(): 104 15448 efa-od-az-1a-dy-c7i-48xlarge-1 {56} [0] MPI startup(): 105 15450 efa-od-az-1a-dy-c7i-48xlarge-1 {57} [0] MPI startup(): 106 15451 efa-od-az-1a-dy-c7i-48xlarge-1 {58} [0] MPI startup(): 107 15452 efa-od-az-1a-dy-c7i-48xlarge-1 {59} [0] MPI startup(): 108 15453 efa-od-az-1a-dy-c7i-48xlarge-1 {60} [0] MPI startup(): 109 15455 efa-od-az-1a-dy-c7i-48xlarge-1 {61} [0] MPI startup(): 110 15456 efa-od-az-1a-dy-c7i-48xlarge-1 {62} [0] MPI startup(): 111 15457 efa-od-az-1a-dy-c7i-48xlarge-1 {63} [0] MPI startup(): 112 15458 efa-od-az-1a-dy-c7i-48xlarge-1 {64} [0] MPI startup(): 113 15459 efa-od-az-1a-dy-c7i-48xlarge-1 {65} [0] MPI startup(): 114 15460 efa-od-az-1a-dy-c7i-48xlarge-1 {66} [0] MPI startup(): 115 15461 efa-od-az-1a-dy-c7i-48xlarge-1 {67} [0] MPI startup(): 116 15462 efa-od-az-1a-dy-c7i-48xlarge-1 {68} [0] MPI startup(): 117 15463 efa-od-az-1a-dy-c7i-48xlarge-1 {69} [0] MPI startup(): 118 15464 efa-od-az-1a-dy-c7i-48xlarge-1 {70} [0] MPI startup(): 119 15465 efa-od-az-1a-dy-c7i-48xlarge-1 {71} [0] MPI startup(): 120 15466 efa-od-az-1a-dy-c7i-48xlarge-1 {72} [0] MPI startup(): 121 15467 efa-od-az-1a-dy-c7i-48xlarge-1 {73} [0] MPI startup(): 122 15468 efa-od-az-1a-dy-c7i-48xlarge-1 {74} [0] MPI startup(): 123 15469 efa-od-az-1a-dy-c7i-48xlarge-1 {75} [0] MPI startup(): 124 15470 efa-od-az-1a-dy-c7i-48xlarge-1 {76} [0] MPI startup(): 125 15471 efa-od-az-1a-dy-c7i-48xlarge-1 {77} [0] MPI startup(): 126 15472 efa-od-az-1a-dy-c7i-48xlarge-1 {78} [0] MPI startup(): 127 15473 efa-od-az-1a-dy-c7i-48xlarge-1 {79} [0] MPI startup(): 128 15474 efa-od-az-1a-dy-c7i-48xlarge-1 {80} [0] MPI startup(): 129 15475 efa-od-az-1a-dy-c7i-48xlarge-1 {81} [0] MPI startup(): 130 15478 efa-od-az-1a-dy-c7i-48xlarge-1 {82} [0] MPI startup(): 131 15479 efa-od-az-1a-dy-c7i-48xlarge-1 {83} [0] MPI startup(): 132 15484 efa-od-az-1a-dy-c7i-48xlarge-1 {84} [0] MPI startup(): 133 15487 efa-od-az-1a-dy-c7i-48xlarge-1 {85} [0] MPI startup(): 134 15490 efa-od-az-1a-dy-c7i-48xlarge-1 {86} [0] MPI startup(): 135 15495 efa-od-az-1a-dy-c7i-48xlarge-1 {87} [0] MPI startup(): 136 15497 efa-od-az-1a-dy-c7i-48xlarge-1 {88} [0] MPI startup(): 137 15501 efa-od-az-1a-dy-c7i-48xlarge-1 {89} [0] MPI startup(): 138 15509 efa-od-az-1a-dy-c7i-48xlarge-1 {90} [0] MPI startup(): 139 15515 efa-od-az-1a-dy-c7i-48xlarge-1 {91} [0] MPI startup(): 140 15516 efa-od-az-1a-dy-c7i-48xlarge-1 {92} [0] MPI startup(): 141 15521 efa-od-az-1a-dy-c7i-48xlarge-1 {93} [0] MPI startup(): 142 15529 efa-od-az-1a-dy-c7i-48xlarge-1 {94} [0] MPI startup(): 143 15535 efa-od-az-1a-dy-c7i-48xlarge-1 {95} [0] MPI startup(): 144 15547 efa-od-az-1a-dy-c7i-48xlarge-1 {0} [0] MPI startup(): 145 15548 efa-od-az-1a-dy-c7i-48xlarge-1 {1} [0] MPI startup(): 146 15549 efa-od-az-1a-dy-c7i-48xlarge-1 {2} [0] MPI startup(): 147 15550 efa-od-az-1a-dy-c7i-48xlarge-1 {3} [0] MPI startup(): 148 15551 efa-od-az-1a-dy-c7i-48xlarge-1 {4} [0] MPI startup(): 149 15552 efa-od-az-1a-dy-c7i-48xlarge-1 {5} [0] MPI startup(): 150 15553 efa-od-az-1a-dy-c7i-48xlarge-1 {6} [0] MPI startup(): 151 15554 efa-od-az-1a-dy-c7i-48xlarge-1 {7} [0] MPI startup(): 152 15555 efa-od-az-1a-dy-c7i-48xlarge-1 {8} [0] MPI startup(): 153 15556 efa-od-az-1a-dy-c7i-48xlarge-1 {9} [0] MPI startup(): 154 15557 efa-od-az-1a-dy-c7i-48xlarge-1 {10} [0] MPI startup(): 155 15558 efa-od-az-1a-dy-c7i-48xlarge-1 {11} [0] MPI startup(): 156 15559 efa-od-az-1a-dy-c7i-48xlarge-1 {12} [0] MPI startup(): 157 15560 efa-od-az-1a-dy-c7i-48xlarge-1 {13} [0] MPI startup(): 158 15561 efa-od-az-1a-dy-c7i-48xlarge-1 {14} [0] MPI startup(): 159 15562 efa-od-az-1a-dy-c7i-48xlarge-1 {15} [0] MPI startup(): 160 15563 efa-od-az-1a-dy-c7i-48xlarge-1 {16} [0] MPI startup(): 161 15564 efa-od-az-1a-dy-c7i-48xlarge-1 {17} [0] MPI startup(): 162 15565 efa-od-az-1a-dy-c7i-48xlarge-1 {18} [0] MPI startup(): 163 15566 efa-od-az-1a-dy-c7i-48xlarge-1 {19} [0] MPI startup(): 164 15567 efa-od-az-1a-dy-c7i-48xlarge-1 {20} [0] MPI startup(): 165 15568 efa-od-az-1a-dy-c7i-48xlarge-1 {21} [0] MPI startup(): 166 15569 efa-od-az-1a-dy-c7i-48xlarge-1 {22} [0] MPI startup(): 167 15570 efa-od-az-1a-dy-c7i-48xlarge-1 {23} [0] MPI startup(): 168 15572 efa-od-az-1a-dy-c7i-48xlarge-1 {24} [0] MPI startup(): 169 15573 efa-od-az-1a-dy-c7i-48xlarge-1 {25} [0] MPI startup(): 170 15574 efa-od-az-1a-dy-c7i-48xlarge-1 {26} [0] MPI startup(): 171 15575 efa-od-az-1a-dy-c7i-48xlarge-1 {27} [0] MPI startup(): 172 15576 efa-od-az-1a-dy-c7i-48xlarge-1 {28} [0] MPI startup(): 173 15577 efa-od-az-1a-dy-c7i-48xlarge-1 {29} [0] MPI startup(): 174 15578 efa-od-az-1a-dy-c7i-48xlarge-1 {30} [0] MPI startup(): 175 15579 efa-od-az-1a-dy-c7i-48xlarge-1 {31} [0] MPI startup(): 176 15580 efa-od-az-1a-dy-c7i-48xlarge-1 {32} [0] MPI startup(): 177 15581 efa-od-az-1a-dy-c7i-48xlarge-1 {33} [0] MPI startup(): 178 15582 efa-od-az-1a-dy-c7i-48xlarge-1 {34} [0] MPI startup(): 179 15583 efa-od-az-1a-dy-c7i-48xlarge-1 {35} [0] MPI startup(): 180 15584 efa-od-az-1a-dy-c7i-48xlarge-1 {36} [0] MPI startup(): 181 15585 efa-od-az-1a-dy-c7i-48xlarge-1 {37} [0] MPI startup(): 182 15586 efa-od-az-1a-dy-c7i-48xlarge-1 {38} [0] MPI startup(): 183 15587 efa-od-az-1a-dy-c7i-48xlarge-1 {39} [0] MPI startup(): 184 15588 efa-od-az-1a-dy-c7i-48xlarge-1 {40} [0] MPI startup(): 185 15589 efa-od-az-1a-dy-c7i-48xlarge-1 {41} [0] MPI startup(): 186 15590 efa-od-az-1a-dy-c7i-48xlarge-1 {42} [0] MPI startup(): 187 15591 efa-od-az-1a-dy-c7i-48xlarge-1 {43} [0] MPI startup(): 188 15592 efa-od-az-1a-dy-c7i-48xlarge-1 {44} [0] MPI startup(): 189 15593 efa-od-az-1a-dy-c7i-48xlarge-1 {45} [0] MPI startup(): 190 15594 efa-od-az-1a-dy-c7i-48xlarge-1 {46} [0] MPI startup(): 191 15595 efa-od-az-1a-dy-c7i-48xlarge-1 {47} [0] MPI startup(): I\_MPI\_OFI\_LIBRARY\_INTERNAL=1 [0] MPI startup(): I\_MPI\_ROOT=/opt/intel/oneapi/mpi/2021.14 [0] MPI startup(): I\_MPI\_MPIRUN=mpirun [0] MPI startup(): I\_MPI\_BIND\_WIN\_ALLOCATE=localalloc [0] MPI startup(): I\_MPI\_HYDRA\_BOOTSTRAP\_EXEC\_EXTRA\_ARGS=--external-launcher [0] MPI startup(): I\_MPI\_HYDRA\_TOPOLIB=hwloc [0] MPI startup(): I\_MPI\_HYDRA\_BOOTSTRAP=slurm [0] MPI startup(): I\_MPI\_RETURN\_WIN\_MEM\_NUMA=1 [0] MPI startup(): I\_MPI\_INTERNAL\_MEM\_POLICY=default [0] MPI startup(): I\_MPI\_FABRICS=shm:ofi [0] MPI startup(): I\_MPI\_OFI\_PROVIDER=efa [0] MPI startup(): I\_MPI\_DEBUG=10 #---------------------------------------------------------------- # Intel(R) MPI Benchmarks 2021.8, MPI-1 part #---------------------------------------------------------------- # Date : Mon Feb 17 11:30:57 2025 # Machine : x86\_64 # System : Linux # Release : 5.14.0-427.26.1.el9\_4.x86\_64 # Version : #1 SMP PREEMPT\_DYNAMIC Fri Jul 5 11:34:54 EDT 2024 # MPI Version : 3.1 # MPI Thread Environment: # Calling sequence was: # IMB-MPI1 barrier # Minimum message length in bytes: 0 # Maximum message length in bytes: 4194304 # # MPI\_Datatype : MPI\_BYTE # MPI\_Datatype for reductions : MPI\_FLOAT # MPI\_Op : MPI\_SUM # # # List of Benchmarks to run: # Barrier #--------------------------------------------------- # Benchmarking Barrier # #processes = 2 # ( 190 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 1000 0.37 0.37 0.37 #--------------------------------------------------- # Benchmarking Barrier # #processes = 4 # ( 188 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 1000 0.60 0.60 0.60 #--------------------------------------------------- # Benchmarking Barrier # #processes = 8 # ( 184 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 376 0.68 0.68 0.68 #--------------------------------------------------- # Benchmarking Barrier # #processes = 16 # ( 176 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 283 25936.42 26010.59 25971.98 #--------------------------------------------------- # Benchmarking Barrier # #processes = 32 # ( 160 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 194 31180.34 31360.85 31291.58 #--------------------------------------------------- # Benchmarking Barrier # #processes = 64 # ( 128 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 196 43785.67 43984.83 43892.37 #--------------------------------------------------- # Benchmarking Barrier # #processes = 128 # ( 64 additional processes waiting in MPI\_Barrier) #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 140 57157.02 57571.38 57406.40 #--------------------------------------------------- # Benchmarking Barrier # #processes = 192 #--------------------------------------------------- #repetitions t\_min[usec] t\_max[usec] t\_avg[usec] 109 52944.94 53403.62 53220.60 # All processes entering MPI\_Finalize ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 17, 2025, 12:11pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/7 "2025-02-17T12:11:59Z")
@TobiasK   If you see the pinning information, 192 mpi ranks has been assigned to cpu 0-95 (two ranks per cpu). My expectation is that 192 mpi ranks should be assigned to cpu 0-191. Please find the output of ‘mpirun IMB-MPI1 barrier’ in the attachment. Thanks! [IMB-MPI1.txt](https://community.intel.com/uploads/short-url/A4TiyyLYw2n16QHfoTrFKviUd6Z.txt) (25.4 KB) ---
**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [February 17, 2025, 5:32pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/8 "2025-02-17T17:32:54Z")
Thank you for the output. I see slurm is used, how is slurm configured? Can you provide all Slurm environment variables, e.g. ```markup export | grep SLURM ``` do you see the same pinning issue if you use srun instead of mpirun? ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 17, 2025, 6:46pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/9 "2025-02-17T18:46:10Z")
@TobiasK  Thanks! ![mgHarish_0-1739815914325.png](https://us1.discourse-cdn.com/intel/original/3X/a/1/a110f1561954932e92686d046e2dfc917de04199.png) Yes, I see the same behavior when i use srun to stress all the 192 cpus, but from the htop i see only 50 percent of total cpu’s (96 cpu’s) went to 100 percent utilization. Is this due to slurm configuration issue? ![mgHarish_1-1739817841377.png](https://us1.discourse-cdn.com/intel/original/3X/b/c/bc5a458bd0e5001622dfd2b856c3e594528bf3da.png) command used: srun -n 192 -p efa-od-az-1a -w efa-od-az-1a-dy-c7i-48xlarge-1 --pty /usr/bin/stress --cpu 1 --timeout 10 ---
**Author:** ![TobiasK](https://avatars.discourse-cdn.com/v4/letter/t/00C7FD/32.png) [@TobiasK](https://community.intel.com/u/TobiasK)\ **Post date:** [February 17, 2025, 8:09pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/10 "2025-02-17T20:09:09Z")
@mgHarish  most likely yes. Can you try with: ```bash I_MPI_PIN_RESPECT_CPUSET=0 I_MPI_DEBUG=10 mpirun -np 192 IMB-MPI1 -npmin 192 barrier ``` What does ```markup scontrol show node efa-od-az-1a-dy-c7i-48xlarge-1 ``` show? ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 18, 2025, 11:05am UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/11 "2025-02-18T11:05:21Z")
@TobiasK  I get the same behavior with the command ```markup I_MPI_PIN_RESPECT_CPUSET=0 I_MPI_DEBUG=10 mpirun -np 192 IMB-MPI1 -npmin 192 barrier ``` But when I tried ssh into the node and ran the same command, I get the desired output (both will be in the attachments). Please find the output of _scontrol show node command_ while the below command is running ```markup srun -n 192 -p efa-od-az-1a -w efa-od-az-1a-dy-c7i-48xlarge-1 --pty /usr/bin/stress --cpu 1 --timeout 300 ``` ![mgHarish_1-1739876251359.png](https://us1.discourse-cdn.com/intel/original/3X/c/2/c23af98803b95e364a6d1d7568c7cc8e78be7d3d.png) Thanks [IMB-MPI1-ssh.txt](https://community.intel.com/uploads/short-url/32o5yJiR73U9oBUxnfGaUvSRRVz.txt) (28 KB) [IMB-MPI1-srun.txt](https://community.intel.com/uploads/short-url/AfrvQNS8nibEExlA3UpSkWHauzM.txt) (28.1 KB) ---
**Author:** ![mgHarish](https://avatars.discourse-cdn.com/v4/letter/m/0046C8/32.png) [@mgHarish](https://community.intel.com/u/mgHarish)\ **Post date:** [February 26, 2025, 12:46pm UTC](https://community.intel.com/t/intelmpi-cpuinfo-showing-one-thread-per-core/130221/12 "2025-02-26T12:46:25Z")
@TobiasK   I found the cause of the issue. By, default slurm was configured with 1 thread per core, when I configured with 2 threads per core as suggested in [(3.9.0‐3.9.1) Default ThreadsPerCore Slurm setting causes reduced CPU utilization · aws/aws-parallelcluster Wiki · GitHub](https://github.com/aws/aws-parallelcluster/wiki/(3.9.0%E2%80%903.9.1)-Default-ThreadsPerCore-Slurm-setting-causes-reduced-CPU-utilization), Intel MPI also shows threads per core equals 2. Help me to understand why the change in slurm configuration has resolved my issue. Exactly how slurm and intelmpi is linked?