machinefile run [0] MPI startup(): Intel(R) MPI Library, Version 2019 Update 8 Build 20200624 (id: 4f16ad915) [0] MPI startup(): Copyright (C) 2003-2020 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric version: 1.10.1-impi libfabric:6800:core:core:ofi_reg_dl_prov():578 dlopen(/opt/intel/compilers_and_libraries_2020.2.254/linux/mpi/intel64/libfabric/lib/prov/libpsmx2-fi.so): libpsm2.so.2: cannot open shared object file: No such file or directory libfabric:6800:core:core:ofi_register_provider():418 registering provider: sockets (110.10) libfabric:6800:core:core:ofi_reg_dl_prov():578 dlopen(/opt/intel/compilers_and_libraries_2020.2.254/linux/mpi/intel64/libfabric/lib/prov/libmlx-fi.so): libucp.so.0: cannot open shared object file: No such file or directory libfabric:6800:core:core:ofi_register_provider():418 registering provider: efa (2.0) libfabric:6800:core:core:ofi_register_provider():418 registering provider: ofi_rxm (110.10) libfabric:6800:core:core:ofi_register_provider():418 registering provider: tcp (110.10) libfabric:6800:core:core:ofi_register_provider():418 registering provider: shm (110.10) libfabric:6800:core:core:ofi_register_provider():418 registering provider: ofi_hook_noop (110.10) libfabric:6800:core:core:fi_getinfo_():1066 Found provider with the highest priority efa, must_use_util_prov = 0 libfabric:6800:efa:core:efa_check_hints():196 Required hints mode bits not set libfabric:6800:efa:core:efa_check_hints():197 Expected: FI_MSG_PREFIX libfabric:6800:efa:core:efa_check_hints():197 Given: libfabric:6800:core:core:fi_getinfo_():1066 Found provider with the highest priority shm, must_use_util_prov = 0 libfabric:6800:core:core:fi_getinfo_():1093 Since shm can be used, efa has been skipped. To use efa, please, set FI_PROVIDER=efa libfabric:6800:core:core:fi_getinfo_():1093 Since shm can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:6800:core:core:fi_getinfo_():1093 Since shm can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:6800:core:core:fi_getinfo_():1093 Since efa can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:6800:core:core:fi_getinfo_():1093 Since efa can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:6800:core:core:fi_getinfo_():1093 Since efa can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:6800:core:core:fi_getinfo_():1066 Found provider with the highest priority efa, must_use_util_prov = 0 libfabric:6800:efa:core:efa_check_hints():196 Required hints mode bits not set libfabric:6800:efa:core:efa_check_hints():197 Expected: FI_MSG_PREFIX, FI_LOCAL_MR libfabric:6800:efa:core:efa_check_hints():197 Given: FI_LOCAL_MR libfabric:6800:efa:core:efa_check_hints():188 Unsupported capabilities libfabric:6800:efa:core:efa_check_hints():189 Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM, FI_SOURCE libfabric:6800:efa:core:efa_check_hints():189 Requested: FI_MSG, FI_RMA, FI_TAGGED, FI_ATOMIC, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_MULTI_RECV, FI_LOCAL_COMM, FI_REMOTE_COMM, FI_SOURCE, FI_DIRECTED_RECV libfabric:6800:efa:core:efa_check_hints():188 Unsupported capabilities libfabric:6800:efa:core:efa_check_hints():189 Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM, FI_SOURCE libfabric:6800:efa:core:efa_check_hints():189 Requested: FI_MSG, FI_RMA, FI_TAGGED, FI_ATOMIC, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_MULTI_RECV, FI_LOCAL_COMM, FI_REMOTE_COMM, FI_SOURCE, FI_DIRECTED_RECV libfabric:6800:core:core:fi_getinfo_():1093 Since efa can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:6800:core:core:fi_getinfo_():1093 Since efa can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets [0] MPI startup(): libfabric provider: efa libfabric:6800:core:core:fi_getinfo_():1093 Since efa can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:6800:core:core:fi_fabric_():1346 Opened fabric: shm libfabric:6800:core:core:fi_fabric_():1346 Opened fabric: EFA-fe80::c36:c9ff:feb6:e3a1 libfabric:6800:efa:core:efa_check_hints():196 Required hints mode bits not set libfabric:6800:efa:core:efa_check_hints():197 Expected: FI_MSG_PREFIX, FI_LOCAL_MR libfabric:6800:efa:core:efa_check_hints():197 Given: FI_LOCAL_MR [0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 1, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1 libfabric:6800:efa:core:efa_check_hints():196 Required hints mode bits not set libfabric:6800:efa:core:efa_check_hints():197 Expected: FI_MSG_PREFIX, FI_LOCAL_MR libfabric:6800:efa:core:efa_check_hints():197 Given: FI_LOCAL_MR libfabric:6800:efa:core:ofi_check_tx_attr():880 Rx only caps ignored in Tx caps libfabric:6800:efa:core:ofi_check_rx_attr():782 Tx only caps ignored in Rx caps libfabric:6800:efa:core:ofi_check_rx_attr():782 Tx only caps ignored in Rx caps libfabric:6800:efa:core:ofi_check_tx_attr():880 Rx only caps ignored in Tx caps libfabric:6800:efa:core:ofi_check_ep_type():654 unsupported endpoint type libfabric:6800:efa:core:ofi_check_ep_type():655 Supported: FI_EP_DGRAM libfabric:6800:efa:core:ofi_check_ep_type():655 Requested: FI_EP_RDM [0] MPI startup(): addrnamelen: 32 [0] MPI startup(): Load tuning file: "/opt/intel/compilers_and_libraries_2020.2.254/linux/mpi/intel64/etc/tuning_skx_shm-ofi_efa.dat" [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 6800 ip-10-0-36-16 {0,36} [0] MPI startup(): 1 6426 ip-10-0-37-134 {0,36} [0] MPI startup(): 2 6460 ip-10-0-40-160 {0,36} [0] MPI startup(): 3 6801 ip-10-0-36-16 {1,37} [0] MPI startup(): 4 6802 ip-10-0-36-16 {2,38} [0] MPI startup(): 5 5728 ip-10-0-40-233 {0,36} [0] MPI startup(): 6 6427 ip-10-0-37-134 {1,37} [0] MPI startup(): 7 6461 ip-10-0-40-160 {1,37} [0] MPI startup(): 8 5729 ip-10-0-40-233 {1,37} [0] MPI startup(): 9 6462 ip-10-0-40-160 {2,38} [0] MPI startup(): 10 6502 ip-10-0-42-146 {0,36} [0] MPI startup(): 11 6564 ip-10-0-44-36 {0,36} [0] MPI startup(): 12 6803 ip-10-0-36-16 {3,39} [0] MPI startup(): 13 6503 ip-10-0-42-146 {1,37} [0] MPI startup(): 14 6439 ip-10-0-44-244 {0,36} [0] MPI startup(): 15 5730 ip-10-0-40-233 {2,38} [0] MPI startup(): 16 5731 ip-10-0-40-233 {3,39} [0] MPI startup(): 17 6445 ip-10-0-47-192 {0,36} [0] MPI startup(): 18 6804 ip-10-0-36-16 {4,40} [0] MPI startup(): 19 6565 ip-10-0-44-36 {1,37} [0] MPI startup(): 20 6428 ip-10-0-37-134 {2,38} [0] MPI startup(): 21 6429 ip-10-0-37-134 {3,39} [0] MPI startup(): 22 6463 ip-10-0-40-160 {3,39} [0] MPI startup(): 23 6430 ip-10-0-37-134 {4,40} [0] MPI startup(): 24 5732 ip-10-0-40-233 {4,40} [0] MPI startup(): 25 6464 ip-10-0-40-160 {4,40} [0] MPI startup(): 26 6566 ip-10-0-44-36 {2,38} [0] MPI startup(): 27 5733 ip-10-0-40-233 {5,41} [0] MPI startup(): 28 6440 ip-10-0-44-244 {1,37} [0] MPI startup(): 29 6504 ip-10-0-42-146 {2,38} [0] MPI startup(): 30 6805 ip-10-0-36-16 {5,41} [0] MPI startup(): 31 6567 ip-10-0-44-36 {3,39} [0] MPI startup(): 32 6441 ip-10-0-44-244 {2,38} [0] MPI startup(): 33 6431 ip-10-0-37-134 {5,41} [0] MPI startup(): 34 6505 ip-10-0-42-146 {3,39} [0] MPI startup(): 35 6465 ip-10-0-40-160 {5,41} [0] MPI startup(): 36 6442 ip-10-0-44-244 {3,39} [0] MPI startup(): 37 6443 ip-10-0-44-244 {4,40} [0] MPI startup(): 38 6466 ip-10-0-40-160 {6,42} [0] MPI startup(): 39 6568 ip-10-0-44-36 {4,40} [0] MPI startup(): 40 6467 ip-10-0-40-160 {7,43} [0] MPI startup(): 41 5734 ip-10-0-40-233 {6,42} [0] MPI startup(): 42 6569 ip-10-0-44-36 {5,41} [0] MPI startup(): 43 6444 ip-10-0-44-244 {5,41} [0] MPI startup(): 44 5735 ip-10-0-40-233 {7,43} [0] MPI startup(): 45 6445 ip-10-0-44-244 {6,42} [0] MPI startup(): 46 6806 ip-10-0-36-16 {6,42} [0] MPI startup(): 47 6446 ip-10-0-44-244 {7,43} [0] MPI startup(): 48 6570 ip-10-0-44-36 {6,42} [0] MPI startup(): 49 6571 ip-10-0-44-36 {7,43} [0] MPI startup(): 50 6446 ip-10-0-47-192 {1,37} [0] MPI startup(): 51 6572 ip-10-0-44-36 {8,44} [0] MPI startup(): 52 6573 ip-10-0-44-36 {9,45} [0] MPI startup(): 53 6506 ip-10-0-42-146 {4,40} [0] MPI startup(): 54 6432 ip-10-0-37-134 {6,42} [0] MPI startup(): 55 6807 ip-10-0-36-16 {7,43} [0] MPI startup(): 56 6433 ip-10-0-37-134 {7,43} [0] MPI startup(): 57 6447 ip-10-0-47-192 {2,38} [0] MPI startup(): 58 6447 ip-10-0-44-244 {8,44} [0] MPI startup(): 59 5736 ip-10-0-40-233 {8,44} [0] MPI startup(): 60 5737 ip-10-0-40-233 {9,45} [0] MPI startup(): 61 6808 ip-10-0-36-16 {8,44} [0] MPI startup(): 62 6448 ip-10-0-47-192 {3,39} [0] MPI startup(): 63 6434 ip-10-0-37-134 {8,44} [0] MPI startup(): 64 6435 ip-10-0-37-134 {9,45} [0] MPI startup(): 65 6448 ip-10-0-44-244 {9,45} [0] MPI startup(): 66 6436 ip-10-0-37-134 {10,46} [0] MPI startup(): 67 6507 ip-10-0-42-146 {5,41} [0] MPI startup(): 68 6449 ip-10-0-47-192 {4,40} [0] MPI startup(): 69 6437 ip-10-0-37-134 {11,47} [0] MPI startup(): 70 5738 ip-10-0-40-233 {10,46} [0] MPI startup(): 71 6449 ip-10-0-44-244 {10,46} [0] MPI startup(): 72 6438 ip-10-0-37-134 {12,48} [0] MPI startup(): 73 6809 ip-10-0-36-16 {9,45} [0] MPI startup(): 74 6439 ip-10-0-37-134 {13,49} [0] MPI startup(): 75 6508 ip-10-0-42-146 {6,42} [0] MPI startup(): 76 6574 ip-10-0-44-36 {10,46} [0] MPI startup(): 77 6575 ip-10-0-44-36 {11,47} [0] MPI startup(): 78 6576 ip-10-0-44-36 {12,48} [0] MPI startup(): 79 6509 ip-10-0-42-146 {7,43} [0] MPI startup(): 80 6450 ip-10-0-47-192 {5,41} [0] MPI startup(): 81 6810 ip-10-0-36-16 {10,46} [0] MPI startup(): 82 6577 ip-10-0-44-36 {13,49} [0] MPI startup(): 83 6468 ip-10-0-40-160 {8,44} [0] MPI startup(): 84 6811 ip-10-0-36-16 {11,47} [0] MPI startup(): 85 6510 ip-10-0-42-146 {8,44} [0] MPI startup(): 86 6440 ip-10-0-37-134 {14,50} [0] MPI startup(): 87 6450 ip-10-0-44-244 {11,47} [0] MPI startup(): 88 6578 ip-10-0-44-36 {14,50} [0] MPI startup(): 89 6441 ip-10-0-37-134 {15,51} [0] MPI startup(): 90 5739 ip-10-0-40-233 {11,47} [0] MPI startup(): 91 6579 ip-10-0-44-36 {15,51} [0] MPI startup(): 92 6469 ip-10-0-40-160 {9,45} [0] MPI startup(): 93 6442 ip-10-0-37-134 {16,52} [0] MPI startup(): 94 6470 ip-10-0-40-160 {10,46} [0] MPI startup(): 95 6451 ip-10-0-47-192 {6,42} [0] MPI startup(): 96 6452 ip-10-0-47-192 {7,43} [0] MPI startup(): 97 6451 ip-10-0-44-244 {12,48} [0] MPI startup(): 98 6511 ip-10-0-42-146 {9,45} [0] MPI startup(): 99 5740 ip-10-0-40-233 {12,48} [0] MPI startup(): 100 6580 ip-10-0-44-36 {16,52} [0] MPI startup(): 101 6512 ip-10-0-42-146 {10,46} [0] MPI startup(): 102 6471 ip-10-0-40-160 {11,47} [0] MPI startup(): 103 6443 ip-10-0-37-134 {17,53} [0] MPI startup(): 104 6444 ip-10-0-37-134 {18,54} [0] MPI startup(): 105 6472 ip-10-0-40-160 {12,48} [0] MPI startup(): 106 6452 ip-10-0-44-244 {13,49} [0] MPI startup(): 107 6453 ip-10-0-47-192 {8,44} [0] MPI startup(): 108 6581 ip-10-0-44-36 {17,53} [0] MPI startup(): 109 5741 ip-10-0-40-233 {13,49} [0] MPI startup(): 110 6445 ip-10-0-37-134 {19,55} [0] MPI startup(): 111 6473 ip-10-0-40-160 {13,49} [0] MPI startup(): 112 6454 ip-10-0-47-192 {9,45} [0] MPI startup(): 113 6453 ip-10-0-44-244 {14,50} [0] MPI startup(): 114 6513 ip-10-0-42-146 {11,47} [0] MPI startup(): 115 6474 ip-10-0-40-160 {14,50} [0] MPI startup(): 116 6446 ip-10-0-37-134 {20,56} [0] MPI startup(): 117 6475 ip-10-0-40-160 {15,51} [0] MPI startup(): 118 6476 ip-10-0-40-160 {16,52} [0] MPI startup(): 119 5742 ip-10-0-40-233 {14,50} [0] MPI startup(): 120 6812 ip-10-0-36-16 {12,48} [0] MPI startup(): 121 6447 ip-10-0-37-134 {21,57} [0] MPI startup(): 122 5743 ip-10-0-40-233 {15,51} [0] MPI startup(): 123 6582 ip-10-0-44-36 {18,54} [0] MPI startup(): 124 6454 ip-10-0-44-244 {15,51} [0] MPI startup(): 125 6455 ip-10-0-44-244 {16,52} [0] MPI startup(): 126 6456 ip-10-0-44-244 {17,53} [0] MPI startup(): 127 6514 ip-10-0-42-146 {12,48} [0] MPI startup(): 128 6448 ip-10-0-37-134 {22,58} [0] MPI startup(): 129 5744 ip-10-0-40-233 {16,52} [0] MPI startup(): 130 6455 ip-10-0-47-192 {10,46} [0] MPI startup(): 131 6813 ip-10-0-36-16 {13,49} [0] MPI startup(): 132 5745 ip-10-0-40-233 {17,53} [0] MPI startup(): 133 6583 ip-10-0-44-36 {19,55} [0] MPI startup(): 134 6515 ip-10-0-42-146 {13,49} [0] MPI startup(): 135 6814 ip-10-0-36-16 {14,50} [0] MPI startup(): 136 6456 ip-10-0-47-192 {11,47} [0] MPI startup(): 137 6815 ip-10-0-36-16 {15,51} [0] MPI startup(): 138 6449 ip-10-0-37-134 {23,59} [0] MPI startup(): 139 6516 ip-10-0-42-146 {14,50} [0] MPI startup(): 140 6457 ip-10-0-47-192 {12,48} [0] MPI startup(): 141 6816 ip-10-0-36-16 {16,52} [0] MPI startup(): 142 6457 ip-10-0-44-244 {18,54} [0] MPI startup(): 143 6458 ip-10-0-47-192 {13,49} [0] MPI startup(): 144 6517 ip-10-0-42-146 {15,51} [0] MPI startup(): 145 6518 ip-10-0-42-146 {16,52} [0] MPI startup(): 146 6477 ip-10-0-40-160 {17,53} [0] MPI startup(): 147 6817 ip-10-0-36-16 {17,53} [0] MPI startup(): 148 6459 ip-10-0-47-192 {14,50} [0] MPI startup(): 149 6460 ip-10-0-47-192 {15,51} [0] MPI startup(): 150 6458 ip-10-0-44-244 {19,55} [0] MPI startup(): 151 6818 ip-10-0-36-16 {18,54} [0] MPI startup(): 152 6478 ip-10-0-40-160 {18,54} [0] MPI startup(): 153 6819 ip-10-0-36-16 {19,55} [0] MPI startup(): 154 6461 ip-10-0-47-192 {16,52} [0] MPI startup(): 155 6450 ip-10-0-37-134 {24,60} [0] MPI startup(): 156 6462 ip-10-0-47-192 {17,53} [0] MPI startup(): 157 6820 ip-10-0-36-16 {20,56} [0] MPI startup(): 158 6584 ip-10-0-44-36 {20,56} [0] MPI startup(): 159 6585 ip-10-0-44-36 {21,57} [0] MPI startup(): 160 6451 ip-10-0-37-134 {25,61} [0] MPI startup(): 161 6821 ip-10-0-36-16 {21,57} [0] MPI startup(): 162 6452 ip-10-0-37-134 {26,62} [0] MPI startup(): 163 6586 ip-10-0-44-36 {22,58} [0] MPI startup(): 164 6479 ip-10-0-40-160 {19,55} [0] MPI startup(): 165 6463 ip-10-0-47-192 {18,54} [0] MPI startup(): 166 6822 ip-10-0-36-16 {22,58} [0] MPI startup(): 167 6519 ip-10-0-42-146 {17,53} [0] MPI startup(): 168 6459 ip-10-0-44-244 {20,56} [0] MPI startup(): 169 6480 ip-10-0-40-160 {20,56} [0] MPI startup(): 170 6587 ip-10-0-44-36 {23,59} [0] MPI startup(): 171 6460 ip-10-0-44-244 {21,57} [0] MPI startup(): 172 6481 ip-10-0-40-160 {21,57} [0] MPI startup(): 173 6588 ip-10-0-44-36 {24,60} [0] MPI startup(): 174 5746 ip-10-0-40-233 {18,54} [0] MPI startup(): 175 6520 ip-10-0-42-146 {18,54} [0] MPI startup(): 176 6482 ip-10-0-40-160 {22,58} [0] MPI startup(): 177 5747 ip-10-0-40-233 {19,55} [0] MPI startup(): 178 6483 ip-10-0-40-160 {23,59} [0] MPI startup(): 179 6453 ip-10-0-37-134 {27,63} [0] MPI startup(): 180 6484 ip-10-0-40-160 {24,60} [0] MPI startup(): 181 6521 ip-10-0-42-146 {19,55} [0] MPI startup(): 182 5748 ip-10-0-40-233 {20,56} [0] MPI startup(): 183 6464 ip-10-0-47-192 {19,55} [0] MPI startup(): 184 6522 ip-10-0-42-146 {20,56} [0] MPI startup(): 185 6461 ip-10-0-44-244 {22,58} [0] MPI startup(): 186 6823 ip-10-0-36-16 {23,59} [0] MPI startup(): 187 5749 ip-10-0-40-233 {21,57} [0] MPI startup(): 188 6523 ip-10-0-42-146 {21,57} [0] MPI startup(): 189 5750 ip-10-0-40-233 {22,58} [0] MPI startup(): 190 6462 ip-10-0-44-244 {23,59} [0] MPI startup(): 191 6485 ip-10-0-40-160 {25,61} [0] MPI startup(): 192 6463 ip-10-0-44-244 {24,60} [0] MPI startup(): 193 6454 ip-10-0-37-134 {28,64} [0] MPI startup(): 194 6524 ip-10-0-42-146 {22,58} [0] MPI startup(): 195 6486 ip-10-0-40-160 {26,62} [0] MPI startup(): 196 6464 ip-10-0-44-244 {25,61} [0] MPI startup(): 197 6465 ip-10-0-47-192 {20,56} [0] MPI startup(): 198 6465 ip-10-0-44-244 {26,62} [0] MPI startup(): 199 5751 ip-10-0-40-233 {23,59} [0] MPI startup(): 200 5752 ip-10-0-40-233 {24,60} [0] MPI startup(): 201 6466 ip-10-0-47-192 {21,57} [0] MPI startup(): 202 6467 ip-10-0-47-192 {22,58} [0] MPI startup(): 203 6525 ip-10-0-42-146 {23,59} [0] MPI startup(): 204 6466 ip-10-0-44-244 {27,63} [0] MPI startup(): 205 6589 ip-10-0-44-36 {25,61} [0] MPI startup(): 206 6487 ip-10-0-40-160 {27,63} [0] MPI startup(): 207 6468 ip-10-0-47-192 {23,59} [0] MPI startup(): 208 6590 ip-10-0-44-36 {26,62} [0] MPI startup(): 209 6526 ip-10-0-42-146 {24,60} [0] MPI startup(): 210 5753 ip-10-0-40-233 {25,61} [0] MPI startup(): 211 6469 ip-10-0-47-192 {24,60} [0] MPI startup(): 212 6470 ip-10-0-47-192 {25,61} [0] MPI startup(): 213 6824 ip-10-0-36-16 {24,60} [0] MPI startup(): 214 6488 ip-10-0-40-160 {28,64} [0] MPI startup(): 215 5754 ip-10-0-40-233 {26,62} [0] MPI startup(): 216 6471 ip-10-0-47-192 {26,62} [0] MPI startup(): 217 6467 ip-10-0-44-244 {28,64} [0] MPI startup(): 218 6472 ip-10-0-47-192 {27,63} [0] MPI startup(): 219 6473 ip-10-0-47-192 {28,64} [0] MPI startup(): 220 5755 ip-10-0-40-233 {27,63} [0] MPI startup(): 221 6474 ip-10-0-47-192 {29,65} [0] MPI startup(): 222 6475 ip-10-0-47-192 {30,66} [0] MPI startup(): 223 6591 ip-10-0-44-36 {27,63} [0] MPI startup(): 224 6592 ip-10-0-44-36 {28,64} [0] MPI startup(): 225 6527 ip-10-0-42-146 {25,61} [0] MPI startup(): 226 6528 ip-10-0-42-146 {26,62} [0] MPI startup(): 227 6593 ip-10-0-44-36 {29,65} [0] MPI startup(): 228 6468 ip-10-0-44-244 {29,65} [0] MPI startup(): 229 5756 ip-10-0-40-233 {28,64} [0] MPI startup(): 230 6455 ip-10-0-37-134 {29,65} [0] MPI startup(): 231 6529 ip-10-0-42-146 {27,63} [0] MPI startup(): 232 5757 ip-10-0-40-233 {29,65} [0] MPI startup(): 233 6469 ip-10-0-44-244 {30,66} [0] MPI startup(): 234 6456 ip-10-0-37-134 {30,66} [0] MPI startup(): 235 6825 ip-10-0-36-16 {25,61} [0] MPI startup(): 236 6489 ip-10-0-40-160 {29,65} [0] MPI startup(): 237 6826 ip-10-0-36-16 {26,62} [0] MPI startup(): 238 6470 ip-10-0-44-244 {31,67} [0] MPI startup(): 239 6490 ip-10-0-40-160 {30,66} [0] MPI startup(): 240 6491 ip-10-0-40-160 {31,67} [0] MPI startup(): 241 6594 ip-10-0-44-36 {30,66} [0] MPI startup(): 242 6827 ip-10-0-36-16 {27,63} [0] MPI startup(): 243 6828 ip-10-0-36-16 {28,64} [0] MPI startup(): 244 6595 ip-10-0-44-36 {31,67} [0] MPI startup(): 245 6530 ip-10-0-42-146 {28,64} [0] MPI startup(): 246 6829 ip-10-0-36-16 {29,65} [0] MPI startup(): 247 5758 ip-10-0-40-233 {30,66} [0] MPI startup(): 248 6596 ip-10-0-44-36 {32,68} [0] MPI startup(): 249 6476 ip-10-0-47-192 {31,67} [0] MPI startup(): 250 6531 ip-10-0-42-146 {29,65} [0] MPI startup(): 251 6532 ip-10-0-42-146 {30,66} [0] MPI startup(): 252 6533 ip-10-0-42-146 {31,67} [0] MPI startup(): 253 6534 ip-10-0-42-146 {32,68} [0] MPI startup(): 254 6830 ip-10-0-36-16 {30,66} [0] MPI startup(): 255 6492 ip-10-0-40-160 {32,68} [0] MPI startup(): 256 6597 ip-10-0-44-36 {33,69} [0] MPI startup(): 257 6831 ip-10-0-36-16 {31,67} [0] MPI startup(): 258 6493 ip-10-0-40-160 {33,69} [0] MPI startup(): 259 5759 ip-10-0-40-233 {31,67} [0] MPI startup(): 260 6494 ip-10-0-40-160 {34,70} [0] MPI startup(): 261 5760 ip-10-0-40-233 {32,68} [0] MPI startup(): 262 6457 ip-10-0-37-134 {31,67} [0] MPI startup(): 263 5761 ip-10-0-40-233 {33,69} [0] MPI startup(): 264 6458 ip-10-0-37-134 {32,68} [0] MPI startup(): 265 6471 ip-10-0-44-244 {32,68} [0] MPI startup(): 266 5762 ip-10-0-40-233 {34,70} [0] MPI startup(): 267 6832 ip-10-0-36-16 {32,68} [0] MPI startup(): 268 5763 ip-10-0-40-233 {35,71} [0] MPI startup(): 269 6472 ip-10-0-44-244 {33,69} [0] MPI startup(): 270 6598 ip-10-0-44-36 {34,70} [0] MPI startup(): 271 6477 ip-10-0-47-192 {32,68} [0] MPI startup(): 272 6535 ip-10-0-42-146 {33,69} [0] MPI startup(): 273 6478 ip-10-0-47-192 {33,69} [0] MPI startup(): 274 6479 ip-10-0-47-192 {34,70} [0] MPI startup(): 275 6833 ip-10-0-36-16 {33,69} [0] MPI startup(): 276 6473 ip-10-0-44-244 {34,70} [0] MPI startup(): 277 6834 ip-10-0-36-16 {34,70} [0] MPI startup(): 278 6599 ip-10-0-44-36 {35,71} [0] MPI startup(): 279 6459 ip-10-0-37-134 {33,69} [0] MPI startup(): 280 6460 ip-10-0-37-134 {34,70} [0] MPI startup(): 281 6536 ip-10-0-42-146 {34,70} [0] MPI startup(): 282 6461 ip-10-0-37-134 {35,71} [0] MPI startup(): 283 6474 ip-10-0-44-244 {35,71} [0] MPI startup(): 284 6835 ip-10-0-36-16 {35,71} [0] MPI startup(): 285 6495 ip-10-0-40-160 {35,71} [0] MPI startup(): 286 6480 ip-10-0-47-192 {35,71} [0] MPI startup(): 287 6537 ip-10-0-42-146 {35,71} [0] MPI startup(): I_MPI_ROOT=/opt/intel/compilers_and_libraries_2020.2.254/linux/mpi [0] MPI startup(): I_MPI_MPIRUN=mpirun [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off [0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm [0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default [0] MPI startup(): I_MPI_DEBUG=16 I am 251 of 288, on host ip-10-0-42-146 I am 253 of 288, on host ip-10-0-42-146 I am 145 of 288, on host ip-10-0-42-146 I am 243 of 288, on host ip-10-0-36-16 I am 37 of 288, on host ip-10-0-44-244 I am 125 of 288, on host ip-10-0-44-244 I am 49 of 288, on host ip-10-0-44-36 I am 159 of 288, on host ip-10-0-44-36 I am 77 of 288, on host ip-10-0-44-36 I am 219 of 288, on host ip-10-0-47-192 I am 218 of 288, on host ip-10-0-47-192 I am 149 of 288, on host ip-10-0-47-192 I am 21 of 288, on host ip-10-0-37-134 I am 121 of 288, on host ip-10-0-37-134 I am 155 of 288, on host ip-10-0-37-134 I am 179 of 288, on host ip-10-0-37-134 I am 63 of 288, on host ip-10-0-37-134 I am 103 of 288, on host ip-10-0-37-134 I am 193 of 288, on host ip-10-0-37-134 I am 279 of 288, on host ip-10-0-37-134 I am 1 of 288, on host ip-10-0-37-134 I am 93 of 288, on host ip-10-0-37-134 I am 225 of 288, on host ip-10-0-42-146 I am 250 of 288, on host ip-10-0-42-146 I am 287 of 288, on host ip-10-0-42-146 I am 209 of 288, on host ip-10-0-42-146 I am 33 of 288, on host ip-10-0-37-134 I am 69 of 288, on host ip-10-0-37-134 I am 23 of 288, on host ip-10-0-37-134 I am 231 of 288, on host ip-10-0-42-146 I am 89 of 288, on host ip-10-0-37-134 I am 169 of 288, on host ip-10-0-40-160 I am 255 of 288, on host ip-10-0-40-160 I am 285 of 288, on host ip-10-0-40-160 I am 29 of 288, on host ip-10-0-42-146 I am 101 of 288, on host ip-10-0-42-146 I am 191 of 288, on host ip-10-0-40-160 I am 181 of 288, on host ip-10-0-42-146 I am 281 of 288, on host ip-10-0-42-146 I am 67 of 288, on host ip-10-0-42-146 I am 195 of 288, on host ip-10-0-40-160 I am 245 of 288, on host ip-10-0-42-146 I am 55 of 288, on host ip-10-0-36-16 I am 269 of 288, on host ip-10-0-44-244 I am 135 of 288, on host ip-10-0-36-16 I am 75 of 288, on host ip-10-0-42-146 I am 283 of 288, on host ip-10-0-44-244 I am 137 of 288, on host ip-10-0-36-16 I am 9 of 288, on host ip-10-0-40-160 I am 167 of 288, on host ip-10-0-42-146 I am 45 of 288, on host ip-10-0-44-244 I am 141 of 288, on host ip-10-0-36-16 I am 105 of 288, on host ip-10-0-40-160 I am 47 of 288, on host ip-10-0-44-244 I am 3 of 288, on host ip-10-0-36-16 I am 185 of 288, on host ip-10-0-44-244 I am 147 of 288, on host ip-10-0-36-16 I am 239 of 288, on host ip-10-0-40-160 I am 85 of 288, on host ip-10-0-42-146 I am 233 of 288, on host ip-10-0-44-244 I am 7 of 288, on host ip-10-0-40-160 I am 71 of 288, on host ip-10-0-44-244 I am 81 of 288, on host ip-10-0-36-16 I am 117 of 288, on host ip-10-0-40-160 I am 113 of 288, on host ip-10-0-44-244 I am 83 of 288, on host ip-10-0-40-160 I am 11 of 288, on host ip-10-0-44-36 I am 175 of 288, on host ip-10-0-42-146 I am 53 of 288, on host ip-10-0-42-146 I am 19 of 288, on host ip-10-0-44-36 I am 39 of 288, on host ip-10-0-44-36 I am 87 of 288, on host ip-10-0-44-244 I am 51 of 288, on host ip-10-0-44-36 I am 139 of 288, on host ip-10-0-42-146 I am 157 of 288, on host ip-10-0-36-16 I am 203 of 288, on host ip-10-0-42-146 I am 107 of 288, on host ip-10-0-47-192 I am 91 of 288, on host ip-10-0-44-36 I am 213 of 288, on host ip-10-0-36-16 I am 277 of 288, on host ip-10-0-36-16 I am 115 of 288, on host ip-10-0-40-160 I am 73 of 288, on host ip-10-0-36-16 I am 235 of 288, on host ip-10-0-36-16 I am 79 of 288, on host ip-10-0-42-146 I am 217 of 288, on host ip-10-0-44-244 I am 275 of 288, on host ip-10-0-36-16 I am 35 of 288, on host ip-10-0-40-160 I am 13 of 288, on host ip-10-0-42-146 I am 65 of 288, on host ip-10-0-44-244 I am 265 of 288, on host ip-10-0-44-244 I am 143 of 288, on host ip-10-0-47-192 I am 97 of 288, on host ip-10-0-44-244 I am 257 of 288, on host ip-10-0-36-16 I am 171 of 288, on host ip-10-0-44-244 I am 127 of 288, on host ip-10-0-42-146 I am 61 of 288, on host ip-10-0-36-16 I am 31 of 288, on host ip-10-0-44-36 I am 223 of 288, on host ip-10-0-44-36 I am 237 of 288, on host ip-10-0-36-16 I am 267 of 288, on host ip-10-0-36-16 I am 111 of 288, on host ip-10-0-40-160 I am 151 of 288, on host ip-10-0-36-16 I am 153 of 288, on host ip-10-0-36-16 I am 163 of 288, on host ip-10-0-44-36 I am 17 of 288, on host ip-10-0-47-192 I am 158 of 288, on host ip-10-0-44-36 I am 221 of 288, on host ip-10-0-47-192 I am 99 of 288, on host ip-10-0-40-233 I am 57 of 288, on host ip-10-0-47-192 I am 161 of 288, on host ip-10-0-36-16 I am 242 of 288, on host ip-10-0-36-16 I am 95 of 288, on host ip-10-0-47-192 I am 129 of 288, on host ip-10-0-40-233 I am 227 of 288, on host ip-10-0-44-36 I am 15 of 288, on host ip-10-0-40-233 I am 119 of 288, on host ip-10-0-40-233 I am 205 of 288, on host ip-10-0-44-36 I am 133 of 288, on host ip-10-0-44-36 I am 109 of 288, on host ip-10-0-40-233 I am 229 of 288, on host ip-10-0-40-233 I am 173 of 288, on host ip-10-0-44-36 I am 241 of 288, on host ip-10-0-44-36 I am 5 of 288, on host ip-10-0-40-233 I am 261 of 288, on host ip-10-0-40-233 I am 27 of 288, on host ip-10-0-40-233 I am 41 of 288, on host ip-10-0-40-233 I am 165 of 288, on host ip-10-0-47-192 I am 215 of 288, on host ip-10-0-40-233 I am 247 of 288, on host ip-10-0-40-233 I am 197 of 288, on host ip-10-0-47-192 I am 199 of 288, on host ip-10-0-40-233 I am 189 of 288, on host ip-10-0-40-233 I am 259 of 288, on host ip-10-0-40-233 I am 59 of 288, on host ip-10-0-40-233 I am 207 of 288, on host ip-10-0-47-192 I am 249 of 288, on host ip-10-0-47-192 I am 177 of 288, on host ip-10-0-40-233 I am 201 of 288, on host ip-10-0-47-192 I am 211 of 288, on host ip-10-0-47-192 I am 263 of 288, on host ip-10-0-40-233 I am 187 of 288, on host ip-10-0-40-233 I am 271 of 288, on host ip-10-0-47-192 I am 183 of 288, on host ip-10-0-47-192 I am 273 of 288, on host ip-10-0-47-192 I am 25 of 288, on host ip-10-0-40-160 I am 43 of 288, on host ip-10-0-44-244 I am 131 of 288, on host ip-10-0-36-16 I am 123 of 288, on host ip-10-0-44-36 I am 178 of 288, on host ip-10-0-40-160 I am 66 of 288, on host ip-10-0-37-134 I am 74 of 288, on host ip-10-0-37-134 I am 78 of 288, on host ip-10-0-44-36 I am 126 of 288, on host ip-10-0-44-244 I am 254 of 288, on host ip-10-0-36-16 I am 278 of 288, on host ip-10-0-44-36 I am 194 of 288, on host ip-10-0-42-146 I am 102 of 288, on host ip-10-0-40-160 I am 62 of 288, on host ip-10-0-47-192 I am 166 of 288, on host ip-10-0-36-16 I am 22 of 288, on host ip-10-0-40-160 I am 282 of 288, on host ip-10-0-37-134 I am 134 of 288, on host ip-10-0-42-146 I am 154 of 288, on host ip-10-0-47-192 I am 94 of 288, on host ip-10-0-40-160 I am 10 of 288, on host ip-10-0-42-146 I am 4 of 288, on host ip-10-0-36-16 I am 46 of 288, on host ip-10-0-36-16 I am 6 of 288, on host ip-10-0-37-134 I am 18 of 288, on host ip-10-0-36-16 I am 54 of 288, on host ip-10-0-37-134 I am 52 of 288, on host ip-10-0-44-36 I am 34 of 288, on host ip-10-0-42-146 I am 106 of 288, on host ip-10-0-44-244 I am 86 of 288, on host ip-10-0-37-134 I am 82 of 288, on host ip-10-0-44-36 I am 162 of 288, on host ip-10-0-37-134 I am 2 of 288, on host ip-10-0-40-160 I am 38 of 288, on host ip-10-0-40-160 I am 146 of 288, on host ip-10-0-40-160 I am 70 of 288, on host ip-10-0-40-233 I am 90 of 288, on host ip-10-0-40-233 I am 174 of 288, on host ip-10-0-40-233 I am 226 of 288, on host ip-10-0-42-146 I am 98 of 288, on host ip-10-0-42-146 I am 202 of 288, on host ip-10-0-47-192 I am 188 of 288, on host ip-10-0-42-146 I am 274 of 288, on host ip-10-0-47-192 I am 230 of 288, on host ip-10-0-37-134 I am 30 of 288, on host ip-10-0-36-16 I am 190 of 288, on host ip-10-0-44-244 I am 222 of 288, on host ip-10-0-47-192 I am 238 of 288, on host ip-10-0-44-244 I am 196 of 288, on host ip-10-0-44-244 I am 198 of 288, on host ip-10-0-44-244 I am 266 of 288, on host ip-10-0-40-233 I am 276 of 288, on host ip-10-0-44-244 I am 170 of 288, on host ip-10-0-44-36 I am 14 of 288, on host ip-10-0-44-244 I am 100 of 288, on host ip-10-0-44-36 I am 50 of 288, on host ip-10-0-47-192 I am 26 of 288, on host ip-10-0-44-36 I am 246 of 288, on host ip-10-0-36-16 I am 140 of 288, on host ip-10-0-47-192 I am 92 of 288, on host ip-10-0-40-160 I am 142 of 288, on host ip-10-0-44-244 I am 58 of 288, on host ip-10-0-44-244 I am 286 of 288, on host ip-10-0-47-192 I am 114 of 288, on host ip-10-0-42-146 I am 44 of 288, on host ip-10-0-40-233 I am 138 of 288, on host ip-10-0-37-134 I am 150 of 288, on host ip-10-0-44-244 I am 234 of 288, on host ip-10-0-37-134 I am 84 of 288, on host ip-10-0-36-16 I am 118 of 288, on host ip-10-0-40-160 I am 186 of 288, on host ip-10-0-36-16 I am 210 of 288, on host ip-10-0-40-233 I am 68 of 288, on host ip-10-0-47-192 I am 110 of 288, on host ip-10-0-37-134 I am 228 of 288, on host ip-10-0-44-244 I am 236 of 288, on host ip-10-0-40-160 I am 28 of 288, on host ip-10-0-44-244 I am 220 of 288, on host ip-10-0-40-233 I am 270 of 288, on host ip-10-0-44-36 I am 182 of 288, on host ip-10-0-40-233 I am 180 of 288, on host ip-10-0-40-160 I am 12 of 288, on host ip-10-0-36-16 I am 214 of 288, on host ip-10-0-40-160 I am 258 of 288, on host ip-10-0-40-160 I am 244 of 288, on host ip-10-0-44-36 I am 206 of 288, on host ip-10-0-40-160 I am 88 of 288, on host ip-10-0-44-36 I am 284 of 288, on host ip-10-0-36-16 I am 262 of 288, on host ip-10-0-37-134 I am 136 of 288, on host ip-10-0-47-192 I am 116 of 288, on host ip-10-0-37-134 I am 108 of 288, on host ip-10-0-44-36 I am 184 of 288, on host ip-10-0-42-146 I am 232 of 288, on host ip-10-0-40-233 I am 216 of 288, on host ip-10-0-47-192 I am 268 of 288, on host ip-10-0-40-233 I am 8 of 288, on host ip-10-0-40-233 I am 42 of 288, on host ip-10-0-44-36 I am 212 of 288, on host ip-10-0-47-192 I am 204 of 288, on host ip-10-0-44-244 I am 130 of 288, on host ip-10-0-47-192 I am 80 of 288, on host ip-10-0-47-192 I am 280 of 288, on host ip-10-0-37-134 I am 260 of 288, on host ip-10-0-40-160 I am 176 of 288, on host ip-10-0-40-160 I am 122 of 288, on host ip-10-0-40-233 I am 104 of 288, on host ip-10-0-37-134 I am 264 of 288, on host ip-10-0-37-134 I am 200 of 288, on host ip-10-0-40-233 I am 272 of 288, on host ip-10-0-42-146 I am 256 of 288, on host ip-10-0-44-36 I am 76 of 288, on host ip-10-0-44-36 I am 72 of 288, on host ip-10-0-37-134 I am 36 of 288, on host ip-10-0-44-244 I am 124 of 288, on host ip-10-0-44-244 I am 252 of 288, on host ip-10-0-42-146 I am 20 of 288, on host ip-10-0-37-134 I am 120 of 288, on host ip-10-0-36-16 I am 112 of 288, on host ip-10-0-47-192 I am 148 of 288, on host ip-10-0-47-192 I am 96 of 288, on host ip-10-0-47-192 I am 248 of 288, on host ip-10-0-44-36 I am 156 of 288, on host ip-10-0-47-192 I am 132 of 288, on host ip-10-0-40-233 I am 172 of 288, on host ip-10-0-40-160 I am 152 of 288, on host ip-10-0-40-160 I am 24 of 288, on host ip-10-0-40-233 I am 240 of 288, on host ip-10-0-40-160 I am 164 of 288, on host ip-10-0-40-160 I am 40 of 288, on host ip-10-0-40-160 I am 168 of 288, on host ip-10-0-44-244 I am 16 of 288, on host ip-10-0-40-233 I am 144 of 288, on host ip-10-0-42-146 I am 224 of 288, on host ip-10-0-44-36 I am 208 of 288, on host ip-10-0-44-36 I am 160 of 288, on host ip-10-0-37-134 I am 64 of 288, on host ip-10-0-37-134 I am 60 of 288, on host ip-10-0-40-233 I am 56 of 288, on host ip-10-0-37-134 I am 48 of 288, on host ip-10-0-44-36 I am 32 of 288, on host ip-10-0-44-244 I am 192 of 288, on host ip-10-0-44-244 I am 128 of 288, on host ip-10-0-37-134 I am 0 of 288, on host ip-10-0-36-16