launching mpiexec.hydra from host

[YH]

Hi,

I am trying to do WRF code on a mic card following the instruction from https://software.intel.com/en-us/articles/how-to-get-wrf-running-on-the-intelr-xeon-phitm-coprocessor

However, when I try to do 

mpiexec.hydra -np 1 ./wrf.exe

It gave the following error

-bash: /opt/intel//impi/5.0.2.044/mic/bin/mpiexec.hydra: cannot execute binary file

I am wondering whether anyone had this problem before? mpiexe.hydra is set using "source /opt/intel/impi/5.0.2.044/mic/bin/mpivars.sh"

Thanks

[Sunny_G_Intel]

Hi YH,

I see an error in your program launch command

mpiexec.hydra -host -iface br0 -iface `hostname` -iface br0 -host `hostname` -np 2 ./test.host : -iface br0 -host mic0 -np 2 /tmp/test.mic

[YH]

Sunny,

I over copied the command, it should started from line #6 of the previous post. 

I lost the connection, And reconnect, I ran into a different problem

mpiexec.hydra -iface br0 -host eagle-mic0 -n 2 /tmp/test.mic
bash: /opt/intel/impi/5.0.2.044/intel64/bin/pmi_proxy: cannot execute binary file

Thanks,

[YH]

Didn't set the I_MPI_MIC=1

But the symmetric mode problem remain.

 mpiexec.hydra -iface br0 -host mic0 -np 2 /tmp/test.mic : -host `hostname` -np 2 ~/test/test.host
eagle:SCM:251e:285d3b40: 63 us(63 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 62 us(62 us):  open_hca: device mlx4_0 not found
eagle:CMA:251e:285d3b40: 30 us(30 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle:CMA:251e:285d3b40: 28 us(28 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle:SCM:251d:78a4bb40: 75 us(75 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 62 us(62 us):  open_hca: device mthca0 not found
eagle:SCM:251d:78a4bb40: 77 us(77 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 63 us(63 us):  open_hca: device mthca0 not found
eagle:SCM:251e:285d3b40: 55 us(55 us):  open_hca: device ipath0 not found
eagle:CMA:251d:78a4bb40: 41 us(41 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle:SCM:251e:285d3b40: 56 us(56 us):  open_hca: device ipath0 not found
eagle:CMA:251d:78a4bb40: 35 us(35 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle:SCM:251e:285d3b40: 65 us(65 us):  open_hca: device ehca0 not found
eagle:SCM:251d:78a4bb40: 76 us(76 us):  open_hca: device mthca0 not found
eagle:CMA:251e:285d3b40: 38 us(38 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle:SCM:251d:78a4bb40: 63 us(63 us):  open_hca: device mthca0 not found
eagle:UCM:251e:285d3b40: 64 us(64 us):  open_hca: mlx4_0 not found
eagle:SCM:251d:78a4bb40: 59 us(59 us):  open_hca: device ipath0 not found
eagle:UCM:251e:285d3b40: 66 us(66 us):  open_hca: mlx4_0 not found
eagle:SCM:251d:78a4bb40: 58 us(58 us):  open_hca: device ipath0 not found
eagle:UCM:251e:285d3b40: 65 us(65 us):  open_hca: mthca0 not found
eagle:SCM:251d:78a4bb40: 63 us(63 us):  open_hca: device ehca0 not found
eagle:UCM:251e:285d3b40: 56 us(56 us):  open_hca: mthca0 not found
eagle:CMA:251d:78a4bb40: 28 us(28 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle:CMA:251e:285d3b40: 31 us(31 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle:UCM:251d:78a4bb40: 55 us(55 us):  open_hca: mlx4_0 not found
eagle:CMA:251e:285d3b40: 35 us(35 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle:UCM:251d:78a4bb40: 62 us(62 us):  open_hca: mlx4_0 not found
eagle:SCM:251e:285d3b40: 64 us(64 us):  open_hca: device mlx4_0 not found
eagle:UCM:251d:78a4bb40: 55 us(55 us):  open_hca: mthca0 not found
eagle:SCM:251e:285d3b40: 59 us(59 us):  open_hca: device mlx4_0 not found
eagle:UCM:251d:78a4bb40: 57 us(57 us):  open_hca: mthca0 not found
eagle:SCM:251e:285d3b40: 59 us(59 us):  open_hca: device scif0 not found
eagle:CMA:251d:78a4bb40: 30 us(30 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle:UCM:251e:285d3b40: 64 us(64 us):  open_hca: scif0 not found
eagle:CMA:251d:78a4bb40: 29 us(29 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle:CMA:251e:285d3b40: 30 us(30 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle:SCM:251d:78a4bb40: 62 us(62 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 68 us(68 us):  open_hca: device mlx4_0 not found
eagle:SCM:251d:78a4bb40: 60 us(60 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 62 us(62 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 53 us(53 us):  open_hca: device mlx4_1 not found
eagle:SCM:251d:78a4bb40: 59 us(59 us):  open_hca: device scif0 not found
eagle:SCM:251e:285d3b40: 58 us(58 us):  open_hca: device mlx4_1 not found
eagle:UCM:251d:78a4bb40: 57 us(57 us):  open_hca: scif0 not found
eagle:UCM:251e:285d3b40: 59 us(59 us):  open_hca: mlx4_1 not found
eagle:CMA:251d:78a4bb40: 28 us(28 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle:UCM:251e:285d3b40: 57 us(57 us):  open_hca: mlx4_1 not found
eagle:SCM:251d:78a4bb40: 60 us(60 us):  open_hca: device mlx4_0 not found
eagle:SCM:251d:78a4bb40: 57 us(57 us):  open_hca: device mlx4_0 not found
eagle:SCM:251e:285d3b40: 59 us(59 us):  open_hca: device mlx5_0 not found
eagle:SCM:251d:78a4bb40: 67 us(67 us):  open_hca: device mlx4_1 not found
eagle:SCM:251e:285d3b40: 57 us(57 us):  open_hca: device mlx5_0 not found
eagle:SCM:251d:78a4bb40: 58 us(58 us):  open_hca: device mlx4_1 not found
eagle:SCM:251e:285d3b40: 56 us(56 us):  open_hca: device mlx5_1 not found
eagle:UCM:251d:78a4bb40: 58 us(58 us):  open_hca: mlx4_1 not found
eagle:SCM:251e:285d3b40: 57 us(57 us):  open_hca: device mlx5_1 not found
eagle:UCM:251d:78a4bb40: 59 us(59 us):  open_hca: mlx4_1 not found
eagle:UCM:251e:285d3b40: 57 us(57 us):  open_hca: mlx5_0 not found
eagle:UCM:251e:285d3b40: 61 us(61 us):  open_hca: mlx5_0 not found
eagle:SCM:251d:78a4bb40: 57 us(57 us):  open_hca: device mlx5_0 not found
eagle:UCM:251e:285d3b40: 57 us(57 us):  open_hca: mlx5_1 not found
eagle:SCM:251d:78a4bb40: 55 us(55 us):  open_hca: device mlx5_0 not found
eagle:UCM:251e:285d3b40: 58 us(58 us):  open_hca: mlx5_1 not found
eagle:SCM:251d:78a4bb40: 56 us(56 us):  open_hca: device mlx5_1 not found
eagle:SCM:251d:78a4bb40: 54 us(54 us):  open_hca: device mlx5_1 not found
eagle:UCM:251d:78a4bb40: 55 us(55 us):  open_hca: mlx5_0 not found
eagle:UCM:251d:78a4bb40: 52 us(52 us):  open_hca: mlx5_0 not found
eagle:UCM:251d:78a4bb40: 52 us(52 us):  open_hca: mlx5_1 not found
eagle:UCM:251d:78a4bb40: 52 us(52 us):  open_hca: mlx5_1 not found
eagle-mic0:SCM:129e:7ea00b40: 292 us(292 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 263 us(263 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 211 us(211 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 215 us(215 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:CMA:129e:7ea00b40: 585 us(585 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle-mic0:CMA:129f:a898eb40: 774 us(774 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle-mic0:CMA:129e:7ea00b40: 544 us(544 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle-mic0:CMA:129f:a898eb40: 688 us(688 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle-mic0:SCM:129e:7ea00b40: 236 us(236 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 456 us(456 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 232 us(232 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 461 us(461 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 210 us(210 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 228 us(228 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 213 us(213 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 235 us(235 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 211 us(211 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 230 us(230 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:CMA:129e:7ea00b40: 560 us(560 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:129f:a898eb40: 697 us(697 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:UCM:129e:7ea00b40: 204 us(204 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 212 us(212 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 182 us(182 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 205 us(205 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 192 us(192 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 205 us(205 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 174 us(174 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 199 us(199 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:CMA:129e:7ea00b40: 555 us(555 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:129f:a898eb40: 546 us(546 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:129e:7ea00b40: 541 us(541 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle-mic0:CMA:129f:a898eb40: 579 us(579 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle-mic0:SCM:129e:7ea00b40: 224 us(224 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 224 us(224 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 220 us(220 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 246 us(246 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 216 us(216 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 223 us(223 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 175 us(175 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 186 us(186 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:CMA:129e:7ea00b40: 557 us(557 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle-mic0:CMA:129f:a898eb40: 554 us(554 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle-mic0:SCM:129e:7ea00b40: 733 us(733 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 226 us(226 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 219 us(219 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 217 us(217 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 213 us(213 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 281 us(281 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 334 us(334 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 171 us(171 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 213 us(213 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 175 us(175 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 208 us(208 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 201 us(201 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 215 us(215 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 220 us(220 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 234 us(234 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 240 us(240 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 235 us(235 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129e:7ea00b40: 213 us(213 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 289 us(289 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 175 us(175 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:129f:a898eb40: 239 us(239 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 179 us(179 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 186 us(186 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 172 us(172 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 189 us(189 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129e:7ea00b40: 181 us(181 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 181 us(181 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:129f:a898eb40: 185 us(185 us):  open_hca: ibv_get_device_list() failed

[YH]

Or trying using two mic cards.

mpiexec.hydra -iface br0 -host mic1 -np 2 /tmp/test.mic : -host mic0 -np 2 /tmp/test.mic
eagle-mic1:SCM:1293:8260eb40: 222 us(222 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 206 us(206 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 299 us(299 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 219 us(219 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 212 us(212 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 256 us(256 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 219 us(219 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 215 us(215 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:CMA:1293:8260eb40: 567 us(567 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle-mic1:CMA:1294:d2f2db40: 710 us(710 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle-mic0:CMA:12bd:34781b40: 577 us(577 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle-mic0:CMA:12be:f1fdab40: 763 us(763 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib0 configured?
eagle-mic1:CMA:1293:8260eb40: 540 us(540 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle-mic1:CMA:1294:d2f2db40: 818 us(818 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle-mic0:CMA:12bd:34781b40: 541 us(541 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle-mic0:CMA:12be:f1fdab40: 702 us(702 us):  open_hca: getaddr_netdev ERROR:No such device. Is ib1 configured?
eagle-mic1:SCM:1293:8260eb40: 219 us(219 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 262 us(262 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 223 us(223 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 218 us(218 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 228 us(228 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 208 us(208 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 257 us(257 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 206 us(206 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 219 us(219 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 207 us(207 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 387 us(387 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 254 us(254 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 210 us(210 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 216 us(216 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 277 us(277 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 253 us(253 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 215 us(215 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 203 us(203 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 272 us(272 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 244 us(244 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:CMA:1293:8260eb40: 554 us(554 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:12be:f1fdab40: 546 us(546 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:12bd:34781b40: 536 us(536 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic1:CMA:1294:d2f2db40: 577 us(577 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic1:UCM:1293:8260eb40: 199 us(199 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 170 us(170 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 182 us(182 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 186 us(186 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 172 us(172 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 199 us(199 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 197 us(197 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 190 us(190 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 177 us(177 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 171 us(171 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 169 us(169 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 193 us(193 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 164 us(164 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 181 us(181 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 182 us(182 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 186 us(186 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:CMA:1293:8260eb40: 545 us(545 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:12be:f1fdab40: 551 us(551 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic0:CMA:12bd:34781b40: 738 us(738 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic1:CMA:1294:d2f2db40: 562 us(562 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth2 configured?
eagle-mic1:CMA:1293:8260eb40: 550 us(550 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle-mic0:CMA:12be:f1fdab40: 543 us(543 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle-mic0:CMA:12bd:34781b40: 691 us(691 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle-mic1:SCM:1293:8260eb40: 230 us(230 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:CMA:1294:d2f2db40: 579 us(579 us):  open_hca: getaddr_netdev ERROR:No such device. Is eth3 configured?
eagle-mic0:SCM:12be:f1fdab40: 231 us(231 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 527 us(527 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 207 us(207 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 223 us(223 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 210 us(210 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 305 us(305 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 289 us(289 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 268 us(268 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 209 us(209 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 413 us(413 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 170 us(170 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 172 us(172 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 229 us(229 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 163 us(163 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:CMA:1293:8260eb40: 559 us(559 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle-mic1:UCM:1294:d2f2db40: 189 us(189 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:CMA:12be:f1fdab40: 551 us(551 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle-mic1:SCM:1293:8260eb40: 217 us(217 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:CMA:12bd:34781b40: 656 us(656 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle-mic0:SCM:12be:f1fdab40: 219 us(219 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 213 us(213 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 301 us(301 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:CMA:1294:d2f2db40: 835 us(835 us):  open_hca: getaddr_netdev ERROR:Cannot assign requested address. Is mic0:ib configured?
eagle-mic0:SCM:12be:f1fdab40: 210 us(210 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 200 us(200 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 205 us(205 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 257 us(257 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 206 us(206 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 327 us(327 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 207 us(207 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 249 us(249 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 211 us(211 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 179 us(179 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 286 us(286 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 255 us(255 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 172 us(172 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 172 us(172 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 164 us(164 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 283 us(283 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 170 us(170 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 161 us(161 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 183 us(183 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 183 us(183 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 215 us(215 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 218 us(218 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 234 us(234 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 233 us(233 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 212 us(212 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 205 us(205 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 206 us(206 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1293:8260eb40: 207 us(207 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 250 us(250 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 205 us(205 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 203 us(203 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 177 us(177 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 246 us(246 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12be:f1fdab40: 227 us(227 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:SCM:12bd:34781b40: 197 us(197 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 173 us(173 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 222 us(222 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 178 us(178 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 173 us(173 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 168 us(168 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 179 us(179 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 177 us(177 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:SCM:1294:d2f2db40: 220 us(220 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1293:8260eb40: 422 us(422 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 177 us(177 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 177 us(177 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 197 us(197 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12bd:34781b40: 202 us(202 us):  open_hca: ibv_get_device_list() failed
eagle-mic0:UCM:12be:f1fdab40: 195 us(195 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 181 us(181 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 184 us(184 us):  open_hca: ibv_get_device_list() failed
eagle-mic1:UCM:1294:d2f2db40: 177 us(177 us):  open_hca: ibv_get_device_list() failed
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.

[Sunny_G_Intel]

Hi YH,

Setting the environment variable DAPL_DBG_TYPE to 0 will get rid of dapl related debug output. 

export DAPL_DBG_TYPE=0

Thanks

[YH]

This is the outcome

mpiexec.hydra -iface br0 -host mic0 -np 2 /tmp/test.mic : -host mic1 -np 2 /tmp/test.mic
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
eagle-mic0.4824PSM can't open /dev/ipath for reading and writing (err=23)
eagle-mic0.4823PSM can't open /dev/ipath for reading and writing (err=23)
[0] MPI startup(): tmi fabric is not available and fallback fabric is not exist
[1] MPI startup(): tmi fabric is not available and fallback fabric is not exist
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
scif_connect failed with error 111(Connection refused)
Please check if /usr/sbin/psmd is running on host.
eagle-mic1.4768PSM can't open /dev/ipath for reading and writing (err=23)
eagle-mic1.4767PSM can't open /dev/ipath for reading and writing (err=23)
[2] MPI startup(): tmi fabric is not available and fallback fabric is not exist
[3] MPI startup(): tmi fabric is not available and fallback fabric is not exist

[Sunny_G_Intel]

Hi YH,

It is strange that you are getting those errors while trying to run a test program. I setup a machine with 2 coprocessors and enabled static internal bridge. I mounted a NFS export to share /opt/intel/ directory on both the coprocessors. 

mpiexec.hydra -host `hostname` -np 2 ./test.host : -host mic0 -np 2 /tmp/test.mic : -host mic1 -np 2 /tmp/test.mic
Hello world: rank 0 of 6 running on knightscorner7.jf.intel.com
Hello world: rank 1 of 6 running on knightscorner7.jf.intel.com
Hello world: rank 2 of 6 running on knightscorner7-mic0.jf.intel.com
Hello world: rank 3 of 6 running on knightscorner7-mic0.jf.intel.com
Hello world: rank 4 of 6 running on knightscorner7-mic1.jf.intel.com
Hello world: rank 5 of 6 running on knightscorner7-mic1.jf.intel.com

mpiexec.hydra -host mic0 -np 2 /tmp/test.mic : -host mic1 -np 2 /tmp/test.mic                                                                     
Hello world: rank 0 of 4 running on knightscorner7-mic0.jf.intel.com
Hello world: rank 1 of 4 running on knightscorner7-mic0.jf.intel.com
Hello world: rank 2 of 4 running on knightscorner7-mic1.jf.intel.com
Hello world: rank 3 of 4 running on knightscorner7-mic1.jf.intel.com

Since I was working with internal bridging and using NFS exports there are few things I verified:

1. Mount was successful on both the coprocessors. Check if /opt/intel/ is seen to be populated on both coprocessors (If mount is unsuccessful try turning OFF firewall)

2. You should be able to ping from 1 coprocessor to another and also to hosts. Also you must be able to SSH into both coprocessors without any password.

3. set environment variables: 

source /opt/intel/composerxe/bin/compilervars.sh intel64
source /opt/intel/impi/5.0.0.028/bin64/mpivars.sh     //set MPI environment on Host
​export I_MPI_MIC=1                   //copy environment variables to MIC - but will only make sense if /opt/intel is available on coprocessors through NFS
export DAPL_DBG_TYPE=0              //silent DAPL debug output

If everything is fine, now try executing the above test programs (but first test.mic should be copied over to coprocessors /tmp/) 

Thanks

[Artem_R_Intel1]

Hi YH,

The described output looks like ofed-mic service isn't run on the HOST.

Could please check the status of the following services on the HOST (sudo permissions are required):
sudo service openibd status
sudo service opensmd status
sudo service ofed-mic status
sudo service mpxyd status
They should be online. If not please start them in the same sequence (openibd, opensmd, ofed-mic, mpxyd).
Possibly due to connectivity problems these services weren't started correctly.

In case of any problems, could you please provide the output of the following commands:
sudo ibv_devices
sudo ibv_devinfo
cat /etc/dat.conf
ofed_info
sudo micinfo

[YH]

Artem,

Didn't Install the option ofed-mic. I thought that if I am only using internal bridge I don't need that. Plus, I ran into error while installing the ofed-mic. 

I followed the instruction on MPSS user guide 2.5 (Steps to Install Intel MPSS using Mellanox* OFED 2.1/2.2/2.3) step 3 rebuilding the rpm file, there are error like the following 

include/uapi/linux/types.h:4:23: fatal error: asm/types.h: No such file or directory

/usr/src/ofa_kernel/default/include/linux/export.h:4:27: fatal error: linux/version.h: No such file or directory

At this point, openibd is loaded, opensm is stopped, ofed-mic not installed, there is not mpxyd either. 

Thanks,

YH

[Artem_R_Intel1]

Hi YH,

There're two options to run MPI application on MIC without InfiniBand adapter (IBA):
1. TCP (OFED isn't required):
$ export I_MPI_FABRICS=shm:tcp (or tcp)
In this case you possibly need to remove '-iface br0' option (try both variants).

2. DAPL over SCIF (ofed-mic service is required):
$ export I_MPI_FABRICS=shm:dapl (or dapl)
$ export I_MPI_DAPL_PROVIDER=ofa-v2-scif0

Regarding to Mellanox OFED rpm building failure I'd recommend to check you use OS supported by your OFED/MPSS.
Another option is to try to install OpenFabrics OFED (see the instructions and the supported versions in the MPSS User Guide).

[Sunny_G_Intel]

Hi YH,

If you are using MPI application on MIC with TCP then the following error as per your post:

mpiexec.hydra -host mic0 -np 2 /tmp/test.mic 

[proxy:0:0@eagle-mic0] HYDU_sock_connect (../../utils/sock/sock.c:268): unable to connect from "eagle-mic0" to "127.0.0.1" (Connection refused)
[proxy:0:0@eagle-mic0] main (../../pm/pmiserv/pmip.c:397): unable to connect to server 127.0.0.1 at port 49052 (check for firewalls!)
^C[mpiexec@eagle] Sending Ctrl-C to processes as requested
[mpiexec@eagle] Press Ctrl-C again to force abort
[mpiexec@eagle] HYDU_sock_write (../../utils/sock/sock.c:417): write error (Bad file descriptor)
[mpiexec@eagle] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:244): unable to write data to proxy
[mpiexec@eagle] ui_cmd_cb (../../pm/pmiserv/pmiserv_pmci.c:175): unable to send signal downstream
[mpiexec@eagle] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@eagle] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:495): error waiting for event
[mpiexec@eagle] main (../../ui/mpich/mpiexec.c:1011): process manager error waiting for completion

caan be resolved by just turning off firewall(or opening correct ports) and trying again. 

Thanks

[Deepthi_C_]

Hi ,

I followed this post and was able to solved few of my errors. However, when i execute WRF, it gives the following messages in between.

export I_MPI_MIC=1
export I_MPI_FABRICS=shm:tcp
mpiexec.hydra -hosts tbx-node05 -n 24 -env WRF_NUM_TILES 24 -env KMP_AFFINITY scatter -env OMP_NUM_THREADS 2 -env KMP_LIBRARY turnaround -env OMP_SCHEDULE static -env KMP_STACKSIZE 190M /data/conus2.5km/x86/wrf.exe : -hosts tbx-node05-mic0 -n 8 -env KMP_AFFINITY balanced -env OMP_NUM_THREADS 30 -env KMP_LIBRARY turnaround -env OMP_SCHEDULE static -env KMP_STACKSIZE 190M /data/conus2.5km/mic/wrf.sh

NOTE: /data is accesible by the host and mics.

Part of the output: i can successfully launch the process and following error messages come in-between the run.

error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.

 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.

error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.
 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.
error_dup: cannot open rsl.out.nnnn: Permission denied
...sending output to standard output and continuing.
 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.
 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.
 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.
 Namelist logging not found in namelist.input. Using registry defaults for varia
 bles in logging.
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8
  Ntasks in X            4, ntasks in Y            8

input_wrf.F reading 2d real xlat
  date 2005-06-04_06:00:00
  ds            1           1           1
  de         1500        1200           1
 input_wrf.F reading 2d real xlat
  date 2005-06-04_06:00:00
  ds            1           1           1
  de         1500        1200           1
  ps         1126         901           1
  pe         1500        1050           1
  ms         1119         894           1
  me         1506        1057           1
 module_io.F: in wrf_read_field
  ps         1126         901           1
  pe         1500        1050           1
  ms         1119         894           1
  me         1506        1057           1
 module_io.F: in wrf_read_field
  alloc_space_field: domain            1,             1434915832 bytes allocated
  alloc_space_field: domain            1,             1434915832 bytes allocated
 wrf: calling model_to_grid_config_rec
 wrf: calling set_scalar_indices_from_config
 wrf: calling model_to_grid_config_rec
 wrf: calling set_scalar_indices_from_config
 wrf: calling init_wrfio
 module_io.F: in wrf_ioinit
 wrf: calling init_wrfio
 module_io.F: in wrf_ioinit
 Entering ext_gr1_ioinit
 Entering ext_gr1_ioinit
This is just part of the output. And how do i know that WRF is successfully running on the mic?  i see some processes running on it when i ssh into mic but i don't see any utilization in micsmc.

Any kind of help is appreciated.

Thanks,

Deepthi

[Frances_R_Intel]

The files rsl.out.nnnn and rsl.error.nnnn (there should be a file for each processor/coprocessor you are running on) should be getting written over each time you run. The message saying "cannot open rsl.out.nnnn: Permission denied" implies that the file either does not have write permission or is owned by a different user. Did you change what user you were running as, at some point? Can you find those files on the host and check the ownership and permissions? There is no problem with just removing those files by hand but it would be good to know why you couldn't just write over them.

When you compiled the code, did you enable any debugging - set any variables for the ifdef that would cause you to print additional information? A lot of that output looks like debug messages.

In your mpiexec.hydra command, you are running wrf.exe on the host but wrf.sh on the coprocessor. You need to run wrf.exe on the both host and coprocessor. That script wrf.sh will try to start up a new mpiexec.hydra on the coprocessor, not what you want to do. 

You have 24 WRF tiles and are running 24 MPI ranks on the host. This means there won't be any MPI ranks required on the coprocessor. You might want to increase the number of tiles - or cut down on the number of MPI ranks on the host - if your goal is to make sure the coprocessor does some of the work.

[yong_w_]

Hi, YH. Would you solved your issues listed above?

I am also porting WRF to xeon phi, and across the problems you met. I am appreciate that you tell me you QQ number for my further learning from you. looking forward to your reply, have a nice day.

[yong_w_]

Hi, YH. Would you solved your issues listed above?

I am also porting WRF to xeon phi, and across the problems you met. I am appreciate that you tell me you QQ number for my further learning from you. looking forward to your reply, have a nice day.

[Sunny_G_Intel]

Hello Yong,

Can you please let me know what  particular error are you getting while porting WRF to xeon phi. We may be able to resolve your issue.

Thanks