Software Archive
Read-only legacy content
17061 Discussions

MPSS 3.6, OFED-3.18 and test.c

Dmitry_Y_
Beginner
‎10-08-2015 04:08 AM
735 Views

Hello,

I have installed MPSS 3.6, OFED-3.18 on hos and Intel Xeon Phi

I build simple test from /test directory of Intel MPI 5.1 for host and mic. I run test with command line 

mpirun -n 2 -genv I_MPI_DEBUG=5 -host node ./test :  -n 6 -host node-mic0 ./test-mic

but I got failed run

Intel(R) MPI Library for Linux* OS, Version 5.1.1 Build 20150715 (build id: 12269)
Copyright (C) 2003-2015, Intel Corporation. All rights reserved.

[0] MPI startup(): Multi-threaded optimized library
[0] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[0] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-scif0
[1] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[1] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-scif0
[0] MPI startup(): secondary DAPL provider ofa-v2-scif0
[0] DAPL startup(): trying to open secondary (3) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[1] MPI startup(): secondary DAPL provider ofa-v2-scif0
[1] DAPL startup(): trying to open secondary (3) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[0] MPI startup(): secondary DAPL provider ofa-v2-mcm-1
[0] MPI startup(): shm and dapl data transfer modes
[1] MPI startup(): secondary DAPL provider ofa-v2-mcm-1
[1] MPI startup(): shm and dapl data transfer modes
[6] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[2] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[7] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[3] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[4] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[5] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1u
[6] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[6] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[2] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[2] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[7] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[7] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[3] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[3] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[4] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[5] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[5] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[4] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-1
[6] MPI startup(): secondary DAPL provider ofa-v2-mcm-1
[6] MPI startup(): shm and dapl data transfer modes
[2] MPI startup(): secondary DAPL provider ofa-v2-mcm-1
[2] MPI startup(): shm and dapl data transfer modes
node-mic0:MCM:1695:c0e0cb40: 30037330 us(30037330 us!!!): scif_connect() to port 68, failed with error No such device
node-mic0:MCM:1695:c0e0cb40: 30037760 us(430 us):  open_hca: SCIF init ERR for mlx4_0
node-mic0:MCM:1696:b80b9b40: 30037620 us(30037620 us!!!): scif_connect() to port 68, failed with error No such device
node-mic0:MCM:1696:b80b9b40: 30037743 us(123 us):  open_hca: SCIF init ERR for mlx4_0
[4] DAPL startup(): failed to open DAPL provider ofa-v2-mcm-1
[5] DAPL startup(): failed to open DAPL provider ofa-v2-mcm-1

I would like to run with ofa-v2-mcm-1 provider with other its work.

I run dmesg command and saw

#dmesg node
[ 2191.159984] __scif_register 2607 err -107
[ 2191.184605] __scif_register 2626 err -107
[ 2515.793982] __scif_register 2607 err -107
[ 2515.819731] __scif_register 2626 err -107

#lsmod node
Module                  Size  Used by    Tainted: G  
ibscif                 80845  0 
ibp_sa_server          17302  0 
ibp_cm_server          17651  0 
ibp_server             39970 24 
bridge                 78015  0 
stp                    12437  1 bridge
llc                    12745  2 bridge,stp
mic                   607687 36 ibscif,ibp_sa_server,ibp_cm_server,ibp_server
nfsv3                  37785  1 
nfs_acl                12511  1 nfsv3
nfs                   192309  2 nfsv3
lockd                  83417  2 nfsv3,nfs
fscache                36394  1 nfs
sunrpc                237551  7 nfsv3,nfs_acl,nfs,lockd
rdma_ucm               22055  0 
rdma_cm                39550  1 rdma_ucm
iw_cm                  35426  1 rdma_cm
ib_uverbs              44841  5 rdma_ucm
mlx4_ib               132318  1 
ib_umad                21450  0 
ib_ipoib               85119  0 
ib_cm                  35027  3 ibp_cm_server,rdma_cm,ib_ipoib
ib_sa                  27032  6 ibp_sa_server,rdma_ucm,rdma_cm,mlx4_ib,ib_ipoib,ib_cm
mlx4_core             206809  1 mlx4_ib
ib_mad                 44194  4 mlx4_ib,ib_umad,ib_cm,ib_sa
ib_core                69071 12 ibscif,ibp_server,rdma_ucm,rdma_cm,iw_cm,ib_uverbs,mlx4_ib,ib_umad,ib_ipoib,ib_cm,ib_sa,ib_mad
ib_addr                13099  3 rdma_ucm,rdma_cm,ib_core
compat                 13563 17 ibscif,ibp_sa_server,ibp_cm_server,ibp_server,rdma_ucm,rdma_cm,iw_cm,ib_uverbs,mlx4_ib,ib_umad,ib_ipoib,ib_cm,ib_sa,mlx4_core,ib_mad,ib_core,ib_addr
igb                   171790  0 
ptp                    17460  1 igb
pps_core               17249  1 ptp
i2c_algo_bit           12751  1 igb
i2c_core               28006  2 igb,i2c_algo_bit

-------- lsmod node-mic0
Module                  Size  Used by
blcr                   95175  0 
blcr_imports            2581  1 blcr
sep3_15                45433  2 
ib_ipoib               54466  0 
pm_scif                 4582  0 
rdma_ucm               10191  0 
rdma_cm                23788  1 rdma_ucm
iw_cm                   6398  1 rdma_cm
ib_addr                 4356  1 rdma_cm
ib_qib                 23107  0 
ibp_cm_client          20435  2 ib_ipoib,rdma_cm
ibp_sa_client          16779  3 ib_ipoib,rdma_ucm,rdma_cm
ibp_mlx4               28679  0 
ibp_mlx5               27170  0 
ibp_client             24749  4 ibp_cm_client,ibp_sa_client,ibp_mlx4,ibp_mlx5
ib_uverbs              31618  1 rdma_ucm
ibscif                 67365  0 
ib_core                45751  10 ib_ipoib,rdma_ucm,rdma_cm,iw_cm,ibp_sa_client,ibp_mlx4,ibp_mlx5,ibp_client,ib_uverbs,ibscif
mpssboot                2833  0 
micras                 98653  0 
ramoops                 2077  0 
micscif               292933  26 pm_scif,ib_qib,ibp_cm_client,ibp_sa_client,ibp_client,ibscif,mpssboot,micras
intel_micveth          22875  0 
dma_module             33080  2 micscif,intel_micveth
michvc                  2943  0 
ringbuffer              2297  2 micscif,michvc

As I could understand it is not a problem IMPI it is problem scif0. Does any one have some idea what is wrong?

 

0 Kudos

Link Copied

7 Replies
Artem_R_Intel1
Employee
‎10-08-2015 05:39 AM
735 Views

Hi Dmitry,

It looks like mpxyd daemon isn't running. This daemon is needed for ofa-v2-mcm-1 DAPL provider. You can find more details in the Intel® MPSS User's Guide.

Also you can find the similar example in the Intel MPI Library User's Guide: Troubleshooting on Linux*.

0 Kudos
Copy link
Dmitry_Y_
Beginner
‎10-08-2015 05:44 AM
735 Views

Artem R. (Intel) wrote:

Hi Dmitry,

It looks like mpxyd daemon isn't running. This daemon is needed for ofa-v2-mcm-1 DAPL provider. You can find more details in the Intel® MPSS User's Guide.

Also you can find the similar example in the Intel MPI Library User's Guide: Troubleshooting on Linux*.

Hello  Artem,

In case of mpxyd not running I will got such message:

node-mic0:MCM:1595:43da8700: 4789 us(4789 us): scif_connect() to port 68, failed with error Connection refused
node-mic0:MCM:1595:43da8700: 4893 us(104 us):  open_hca: SCIF init ERR for mlx4_0

Any other ideas?

 

 

0 Kudos
Copy link
Artem_R_Intel1
Employee
‎10-08-2015 06:21 AM
735 Views

Hi Dmitry,

Could you please provide output of the following commands from host and MIC card:
ibv_devinfo
rpm -qa | grep dapl
rpm -qa | grep libibscif

Also please make sure that all the necessary services are running (mpss, openibd, ofed-mic, mpxyd) on the host.

Could you also please share /var/log/mpxyd.log from the problem node?

0 Kudos
Copy link
Dmitry_Y_
Beginner
‎10-08-2015 07:07 AM
735 Views

Hi Artem,

node$ ibv_devinfo
hca_id:    scif0
    transport:            invalid transport (-1)
    fw_ver:                0.0.1
    node_guid:            4c79:baff:fe63:0481
    sys_image_guid:            4c79:baff:fe63:0481
    vendor_id:            0x8086
    vendor_part_id:            0
    hw_ver:                0x1
    phys_port_cnt:            1
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            1
            port_lid:        1000
            port_lmc:        0x00
            link_layer:        Unknown

hca_id:    mlx4_0
    transport:            InfiniBand (0)
    fw_ver:                2.33.5100
    node_guid:            f452:1403:008a:47c0
    sys_image_guid:            f452:1403:008a:47c3
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x0
    board_id:            MT_1100120019
    phys_port_cnt:            1
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            1467
            port_lid:        389
            port_lmc:        0x00
            link_layer:        InfiniBand

node-mic0# ibv_deviinfo
hca_id:	mlx4_0
	transport:			InfiniBand (0)
	fw_ver:				2.33.5100
	node_guid:			f452:1403:008a:47c0
	sys_image_guid:			f452:1403:008a:47c3
	vendor_id:			0x02c9
	vendor_part_id:			4099
	hw_ver:				0x0
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			1467
			port_lid:		389
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	scif0
	transport:			SCIF (2)
	fw_ver:				0.0.1
	node_guid:			4c79:baff:fe63:0480
	sys_image_guid:			4c79:baff:fe63:0480
	vendor_id:			0x8086
	vendor_part_id:			0
	hw_ver:				0x1
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			1
			port_lid:		1001
			port_lmc:		0x00
			link_layer:		SCIF

 
-- node --
node$ rpm -qa | grep dapl

dapl-utils-2.1.6-1.glibc2.12.x86_64
dapl-devel-2.1.6-1.glibc2.12.x86_64
dapl-2.1.6-1.glibc2.12.x86_64

node$ rpm -qa | grep libibscif
libibscif-1.0.0-1.x86_64

-- node-mic0 --

node-mic0:~$ rpm -qa | grep dapl
dapl-2.1.6-r0.k1om

node-mic0:~$ rpm -qa | grep libibscif
libibscif-1.0.0-r0.k1om

All  (mpss, openibd, ofed-mic, mpxyd) are running

/var/log/mpxyd.log from problem node: 


 ef6c8740:106856290: mpxy_log_options: log level 0x0
 ef6c8740:106856304: mpxy_log_options: lock file /var/run/mpxyd.pid
 ef6c8740:106856305: mpxy_log_options: configuration file /etc/mpxyd.conf
 ef6c8740:106856306: mpxy_log_options: SCIF server_port 68
 ef6c8740:106856308: mpxy_log_options: SCIF listen queue length 240
 ef6c8740:106856309: mpxy_log_options: CPU affinity enabled 1
 ef6c8740:106856310: mpxy_log_options: CPU affinity base core_id for MICs 0
 ef6c8740:106856311: mpxy_log_options: OP thread polling disabled
 ef6c8740:106856312: mpxy_log_options: RDMA buffer pool size 32 MB
 ef6c8740:106856312: mpxy_log_options: RDMA buffer segment size 131072
 ef6c8740:106856313: mpxy_log_options: RDMA buffer alignment 64
 ef6c8740:106856314: mpxy_log_options: RDMA Proxy TX queue depth 512
 ef6c8740:106856315: mpxy_log_options: RDMA Proxy RX queue depth 512
 ef6c8740:106856315: mpxy_log_options: RDMA SCIF inline threshold 256
 ef6c8740:106856316: mpxy_log_options: RDMA IB inline threshold 160
 ef6c8740:106856317: mpxy_log_options: RDMA eager completion 1
 ef6c8740:106856317: mpxy_log_options: RDMA proxy-out signal rate 1
 ef6c8740:106856318: mpxy_log_options: RDMA proxy-out/in max WR processing 10
 ef6c8740:106856319: mpxy_log_options: RDMA proxy-out/in max WC processing 10
 ef6c8740:106856320: mpxy_log_options: RDMA proxy-in enabled
 ef6c8740:106856320: mpxy_log_options: RDMA proxy-in signal rate 1
 ef6c8740:106856321: mpxy_log_options: RDMA proxy-in max reads outstanding 40
 ef6c8740:106856322: mpxy_log_options: Maximum message size 16 MB
 ef6c8740:106856322: mpxy_log_options: CM msg queue depth 500
 ef6c8740:106856323: mpxy_log_options: CM msg completion signal rate 100
 ef6c8740:106856324: mpxy_log_options: CM msg reply timeout 4000 ms
 ef6c8740:106856324: mpxy_log_options: CM msg rtu timeout 2000 ms
 ef6c8740:106856325: mpxy_log_options: CM msg retry count 10
 ef6c8740:106856326: mpxy_log_options: Performance Profiling == 0
 ef6c8740:106856327: mpxy_log_options: MPXYD log device counters (print at close) == 0
 ef6c8740:106856327: main: CCL Proxy - SCIF/IB DAPL RDMA Proxy Service, Mix Version 5 (Build-20151001) v2
 ef6c8740:106856481: main: Starting server
 ef6c8740:106856497: mpxy_server: Server started, cpu_id=20
 ef6c8740:29314076: mix_open_device:  New MIC device - /sys/class/mic/mic0/device, numa_node 0, cpu 0 - 0-13
 ee0bd700:29314189: mpxy_tx_thread: TX thread (ee0bd700) MIC node_id 1 bound to numa_node 0 and cpu_id=0
 ed8bc700:29314197: mpxy_cm_thread: CM thread (ed8bc700) MIC node_id 1 bound to numa_node 0 and cpu_id=2
 ee8be700:29314193: mpxy_op_thread: OP thread (ee8be700) MIC node_id 1 bound to numa_node 0 and cpu_id=1
 ed0bb700:29314209: mpxy_rx_thread: RX thread (ed0bb700) MIC node_id 1 bound to numa_node 0 and cpu_id=3
 ef6c8740:29314499: open_ib_device: System GUID == c047:8a00:0314:52f4
 ef6c8740:29314505: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:29314520: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29317151: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:29317174: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29319119: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:29319142: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29320376: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:29320398: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29322178: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:29322185: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29324449: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:29324461: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29326694: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:29326705: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29328905: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:29328920: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29330798: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:29330812: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29332305: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:29332310: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29334224: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:29334230: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29335868: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:29335874: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29337864: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:29337876: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29339948: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:29339971: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29342116: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:29342124: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29343833: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:29343846: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29345999: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:29346005: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29348599: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:29348622: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29350210: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:29350232: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29352090: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:29352104: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29354675: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:29354689: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29357438: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:29357450: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29361506: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:29361517: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29363928: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:29363939: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:29469050: mcm_init_cm_service:  IB LID 185 (0) PORT 1 GID fe80::f452:1403:8a:47c1 QPN 95: mic0 -> mlx4_0 - MXS, mic_ver 5 
 ef6c8740:59716146: create_smd_bpool:  scif_register rx_addr=0x7faccdff9000,41943040 failed Transport endpoint is not connected
 ef6c8740:59719103: mix_open_device:  ERR: mix_open_device failed for mlx4_0 - 1
 ef6c8740:59719111: scif_send_msg: ERR: scif_send - ep 79, Connection reset by peer, (len 118 != ret -1) msg(0x7ffff499ff20 0x7ffff499ff20)
 ef6c8740:59719113: mix_open_device:  ERR: scif_send dev_id 79 op_ep 5714, closing device (nil)
 ef6c8740:59719120: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 79 82 80 
 ef6c8740:90368834: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:90368858: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90371390: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:90371405: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90373476: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:90373490: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90375803: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:90375823: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90377954: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:90377962: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90379917: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:90379932: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90382231: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:90382239: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90384645: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:90384658: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90387300: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:90387312: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90389343: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:90389354: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90391872: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:90391893: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90394828: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:90394836: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90398041: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:90398049: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90399891: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:90399914: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90402992: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:90403006: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90405408: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:90405422: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90408439: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:90408462: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90411047: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:90411058: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90413738: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:90413758: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90416101: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:90416115: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90418116: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:90418124: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90421404: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:90421426: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90426225: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:90426247: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:90428738: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:90428749: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:128221980: create_smd_bpool:  scif_register addr=0x7faccb7f7000,41943040 failed Transport endpoint is not connected
 ef6c8740:128223564: mix_open_device:  ERR: mix_open_device failed for mlx4_0 - 1
 ef6c8740:128223569: scif_send_msg: ERR: scif_send - ep 83, Connection reset by peer, (len 118 != ret -1) msg(0x7ffff499ff20 0x7ffff499ff20)
 ef6c8740:128223571: mix_open_device:  ERR: scif_send dev_id 83 op_ep 5780, closing device (nil)
 ef6c8740:128223579: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 83 86 85 
 ef6c8740:216463824: mix_open_device:  New MIC device - /sys/class/mic/mic1/device, numa_node 1, cpu 14 - 14-27
 e77fe700:216463884: mpxy_tx_thread: TX thread (e77fe700) MIC node_id 2 bound to numa_node 1 and cpu_id=14
 e67fc700:216463916: mpxy_rx_thread: RX thread (e67fc700) MIC node_id 2 bound to numa_node 1 and cpu_id=15
 e7fff700:216463939: mpxy_op_thread: OP thread (e7fff700) MIC node_id 2 bound to numa_node 1 and cpu_id=16
 e6ffd700:216463951: mpxy_cm_thread: CM thread (e6ffd700) MIC node_id 2 bound to numa_node 1 and cpu_id=17
 ef6c8740:216470744: mcm_init_cm_service:  IB LID 185 (0) PORT 1 GID fe80::f452:1403:8a:47c1 QPN f2: mic1 -> mlx4_0 - MSS, mic_ver 5 
 ef6c8740:221761625: mix_open_device:  New MIC device - /sys/class/mic/mic2/device, numa_node 1, cpu 18 - 14-27
 e57fa700:221761693: mpxy_tx_thread: TX thread (e57fa700) MIC node_id 3 bound to numa_node 1 and cpu_id=18
 e4ff9700:221761721: mpxy_cm_thread: CM thread (e4ff9700) MIC node_id 3 bound to numa_node 1 and cpu_id=19
 e5ffb700:221761731: mpxy_op_thread: OP thread (e5ffb700) MIC node_id 3 bound to numa_node 1 and cpu_id=20
 caffb700:221761760: mpxy_rx_thread: RX thread (caffb700) MIC node_id 3 bound to numa_node 1 and cpu_id=21
 ef6c8740:221767476: mcm_init_cm_service:  IB LID 185 (0) PORT 1 GID fe80::f452:1403:8a:47c1 QPN f7: mic2 -> mlx4_0 - MSS, mic_ver 5 
 ef6c8740:1213585: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:1213619: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1215787: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:1215794: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1218267: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:1218282: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1219539: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:1219547: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1222091: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:1222102: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1224584: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:1224605: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1227083: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:1227095: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1229192: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:1229207: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1231362: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:1231373: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1233428: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:1233449: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1234943: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:1234955: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1237281: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:1237287: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1239067: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:1239081: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1241302: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:1241314: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1243699: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:1243712: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1246177: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:1246184: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1248448: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:1248458: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1249723: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:1249733: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1251398: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:1251405: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1253970: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:1253982: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1256387: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:1256399: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1258861: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:1258876: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1261248: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:1261256: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:1264954: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:1264960: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:41713587: create_smd_bpool:  scif_register rx_addr=0x7facb1ff9000,41943040 failed Transport endpoint is not connected
 ef6c8740:41716748: mix_open_device:  ERR: mix_open_device failed for mlx4_0 - 1
 ef6c8740:41716756: scif_send_msg: ERR: scif_send - ep 85, Connection reset by peer, (len 118 != ret -1) msg(0x7ffff499ff20 0x7ffff499ff20)
 ef6c8740:41716758: mix_open_device:  ERR: scif_send dev_id 85 op_ep 10693, closing device (nil)
 ef6c8740:41716765: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 85 87 86 
 ef6c8740:98692293: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:98692324: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98695494: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:98695503: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98698818: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:98698841: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98702757: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:98702766: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98708322: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:98708345: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98711243: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:98711256: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98712855: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:98712865: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98715003: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:98715013: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98717403: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:98717414: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98719535: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:98719547: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98721295: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:98721306: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:98723269: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:98723279: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166940301: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:166940328: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166943990: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:166944001: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166947571: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:166947580: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166950527: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:166950549: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166978356: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:166978380: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166982407: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:166982416: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166986025: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:166986046: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:166989606: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:166989614: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:167046725: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:167046748: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:167051412: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:167051422: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:167055793: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:167055803: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:167059174: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:167059195: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62154504: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:62154530: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62157237: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:62157260: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62160980: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:62160987: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62164401: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:62164413: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62178787: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:62178798: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62182555: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:62182566: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62186812: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:62186833: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62189996: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:62190007: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62192108: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:62192130: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62194957: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:62194964: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62197960: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:62197970: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:62200463: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:62200483: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38168889: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:38168909: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38171233: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:38171259: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38173128: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:38173139: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38175489: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:38175499: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38178043: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:38178051: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38180713: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:38180720: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38182634: mix_open_device:  ERR: mix_open_device failed for qib0 - 1
 ef6c8740:38182641: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38185090: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:38185101: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38187008: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:38187018: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38189289: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:38189300: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38191373: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:38191383: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38193115: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:38193121: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38194811: mix_open_device:  ERR: mix_open_device failed for qib0 - 2
 ef6c8740:38194817: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38196502: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:38196508: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38198214: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:38198221: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38199941: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:38199951: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38202057: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:38202064: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38204227: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:38204233: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38206151: mix_open_device:  ERR: mix_open_device failed for qib1 - 1
 ef6c8740:38206157: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38208113: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:38208121: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38209817: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:38209823: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38214247: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:38214254: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38216619: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:38216625: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:38220887: mix_open_device:  ERR: mix_open_device failed for qib1 - 2
 ef6c8740:38220898: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 70 72 71 
 ef6c8740:69604707: create_smd_bpool:  scif_register rx_addr=0x7facb1ff9000,41943040 failed Transport endpoint is not connected
 ef6c8740:69607583: mix_open_device:  ERR: mix_open_device failed for mlx4_0 - 1
 ef6c8740:69607593: scif_send_msg: ERR: scif_send - ep 85, Connection reset by peer, (len 118 != ret -1) msg(0x7ffff499ff20 0x7ffff499ff20)
 ef6c8740:69607595: mix_open_device:  ERR: scif_send dev_id 85 op_ep 12369, closing device (nil)
 ef6c8740:69607602: mix_scif_accept:  ERR: open_device -> closing SCIF client EPs 85 88 87

 

0 Kudos
Copy link
Dmitry_Y_
Beginner
‎10-12-2015 03:00 AM
735 Views

Hi everyone,

Please help. 

0 Kudos
Copy link
Artem_R_Intel1
Employee
‎10-12-2015 05:16 AM
735 Views

Hi Dmitry,

Could you please specify your OS and Linux* kernel version? Could you also please provide more details how you installed OFED? 

0 Kudos
Copy link
Dmitry_Y_
Beginner
‎10-12-2015 05:48 AM
735 Views

OS: Scientific Linux 7.1, kernel 3.10.0-229.el7.x86_64

OFED 3.18 has been installed by using install.pl with this config file:

#cat ofed.conf
compat-rdma=y
compat-rdma-devel=y
libibverbs=y
libibverbs-devel=y
libibverbs-devel-static=y
libibverbs-utils=y
libibverbs-debuginfo=y
libmthca=n
libmthca-devel-static=n
libmthca-debuginfo=n
libmlx4=y
libmlx4-devel=y
libmlx4-debuginfo=n
libmlx5=n
libmlx5-devel=n
libmlx5-debuginfo=n
libcxgb3=n
libcxgb3-devel=n
libcxgb3-debuginfo=n
libcxgb4=n
libcxgb4-devel=n
libcxgb4-debuginfo=n
libnes=n
libnes-devel-static=n
libnes-debuginfo=n
libocrdma=n
libocrdma-devel=n
libocrdma-debuginfo=n
libipathverbs=y
libipathverbs-devel=y
libipathverbs-debuginfo=y
libibcm=y
libibcm-devel=y
libibcm-debuginfo=y
libibumad=y
libibumad-devel=y
libibumad-static=y
libibumad-debuginfo=y
libibmad=y
libibmad-devel=y
libibmad-static=y
libibmad-debuginfo=y
ibsim=y
ibsim-debuginfo=y
ibacm=y
librdmacm=y
librdmacm-utils=y
librdmacm-devel=y
librdmacm-debuginfo=n
opensm=y
opensm-libs=y
opensm-devel=y
opensm-debuginfo=y
opensm-static=y
dapl=n
dapl-devel=n
dapl-devel-static=n
dapl-utils=n
dapl-debuginfo=n
perftest=n
mstflint=n
libiwpm=n
srptools=n
rds-tools=n
rds-devel=n
ibutils=y
infiniband-diags=y
qperf=n
qperf-debuginfo=n
ofed-docs=n
ofed-scripts=y
libfabric=n
libfabric-devel=n
libfabric-debuginfo=n
fabtests=n
fabtests-debuginfo=n
ibpd=y
libibscif=y
intel-mic-psm=y
intel-mic-psm-devel=n
core=y
mthca=n
mlx4=y
mlx4_en=y
mlx5=n
cxgb3=n
cxgb4=n
nes=n
qib=n
ocrdma=n
ipoib=y
srp=n
iser=n
nfsrdma=n
ibscif=y
ibp-server=y
ibp-debug=n

 

0 Kudos
Copy link
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.