#/opt/intel/impi/4.0.2/bin64/mpiexec -machinefile ./nodes -env I_MPI_DEBUG 100 -n 24 ./xhpl > debuginfo.txt 2>&1 [1] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [1] my_dlopen(): trying to dlopen: libdat.so [0] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [0] my_dlopen(): trying to dlopen: libdat.so [2] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [2] my_dlopen(): trying to dlopen: libdat.so [11] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [7] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [7] my_dlopen(): trying to dlopen: libdat.so [11] my_dlopen(): trying to dlopen: libdat.so [5] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [10] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [10] my_dlopen(): trying to dlopen: libdat.so [5] my_dlopen(): trying to dlopen: libdat.so [4] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [4] my_dlopen(): trying to dlopen: libdat.so [8] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [8] my_dlopen(): trying to dlopen: libdat.so [9] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [9] my_dlopen(): trying to dlopen: libdat.so [3] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [3] my_dlopen(): trying to dlopen: libdat.so [19] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [19] my_dlopen(): trying to dlopen: libdat.so [12] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [12] my_dlopen(): trying to dlopen: libdat.so [16] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [16] my_dlopen(): trying to dlopen: libdat.so [15] I_MPI_init_dat_regestry_info(): [18] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [18] my_dlopen(): trying to dlopen: libdat.so trying to load dat library libdat.so [15] my_dlopen(): [17] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [17] my_dlopen(): trying to dlopen: libdat.so trying to dlopen: libdat.so [6] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [6] my_dlopen(): trying to dlopen: libdat.so [13] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [13] my_dlopen(): trying to dlopen: libdat.so [20] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [20] my_dlopen(): trying to dlopen: libdat.so [21] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [21] my_dlopen(): trying to dlopen: libdat.so [14] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [14] my_dlopen(): trying to dlopen: libdat.so [22] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [22] my_dlopen(): trying to dlopen: libdat.so [23] I_MPI_init_dat_regestry_info(): trying to load dat library libdat.so [23] my_dlopen(): trying to dlopen: libdat.so [2] I_MPI_dlopen_dat(): trying to load dat library libdat.so [2] my_dlopen(): trying to dlopen: libdat.so [1] I_MPI_dlopen_dat(): trying to load dat library libdat.so [1] my_dlopen(): trying to dlopen: libdat.so [10] I_MPI_dlopen_dat(): trying to load dat library libdat.so [10] my_dlopen(): trying to dlopen: libdat.so [18] I_MPI_dlopen_dat(): trying to load dat library libdat.so [18] my_dlopen(): trying to dlopen: libdat.so [7] I_MPI_dlopen_dat(): trying to load dat library libdat.so [7] my_dlopen(): trying to dlopen: libdat.so [15] I_MPI_dlopen_dat(): trying to load dat library libdat.so [15] my_dlopen(): [11] I_MPI_dlopen_dat(): trying to load dat library libdat.so [11] my_dlopen(): trying to dlopen: libdat.so trying to dlopen: libdat.so [21] I_MPI_dlopen_dat(): trying to load dat library libdat.so [21] my_dlopen(): trying to dlopen: libdat.so [5] I_MPI_dlopen_dat(): trying to load dat library libdat.so[23] I_MPI_dlopen_dat(): trying to load dat library libdat.so [23] my_dlopen(): trying to dlopen: libdat.so [5] my_dlopen(): trying to dlopen: libdat.so [12] I_MPI_dlopen_dat(): trying to load dat library libdat.so [12] my_dlopen(): trying to dlopen: libdat.so [3] I_MPI_dlopen_dat(): trying to load dat library libdat.so [3] my_dlopen(): trying to dlopen: libdat.so [17] I_MPI_dlopen_dat(): trying to load dat library libdat.so [17] my_dlopen(): trying to dlopen: libdat.so [0] I_MPI_dlopen_dat(): trying to load dat library libdat.so [0] my_dlopen(): trying to dlopen: libdat.so [14] I_MPI_dlopen_dat(): trying to load dat library libdat.so [14] my_dlopen(): trying to dlopen: libdat.so [9] I_MPI_dlopen_dat(): trying to load dat library libdat.so [9] my_dlopen(): trying to dlopen: libdat.so [13] I_MPI_dlopen_dat(): trying to load dat library libdat.so [13] my_dlopen(): trying to dlopen: libdat.so [6] I_MPI_dlopen_dat(): trying to load dat library libdat.so [6] my_dlopen(): trying to dlopen: libdat.so [19] I_MPI_dlopen_dat(): trying to load dat library libdat.so [19] my_dlopen(): trying to dlopen: libdat.so [8] I_MPI_dlopen_dat(): trying to load dat library libdat.so [8] my_dlopen(): trying to dlopen: libdat.so [16] I_MPI_dlopen_dat(): trying to load dat library libdat.so [16] my_dlopen(): trying to dlopen: libdat.so [20] I_MPI_dlopen_dat(): trying to load dat library libdat.so [20] my_dlopen(): trying to dlopen: libdat.so [4] I_MPI_dlopen_dat(): trying to load dat library libdat.so [4] my_dlopen(): trying to dlopen: libdat.so [22] I_MPI_dlopen_dat(): trying to load dat library libdat.so [22] my_dlopen(): trying to dlopen: libdat.so c07b03:26624: open_hca: rdma_create_id ERR Invalid argument [2] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [2] I_MPI_dlopen_dat(): trying to load dat library libdat.so [2] my_dlopen(): trying to dlopen: libdat.so c07b03:26633: open_hca: rdma_create_id ERR Invalid argument [11] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [11] I_MPI_dlopen_dat(): trying to load dat library libdat.so [11] my_dlopen(): trying to dlopen: libdat.so c07b03:26623: open_hca: rdma_create_id ERR Invalid argument [1] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [1] I_MPI_dlopen_dat(): trying to load dat library libdat.so [1] my_dlopen(): trying to dlopen: libdat.so c07b03:26627: c07b03:26629: open_hca: rdma_create_id ERR Invalid argument [7] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [7] I_MPI_dlopen_dat(): trying to load dat library libdat.so [7] my_dlopen(): trying to dlopen: libdat.so open_hca: rdma_create_id ERR Invalid argument [5] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [5] I_MPI_dlopen_dat(): trying to load dat library libdat.so [5] my_dlopen(): trying to dlopen: libdat.so c07b03:26625: open_hca: rdma_create_id ERR Invalid argument [3] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [3] I_MPI_dlopen_dat(): trying to load dat library libdat.so [3] my_dlopen(): trying to dlopen: libdat.so c07b03:26634: open_hca: rdma_create_id ERR Invalid argument [0] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [0] I_MPI_dlopen_dat(): trying to load dat library libdat.so [0] my_dlopen(): trying to dlopen: libdat.so c07b03:26626: open_hca: rdma_create_id ERR Invalid argument [4] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [4] I_MPI_dlopen_dat(): trying to load dat library libdat.so [4] my_dlopen(): c07b03:26630: open_hca: rdma_create_id ERR Invalid argument [8] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR )trying to dlopen: libdat.so [8] I_MPI_dlopen_dat(): trying to load dat library libdat.so [8] my_dlopen(): trying to dlopen: libdat.so c07b04:7700: open_hca: rdma_create_id ERR Invalid argument [12] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [12] I_MPI_dlopen_dat(): trying to load dat library libdat.so [12] my_dlopen(): trying to dlopen: libdat.so c07b04:7706: open_hca: rdma_create_id ERR Invalid argument [18] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [18] I_MPI_dlopen_dat(): trying to load dat library libdat.so [18] my_dlopen(): trying to dlopen: libdat.so c07b04:7705: open_hca: rdma_create_id ERR Invalid argument [17] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [17] I_MPI_dlopen_dat(): trying to load dat library libdat.so [17] my_dlopen(): trying to dlopen: libdat.so c07b04:7711: open_hca: rdma_create_id ERR Invalid argument [23] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [23] I_MPI_dlopen_dat(): trying to load dat library libdat.so [23] my_dlopen(): trying to dlopen: libdat.so c07b04:7707: open_hca: rdma_create_id ERR Invalid argument [19] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [19] I_MPI_dlopen_dat(): trying to load dat library libdat.so [19] my_dlopen(): trying to dlopen: libdat.so c07b03:26631: open_hca: rdma_create_id ERR Invalid argument [9] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [9] I_MPI_dlopen_dat(): trying to load dat library libdat.so [9] my_dlopen(): trying to dlopen: libdat.soc07b04:7704: open_hca: rdma_create_id ERR Invalid argument [16] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [16] I_MPI_dlopen_dat(): trying to load dat library libdat.so [16] my_dlopen(): trying to dlopen: libdat.so c07b03:26633: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [11] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [11] I_MPI_dlopen_dat(): trying to load dat library libdat.so [11] my_dlopen(): trying to dlopen: libdat.so c07b03:26633: open_hca: device mthca0 not found [11] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [11] I_MPI_dlopen_dat(): trying to load dat library libdat.so [11] my_dlopen(): trying to dlopen: libdat.so c07b03:26633: open_hca: device mthca0 not found [11] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [11] I_MPI_dlopen_dat(): trying to load dat library libdat.so [11] my_dlopen(): trying to dlopen: libdat.so c07b03:26623: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [1] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [1] I_MPI_dlopen_dat(): trying to load dat library libdat.so [1] my_dlopen(): trying to dlopen: libdat.so c07b03:26623: open_hca: device mthca0 not found [1] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [1] I_MPI_dlopen_dat(): trying to load dat library libdat.so [1] my_dlopen(): trying to dlopen: libdat.so c07b03:26623: open_hca: device mthca0 not found [1] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [1] I_MPI_dlopen_dat(): trying to load dat library libdat.so [1] my_dlopen(): trying to dlopen: libdat.so c07b03:26629: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [7] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [7] I_MPI_dlopen_dat(): trying to load dat library libdat.so [7] my_dlopen(): trying to dlopen: libdat.so c07b03:26625: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [3] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [3] I_MPI_dlopen_dat(): trying to load dat library libdat.so [3] my_dlopen(): trying to dlopen: libdat.so c07b03:26629: open_hca: device mthca0 not found [7] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [7] I_MPI_dlopen_dat(): trying to load dat library libdat.so [7] my_dlopen(): trying to dlopen: libdat.so c07b03:26629: open_hca: device mthca0 not found [7] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [7] I_MPI_dlopen_dat(): trying to load dat library libdat.so [7] my_dlopen(): trying to dlopen: libdat.so c07b03:26625: open_hca: device mthca0 not found [3] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [3] I_MPI_dlopen_dat(): trying to load dat library libdat.so [3] my_dlopen(): trying to dlopen: libdat.so c07b03:26625: open_hca: device mthca0 not found [3] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [3] I_MPI_dlopen_dat(): trying to load dat library libdat.so [3] my_dlopen(): trying to dlopen: libdat.so c07b03:26634: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [0] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [0] I_MPI_dlopen_dat(): trying to load dat library libdat.so [0] my_dlopen(): trying to dlopen: libdat.so c07b03:26627: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [5] MPIDI_CH3I_RDMA_can_use_dapl_provider(): c07b03:26634: open_hca: device mthca0 not found [0] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [0] I_MPI_dlopen_dat(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS )trying to load dat library libdat.so [0] my_dlopen(): trying to dlopen: libdat.so c07b03:26634: open_hca: device mthca0 not found [0] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [0] I_MPI_dlopen_dat(): trying to load dat library libdat.so [0] my_dlopen(): trying to dlopen: libdat.so [5] I_MPI_dlopen_dat(): trying to load dat library libdat.so [5] my_dlopen(): trying to dlopen: libdat.so [11] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b03:26627: open_hca: device mthca0 not found [5] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [5] I_MPI_dlopen_dat(): trying to load dat library libdat.so [5] my_dlopen(): trying to dlopen: libdat.so c07b04:7700: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [12] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [12] I_MPI_dlopen_dat(): trying to load dat library libdat.so [12] my_dlopen(): trying to dlopen: libdat.so c07b03:26627: open_hca: device mthca0 not found [5] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [5] I_MPI_dlopen_dat(): trying to load dat library libdat.so c07b04:7700: open_hca: device mthca0 not found [12] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [12] I_MPI_dlopen_dat(): trying to load dat library libdat.so [12] my_dlopen(): trying to dlopen: libdat.so [5] my_dlopen(): trying to dlopen: libdat.so c07b04:7711: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [23] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [23] I_MPI_dlopen_dat(): trying to load dat library libdat.so [23] my_dlopen(): trying to dlopen: libdat.so c07b04:7700: open_hca: device mthca0 not found [12] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [12] I_MPI_dlopen_dat(): trying to load dat library libdat.so [12] my_dlopen(): trying to dlopen: libdat.so c07b04:7711: open_hca: device mthca0 not found [23] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [23] I_MPI_dlopen_dat(): trying to load dat library libdat.so [23] my_dlopen(): trying to dlopen: libdat.so c07b04:7711: open_hca: device mthca0 not found [23] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [23] I_MPI_dlopen_dat(): trying to load dat library libdat.so [23] my_dlopen(): trying to dlopen: libdat.so [0] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [12] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b03:26624: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [2] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [2] I_MPI_dlopen_dat(): trying to load dat library libdat.so [2] my_dlopen(): trying to dlopen: libdat.so c07b03:26624: open_hca: device mthca0 not found [2] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [2] I_MPI_dlopen_dat(): trying to load dat library libdat.so [2] my_dlopen(): trying to dlopen: libdat.so c07b03:26624: open_hca: device mthca0 not found [2] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [2] I_MPI_dlopen_dat(): trying to load dat library libdat.so [2] my_dlopen(): trying to dlopen: libdat.so [5] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [23] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [2] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b03:26626: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [4] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [4] I_MPI_dlopen_dat(): trying to load dat library libdat.so [4] my_dlopen(): trying to dlopen: libdat.so c07b03:26630: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [8] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [8] I_MPI_dlopen_dat(): trying to load dat library libdat.so [8] my_dlopen(): trying to dlopen: libdat.so c07b03:26626: open_hca: device mthca0 not found [4] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [4] I_MPI_dlopen_dat(): trying to load dat library libdat.so [4] my_dlopen(): trying to dlopen: libdat.so c07b03:26626: c07b03:26630: open_hca: device mthca0 not found [8] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [8] I_MPI_dlopen_dat(): trying to load dat library libdat.so [8] my_dlopen(): trying to dlopen: libdat.so c07b03:26630: open_hca: device mthca0 not found [8] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [8] I_MPI_dlopen_dat(): trying to load dat library libdat.so [8] my_dlopen(): trying to dlopen: libdat.so open_hca: device mthca0 not found [4] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [4] I_MPI_dlopen_dat(): trying to load dat library libdat.so [4] my_dlopen(): trying to dlopen: libdat.so [1] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7706: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [18] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [18] I_MPI_dlopen_dat(): trying to load dat library libdat.so [18] my_dlopen(): trying to dlopen: libdat.so c07b03:26631: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [9] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [9] I_MPI_dlopen_dat(): trying to load dat library libdat.so [9] my_dlopen(): trying to dlopen: libdat.so [3] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b03:26631: open_hca: device mthca0 not found [9] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [9] I_MPI_dlopen_dat(): trying to load dat library libdat.so [9] my_dlopen(): trying to dlopen: libdat.so c07b03:26631: open_hca: device mthca0 not found c07b04:7706: open_hca: device mthca0 not found [18] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [9] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [18] I_MPI_dlopen_dat(): trying to load dat library libdat.so [18] my_dlopen(): trying to dlopen: libdat.so c07b04:7706: open_hca: device mthca0 not found [18] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [18] I_MPI_dlopen_dat(): trying to load dat library libdat.so [18] my_dlopen(): trying to dlopen: libdat.so [9] I_MPI_dlopen_dat(): trying to load dat library libdat.so [9] my_dlopen(): trying to dlopen: libdat.so [7] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [8] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7705: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [17] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [17] I_MPI_dlopen_dat(): trying to load dat library libdat.so [17] my_dlopen(): trying to dlopen: libdat.so [4] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7705: open_hca: device mthca0 not found [17] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [17] I_MPI_dlopen_dat(): trying to load dat library libdat.so [17] my_dlopen(): trying to dlopen: libdat.so c07b04:7705: open_hca: device mthca0 not found [17] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [17] I_MPI_dlopen_dat(): trying to load dat library libdat.so [17] my_dlopen(): trying to dlopen: libdat.so c07b04:7704: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [16] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [16] I_MPI_dlopen_dat(): trying to load dat library libdat.so [16] my_dlopen(): trying to dlopen: libdat.so c07b04:7704: open_hca: device mthca0 not found [16] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [16] I_MPI_dlopen_dat(): trying to load dat library libdat.so [16] my_dlopen(): trying to dlopen: libdat.so c07b04:7704: open_hca: device mthca0 not found [16] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [16] I_MPI_dlopen_dat(): trying to load dat library libdat.so [16] my_dlopen(): trying to dlopen: libdat.so c07b04:7707: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [19] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [19] I_MPI_dlopen_dat(): trying to load dat library libdat.so [19] my_dlopen(): trying to dlopen: libdat.so c07b04:7707: open_hca: device mthca0 not found [19] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [19] I_MPI_dlopen_dat(): trying to load dat library libdat.so [19] my_dlopen(): trying to dlopen: libdat.so c07b04:7707: open_hca: device mthca0 not found [19] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [19] I_MPI_dlopen_dat(): trying to load dat library libdat.so [19] my_dlopen(): trying to dlopen: libdat.so [9] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [19] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7708: open_hca: rdma_create_id ERR Invalid argument [20] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [20] I_MPI_dlopen_dat(): trying to load dat library libdat.so [20] my_dlopen(): trying to dlopen: libdat.so c07b04:7708: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [20] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [20] I_MPI_dlopen_dat(): trying to load dat library libdat.so [20] my_dlopen(): trying to dlopen: libdat.so c07b04:7708: open_hca: device mthca0 not found [20] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [20] I_MPI_dlopen_dat(): trying to load dat library libdat.so [20] my_dlopen(): trying to dlopen: libdat.so c07b04:7708: open_hca: device mthca0 not found [20] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [20] I_MPI_dlopen_dat(): trying to load dat library libdat.so [20] my_dlopen(): trying to dlopen: libdat.so [20] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [18] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [16] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [17] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7703: open_hca: rdma_create_id ERR Invalid argument [15] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [15] I_MPI_dlopen_dat(): trying to load dat library libdat.so [15] my_dlopen(): trying to dlopen: libdat.so c07b04:7703: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [15] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [15] I_MPI_dlopen_dat(): trying to load dat library libdat.so [15] my_dlopen(): trying to dlopen: libdat.so c07b04:7703: open_hca: device mthca0 not found [15] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [15] I_MPI_dlopen_dat(): trying to load dat library libdat.so [15] my_dlopen(): trying to dlopen: libdat.so c07b04:7703: open_hca: device mthca0 not found [15] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [15] I_MPI_dlopen_dat(): trying to load dat library libdat.so [15] my_dlopen(): trying to dlopen: libdat.so [15] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b03:26628: open_hca: rdma_create_id ERR Invalid argument [6] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [6] I_MPI_dlopen_dat(): trying to load dat library libdat.so [6] my_dlopen(): trying to dlopen: libdat.so c07b03:26628: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [6] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [6] I_MPI_dlopen_dat(): trying to load dat library libdat.so [6] my_dlopen(): trying to dlopen: libdat.so c07b03:26628: open_hca: device mthca0 not found [6] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [6] I_MPI_dlopen_dat(): trying to load dat library libdat.so [6] my_dlopen(): trying to dlopen: libdat.so c07b03:26628: open_hca: device mthca0 not found [6] MPIDI_CH3I_RDMA_can_use_dapl_c07b04:7701: open_hca: rdma_create_id ERR Invalid argument [13] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [13] I_MPI_dlopen_dat(): trying to load dat library libdat.so [13] my_dlopen(): trying to dlopen: libdat.so c07b04:7701: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [13] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [13] I_MPI_dlopen_dat(): trying to load dat library libdat.so [13] my_dlopen(): trying to dlopen: libdat.so c07b04:7701: open_hca: device mthca0 not found [13] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [13] I_MPI_dlopen_dat(): trying to load dat library libdat.so [13] my_dlopen(): trying to dlopen: libdat.so c07b04:7701: open_hca: device mthca0 not found [13] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [13] I_MPI_dlopen_dat(): trying to load dat library libdat.so [13] my_dlopen(): trying to dlopen: libdat.so [13] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7702: open_hca: rdma_create_id ERR Invalid argument [14] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [14] I_MPI_dlopen_dat(): trying to load dat library libdat.so [14] my_dlopen(): trying to dlopen: libdat.so c07b04:7702: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [14] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [14] I_MPI_dlopen_dat(): trying to load dat library libdat.so [14] my_dlopen(): trying to dlopen: libdat.so c07b04:7702: open_hca: device mthca0 not found [14] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [14] I_MPI_dlopen_dat(): trying to load dat library libdat.so [14] my_dlopen(): trying to dlopen: libdat.so c07b04:7702: open_hca: device mthca0 not found [14] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [14] I_MPI_dlopen_dat(): trying to load dat library libdat.so [14] my_dlopen(): trying to dlopen: libdat.so [14] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [6] I_MPI_dlopen_dat(): trying to load dat library libdat.so [6] my_dlopen(): trying to dlopen: libdat.so [6] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b03:26632: open_hca: rdma_create_id ERR Invalid argument [10] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [10] I_MPI_dlopen_dat(): trying to load dat library libdat.so [10] my_dlopen(): trying to dlopen: libdat.so c07b03:26632: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [10] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [10] I_MPI_dlopen_dat(): trying to load dat library libdat.so [10] my_dlopen(): trying to dlopen: libdat.so c07b03:26632: open_hca: device mthca0 not found [10] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [10] I_MPI_dlopen_dat(): trying to load dat library libdat.so [10] my_dlopen(): trying to dlopen: libdat.so c07b03:26632: open_hca: device mthca0 not found [10] MPIDI_CH3I_RDMA_canc07b04:7709: open_hca: rdma_create_id ERR Invalid argument [21] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [21] I_MPI_dlopen_dat(): trying to load dat library libdat.so [21] my_dlopen(): trying to dlopen: libdat.so c07b04:7709: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [21] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [21] I_MPI_dlopen_dat(): trying to load dat library libdat.so [21] my_dlopen(): trying to dlopen: libdat.so c07b04:7709: open_hca: device mthca0 not found [21] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [21] I_MPI_dlopen_dat(): trying to load dat library libdat.so [21] my_dlopen(): trying to dlopen: libdat.so c07b04:7709: open_hca: device mthca0 not found [21] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [21] I_MPI_dlopen_dat(): trying to load dat library libdat.so [21] my_dlopen(): trying to dlopen: libdat.so [21] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf c07b04:7710: open_hca: rdma_create_id ERR Invalid argument [22] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma (DAT_INTERNAL_ERROR ) [22] I_MPI_dlopen_dat(): trying to load dat library libdat.so [22] my_dlopen(): trying to dlopen: libdat.so c07b04:7710: open_hca: getaddr_netdev ERROR: Connection refused. Is ib1 configured? [22] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-cma-1 (DAT_INVALID_ADDRESS ) [22] I_MPI_dlopen_dat(): trying to load dat library libdat.so [22] my_dlopen(): trying to dlopen: libdat.so c07b04:7710: open_hca: device mthca0 not found [22] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-1 (DAT_INTERNAL_ERROR ) [22] I_MPI_dlopen_dat(): trying to load dat library libdat.so [22] my_dlopen(): trying to dlopen: libdat.so c07b04:7710: open_hca: device mthca0 not found [22] MPIDI_CH3I_RDMA_can_use_dapl_provider(): [ibc07b04] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [22] I_MPI_dlopen_dat(): trying to load dat library libdat.so [22] my_dlopen(): trying to dlopen: libdat.so [22] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf _use_dapl_provider(): [ibc07b03] DAPL provider not ok to use (dat_ia_open failed): OpenIB-mthca0-2 (DAT_INTERNAL_ERROR ) [10] I_MPI_dlopen_dat(): trying to load dat library libdat.so [10] my_dlopen(): trying to dlopen: libdat.so [10] MPI startup(): DAPL provider OpenIB-mlx4_0-1 specified in DAPL configuration file /etc/dat.conf [2] Can use IB extensions %s (): off [2] MPI startup(): will use rendezvous RDMA Read protocol [0] MPI startup(): IA attr adapter_name= [0] MPI startup(): IA attr vendor_name= [0] MPI startup(): IA attr hardware_version=160.0 [0] MPI startup(): IA attr firmware_version=0.0 [0] MPI startup(): IA attr max_eps=261056 [0] MPI startup(): IA attr max_dto_per_ep=16351 [0] MPI startup(): IA attr max_rdma_read_per_ep_in=16 [0] MPI startup(): IA attr max_rdma_read_per_ep_out=128 [0] MPI startup(): IA attr max_evds=65408 [0] MPI startup(): IA attr max_evd_qlen=4194303 [0] MPI startup(): IA attr max_iov_segments_per_dto=32 [0] MPI startup(): IA attr max_lmrs=524272 [0] MPI startup(): IA attr max_lmr_block_size=0xffffffff [0] MPI startup(): IA attr max_lmr_virtual_address=0xffffffffffffffff [0] MPI startup(): IA attr max_pzs=32764 [0] MPI startup(): IA attr max_message_size=0x40000000 [0] MPI startup(): IA attr max_rdma_size=0x40000000 [0] MPI startup(): IA attr max_rmrs=0 [0] MPI startup(): IA attr max_rmr_target_address=0xffffffffffffffff [0] MPI startup(): IA attr max_srqs=0 [0] MPI startup(): IA attr max_ep_per[1] Can use IB extensions %s (): off _srq=0 [0] MPI startup(): IA attr max_recv_per_srq=0 [0] MPI startup(): IA attr max_iov_segments_per_rdma_read=32 [0] MPI startup(): IA attr max_iov_segments_per_rdma_write=32 [0] MPI startup(): IA attr max_rdma_read_in=16 [0] MPI startup(): IA attr max_rdma_read_out=128 [0] MPI startup(): IA attr max_rdma_read_per_ep_in_guaranteed=1 [0] MPI startup(): IA attr max_rdma_read_per_ep_out_guaranteed=1 [0] MPI startup(): IA attr num_transport_attr=0 [0] MPI startup(): IA attr num_vendor_attr=0 [0] MPI startup(): DAPL provider attr provider_name=OpenIB-mlx4_0-1 [0] MPI startup(): DAPL provider attr provider_version=1.0 [0] MPI startup(): DAPL provider attr dapl_version=1.2 [0] MPI startup(): DAPL provider attr lmr_mem_types_supported=0x3 [0] MPI startup(): DAPL provider attr iov_ownership_on_return=0x0 [0] MPI startup(): DAPL provider attr dat_qos_supported=0x0 [0] MPI startup(): DAPL provider attr completion_flags_supported=0x0 [0] MPI startup(): DAPL provider attr is_thread_safe=0 [0] MPI startup(): DAPL provider[1] MPI startup(): will use rendezvous RDMA Read protocol [3] Can use IB extensions %s (): off [3] MPI startup(): will use rendezvous RDMA Read protocol attr max_private_data_size=92 [0] MPI startup(): DAPL provider attr supports_multipath=0 [0] MPI startup(): DAPL provider attr ep_creator=0 [0] MPI startup(): DAPL provider attr pz_support=0 [0] MPI startup(): DAPL provider attr optimal_buffer_alignment=256 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[0][0]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[0][1]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[0][2]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[0][3]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[0][4]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[0][5]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[1][0]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[1][1]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[1][2]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[1][3]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[1][4]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[1][5]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[2][0]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[2][1]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[2][2]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[2][3]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[2][4]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[2][5]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[3][0]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[3][1]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[3][2]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[3][3]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[3][4]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[3][5]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[4][0]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[4][1]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[4][2]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[4][3]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[4][4]=1 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[4][5]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[5][0]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[5][1]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[5][2]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[5][3]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[5][4]=0 [0] MPI startup(): DAPL provider attr evd_stream_merging_supported[5][5]=0 [0] MPI startup(): DAPL provider attr srq_supported=1182404192 [0] MPI startup(): DAPL provider attr srq_watermarks_supported=32767 [0] MPI startup(): DAPL provider attr srq_ep_pz_difference_supported=-1 [0] MPI startup(): DAPL provider attr srq_info_supported=63 [0] MPI startup(): DAPL provider attr ep_recv_info_supported=0 [0] MPI startup(): DAPL provider attr lmr_sync_req=0 [0] MPI startup(): DAPL provider attr dto_async_return_guaranteed=404108448 [0] MPI startup(): DAPL provider attr rdma_write_for_rdma_read_req=0 [0] MPI startup(): DAPL provider attr num_provider_specific_attr=0 [0] Can use IB extensions %s (): off [0] MPI startup(): EP attr service_type=0 [0] MPI startup(): EP attr max_message_size=3 [0] MPI startup(): EP attr max_rdma_size=40000000 [0] MPI startup(): EP attr qos=0x0 [0] MPI startup(): EP attr recv_completion_flags=0x0 [0] MPI startup(): EP attr request_completion_flags=0x4 [0] MPI startup(): EP attr max_recv_dtos=8 [0] MPI startup(): EP attr max_request_dtos=432 [0] MPI startup(): EP attr max_recv_iov=1 [0] MPI startup(): EP attr max_request_iov=1 [0] MPI startup(): EP attr max_rdma_read_in=4 [0] MPI startup(): EP attr max_rdma_read_out=4 [0] MPI startup(): EP attr srq_soft_hw=-1 [0] MPI startup(): EP attr max_rdma_read_iov=1 [0] MPI startup(): EP attr max_rdma_write_iov=1 [0] MPI startup(): EP attr ep_transport_specific_count=0 [0] MPI startup(): EP attr ep_transport_specific_count=0 [0] MPI startup(): rdma_iba_eager_threshold =16456 [0] MPI startup(): rdma_vbuf_threshold =16456 [0] MPI startup(): rdma_real_max_wqe =400 [0] MPI startup(): rdma_use_buf_enlargment =0 [0] MPI startup(): rdma_short_buf_threshold =584 [0] MPI startup(): num_rdma_credit_update =4 [0] MPI startup(): num_rdma_buffer =16 [0] MPI startup(): dapl_lmr_segment_len_off =16 [0] MPI startup(): number_credit_array =6 [0] MPI startup(): rdma_use_exchange_method =-1 [0] MPI startup(): rdma_rndv_buf_align_mask =7f [0] MPI startup(): rdma_rndv_copy_with_align_threshold=131072 [0] MPI startup(): rdma_use_mreg_cache =1 [0] MPI startup(): rdma_default_max_sg_list =1 [0] MPI startup(): rdma_reduced_pkt_size =0 [0] MPI startup(): rdma_vbuf_total_size =16640 [0] MPI startup(): rdma_short_buf_total_size =0 [0] MPI startup(): rtc_max_memory_size =0l [0] MPI startup(): rtc_max_cache_entries =1024 [0] MPI startup(): rdma_max_msg_size =2147483647l [0] MPI startup(): rdma_read_max_num =16 [0] MPI startup(): rdma_default_max_cq_size =2000 [0] MPI startup(): rdma_default_max_wqe =400 [0] MPI startup(): rdma_default_max_recv_wqe =8 [0] MPI startup(): rdma_integer_pool_size =1024 [0] MPI startup(): rdma_read_reserve =32 [0] MPI startup(): rdma_check_max_rdma_attr =0 [0] MPI startup(): rdma_default_conn_evd_qlen=80 [0] MPI startup(): rdma_optimal_buffer_align =256 [0] MPI startup(): rdma_dont_use_rtc_overlapping=0 [0] MPI startup(): rdma_default_recv_evd_qlen =0 [0] MPI startup(): will use rendezvous RDMA Read protocol [6] Can use IB extensions %s (): off [6] MPI startup(): will use rendezvous RDMA Read protocol [4] Can use IB extensions %s (): off [4] MPI startup(): will use rendezvous RDMA Read protocol [5] Can use IB extensions %s (): off [5] MPI startup(): will use rendezvous RDMA Read protocol [8] Can use IB extensions %s (): off [9] Can use IB extensions %s (): off [8] MPI startup(): will use rendezvous RDMA Read protocol [9] MPI startup(): will use rendezvous RDMA Read protocol [7] Can use IB extensions %s (): off [7] MPI startup(): will use rendezvous RDMA Read protocol [12] Can use IB extensions %s (): off [15] Can use IB extensions %s (): off [23] Can use IB extensions %s (): off [13] Can use IB extensions %s (): off [14] Can use IB extensions %s (): off [11] Can use IB extensions %s (): off [15] MPI startup(): will use rendezvous RDMA Read protocol [23] MPI startup(): will use rendezvous RDMA Read protocol [13] MPI startup(): will use rendezvous RDMA Read protocol [11] MPI startup(): will use rendezvous RDMA Read protocol [12] MPI startup(): will use rendezvous RDMA Read protocol [20] Can use IB extensions %s (): off [16] Can use IB extensions %s (): off [19] Can use IB extensions %s (): off [18] Can use IB extensions %s (): off [18] MPI startup(): will use rendezvous RDMA Read protocol [17] Can use IB extensions %s (): off [16] MPI startup(): [17] MPI startup(): will use rendezvous RDMA Read protocol will use rendezvous RDMA Read protocol[19] MPI startup(): will use rendezvous RDMA Read protocol [14] MPI startup(): will use rendezvous RDMA Read protocol [20] MPI startup(): will use rendezvous RDMA Read protocol [21] Can use IB extensions %s (): off [21] MPI startup(): will use rendezvous RDMA Read protocol [22] Can use IB extensions %s (): off [22] MPI startup(): will use rendezvous RDMA Read protocol [10] Can use IB extensions %s (): off [10] MPI startup(): will use rendezvous RDMA Read protocol [0] MPI startup(): RDMA, shared memory, and socket data transfer modes [1] MPI startup(): RDMA, shared memory, and socket data transfer modes [2] MPI startup(): RDMA, shared memory, and socket data transfer modes [3] MPI startup(): RDMA, shared memory, and socket data transfer modes [4] MPI startup(): RDMA, shared memory, and socket data transfer modes [5] MPI startup(): RDMA, shared memory, and socket data transfer modes [6] MPI startup(): RDMA, shared memory, and socket data transfer modes [7] MPI startup(): RDMA, shared memory, and socket data transfer modes [9] MPI startup(): RDMA, shared memory, and socket data transfer modes [8] MPI startup(): RDMA, shared memory, and socket data transfer modes [10] MPI startup(): RDMA, shared memory, and socket data transfer modes [11] MPI startup(): RDMA, shared memory, and socket data transfer modes [12] MPI startup(): RDMA, shared memory, and socket data transfer modes [13] MPI startup(): RDMA, shared memory, and socket data transfer modes [14] MPI startup(): RDMA, shared memory, and socket data transfer modes [15] MPI startup(): RDMA, shared memory, and socket data transfer modes [16] MPI startup(): RDMA, shared memory, and socket data transfer modes [17] MPI startup(): RDMA, shared memory, and socket data transfer modes [18] MPI startup(): RDMA, shared memory, and socket data transfer modes [19] MPI startup(): RDMA, shared memory, and socket data transfer modes [20] MPI startup(): RDMA, shared memory, and socket data transfer modes [21] MPI startup(): RDMA, shared memory, and socket data transfer modes [22] MPI startup(): RDMA, shared memory, and socket data transfer modes [23] MPI startup(): RDMA, shared memory, and socket data transfer modes [11] MPI Startup(): process is pinned to CPU11 on node c07b03 [9] MPI Startup(): process is pinned to CPU09 on node c07b03 [12] MPI Startup(): process is pinned to CPU00 on node c07b04 [21] MPI Startup(): process is pinned to CPU09 on node c07b04 [16] MPI Startup(): process is pinned to CPU04 on node c07b04 [15] MPI Startup(): process is pinned to CPU03 on node c07b04 [10] MPI Startup(): process is pinned to CPU10 on node c07b03 [6] MPI Startup(): process is pinned to CPU06 on node c07b03 [2] MPI Startup(): process is pinned to CPU02 on node c07b03 [13] MPI Startup(): process is pinned to CPU01 on node c07b04 [1] MPI Startup(): process is pinned to CPU01 on node c07b03 [8] MPI Startup(): process is pinned to CPU08 on node c07b03 [17] MPI Startup(): process is pinned to CPU05 on node c07b04 [18] MPI Startup(): process is pinned to CPU06 on node c07b04 [7] MPI Startup(): process is pinned to CPU07 on node c07b03 [4] MPI Startup(): process is pinned to CPU04 on node c07b03 [5] MPI Startup(): process is pinned to CPU05 on node c07b03 [3] MPI Startup(): process is pinned to CPU03 on node c07b03 [23] MPI Startup(): process is pinned to CPU11 on node c07b04 [14] MPI Startup(): process is pinned to CPU02 on node c07b04 [0] MPI startup(): Intel(R) MPI Library, Version 3.2.2 Build 20090827 [0] MPI startup(): Copyright (C) 2003-2009 Intel Corporation. All rights reserved. [0] MPI Startup(): process is pinned to CPU00 on node c07b03 [22] MPI Startup(): process is pinned to CPU10 on node c07b04 [19] MPI Startup(): process is pinned to CPU07 on node c07b04 [20] MPI Startup(): process is pinned to CPU08 on node c07b04 [0] Rank Pid Node name Pin cpu [0] 0 26634 c07b03 0 [0] 1 26623 c07b03 1 [0] 2 26624 c07b03 2 [0] 3 26625 c07b03 3 [0] 4 26626 c07b03 4 [0] 5 26627 c07b03 5 [0] 6 26628 c07b03 6 [0] 7 26629 c07b03 7 [0] 8 26630 c07b03 8 [0] 9 26631 c07b03 9 [0] 10 26632 c07b03 10 [0] 11 26633 c07b03 11 [0] 12 7700 c07b04 0 [0] 13 7701 c07b04 1 [0] 14 7702 c07b04 2 [0] 15 7703 c07b04 3 [0] 16 7704 c07b04 4 [0] 17 7705 c07b04 5 [0] 18 7706 c07b04 6 [0] 19 7707 c07b04 7 [0] 20 7708 c07b04 8 [0] 21 7709 c07b04 9 [0] 22 7710 c07b04 10 [0] 23 7711 c07b04 11 [0] Init(): I_MPI_DEBUG=100