DAPL ERR

[Javier_M_2]

Hi

i`m execute WRF in symetric mode in one coprocessor succesfully but obtain this error on two copprocessors. can help me?:

[21] MPI startup(): shm and dapl data transfer modes
[17] MPI startup(): DAPL provider ofa-v2-scif0
[16] MPI startup(): DAPL provider ofa-v2-scif0
[17] MPI startup(): shm and dapl data transfer modes
[16] MPI startup(): shm and dapl data transfer modes
Meteo-Xeon-Phi-mic1:SCM:2dbb:f305e500: 216177 us(216177 us):  modify_qp_state: ERR type 2 qpn 0xe gid 0x2b3cf40229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2dbb:f305e500: 216348 us(171 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2dbb:f305e500: 216391 us(43 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
Meteo-Xeon-Phi-mic1:SCM:2db8:1f47b500: 186585 us(186585 us):  modify_qp_state: ERR type 2 qpn 0x14 gid 0x2b32240229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2db8:1f47b500: 186763 us(178 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2db8:1f47b500: 186845 us(82 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
[15:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
[16:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
Meteo-Xeon-Phi-mic1:SCM:2dbe:36708500: 196925 us(196925 us):  modify_qp_state: ERR type 2 qpn 0x1a gid 0x2aae3c0229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2dbe:36708500: 197101 us(176 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2dbe:36708500: 197182 us(81 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
Meteo-Xeon-Phi-mic1:SCM:2dbc:15493500: 225066 us(225066 us):  modify_qp_state: ERR type 2 qpn 0x21 gid 0x2acb180229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2dbc:15493500: 225237 us(171 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2dbc:15493500: 225315 us(78 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
[17:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
[18:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
Meteo-Xeon-Phi-mic1:SCM:2db9:60277500: 199595 us(199595 us):  modify_qp_state: ERR type 2 qpn 0x27 gid 0x2aff640229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2db9:60277500: 199760 us(165 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2db9:60277500: 199860 us(100 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
[19:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
Meteo-Xeon-Phi-mic1:SCM:2dba:73fc9500: 231631 us(231631 us):  modify_qp_state: ERR type 2 qpn 0x2e gid 0x2b84780229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2dba:73fc9500: 231800 us(169 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2dba:73fc9500: 231904 us(104 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
[20:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
Meteo-Xeon-Phi-mic1:SCM:2dbd:56c6500: 234974 us(234974 us):  modify_qp_state: ERR type 2 qpn 0x36 gid 0x2b0d000229ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:2dbd:56c6500: 235152 us(178 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:2dbd:56c6500: 235195 us(43 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
[21:10.10.10.2][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 0
[12:10.10.10.1] unexpected DAPL event 0x4006
Assertion failed in file ../../dapl_init_rc.c at line 1402: 0
internal ABORT - process 0
[13:10.10.10.1] unexpected DAPL event 0x4006
Assertion failed in file ../../dapl_init_rc.c at line 1402: 0
internal ABORT - process 0
[8:10.10.10.1] unexpected DAPL event 0x4006
Assertion failed in file ../../dapl_init_rc.c at line 1402: 0
internal ABORT - process 0
[10:10.10.10.1] unexpected DAPL event 0x4006
Assertion failed in file ../../dapl_init_rc.c at line 1402: 0
internal ABORT - process 0
[3:10.10.10.254] unexpected disconnect completion event from [10:10.10.10.1]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 3
[7:10.10.10.254] unexpected disconnect completion event from [10:10.10.10.1]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 7
[1:10.10.10.254] unexpected disconnect completion event from [10:10.10.10.1]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 1
[5:10.10.10.254] unexpected disconnect completion event from [10:10.10.10.1]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 5

My configuration is:

wrf_start.sh 

#!/bin/bash
ulimit -s unlimited
ulimit -l unlimited
export I_MPI_PIN_MODE=mpd
export I_MPI_PIN_DOMAIN=auto
export I_MPI_MIC=1
export I_MPI_DEVICE=rdssm
export I_MPI_DEBUG=5
rm rsl.*
rm wrfout*
mpiexec.hydra -host 10.10.10.254 -n 8 ./wrf_sandy.sh : -host  10.10.10.1 -n 8 ./wrf_phi.sh  : -host  10.10.10.2 -n 8 ./wrf_phi.sh

phi.envars

#!/bin/sh
source /opt/intel/impi/4.1.3.048/mic/bin/mpivars.sh
export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic
export OMP_NUM_THREADS=30
export KMP_LIBRARY=turnaround
export KMP_BLOCKTIME=infinite
export KMP_STACKSIZE=32M
export OMP_SCHEDULE=STATIC
export KMP_AFFINITY=balanced

sandy.envars

#!/bin/sh
export OMP_NUM_THREADS=2
export KMP_LIBRARY=turnaround
export KMP_BLOCKTIME=infinite
export KMP_STACKSIZE=32M
export OMP_SCHEDULE=DYNAMIC

wrf_phi.sh

#!/bin/sh
ulimit -s unlimited
ulimit -l unlimited
source ./phi.envvars
./wrf.mic

wrf_sandy.sh

#!/bin/sh
ulimit -s unlimited
ulimit -l unlimited
source ./sandy.envvars
./wrf.exe

My system is one host with 2 coprocesor internal bridge, OS is SLES SP3 kernel 3.0.76-0.11 with OFED 1.5.4.1 and mpss 3.4

Thx in advance.

[Loc_N_Intel]

Hi Javier,

You indicated that your application ran successfully in symmetric mode with one coprocessor, but failed with two coprocessors! It is possible that you need to enable MPSS peer-to-peer communication. Please try to set this:

# sudo /sbin/sysctl -w net.ipv4.ip_forward=1

[Javier_M_2]

Hi loc-nguyen,

i´ve enabled MPSS peer-to-peer communication. this is my configuraion file /etc/modprobe.d/mic.conf 

options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0

i`ve tried this /sbin/sysctl -w net.ipv4.ip_forward=1 but not fix the problem.

i've run succesfully  in symmetric mode with two coprocessors with this option 

export I_MPI_DEVICE=ssm

but with this

export I_MPI_DEVICE=rdssm

 i obtain errors, i dont know why.

thanks in advance.

[Loc_N_Intel]

Hi Javier,

The error DAT_INTERNAL_ERROR, according to the document https://software.intel.com/en-us/articles/intel-mpi-library-for-linux-experience-with-various-interconnects-and-dapl-providers , suggests that there is a problem with IP address configuration for InfiniBand.

Let me ask an expert in MPI/InfiniBand to take a look in this particular problem.  

- Can you show the content in /etc/dat.conf please?

- If you set I_MPI_DEVICE= rdssm, you can run the program successfully with only one coprocessor. Does it matter whether mic0 or mic1?

Thank you.

[Javier_M_2]

Ok this is my configuration /etc/dat.conf file. No matter if mic1 to mic2.

thxs.

# DAT v2.0, v1.2 configuration file
#  
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
#           <provider_version> <ia_params> <platform_params>
#
# For uDAPL cma provder, <ia_params> is one of the following:
#       network address, network hostname, or netdev name and 0 for port
#
# For uDAPL scm provider, <ia_params> is device name and port
# For uDAPL ucm provider, <ia_params> is device name and port
# For uDAPL iWARP provider, <ia_params> is netdev device name and 0 
# For uDAPL iWARP provider, <ia_params> is netdev device name and 0 
# For uDAPL RoCE provider, <ia_params> is device name and 0 
# 
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib0 0" ""
ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib1 0" ""
ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 1" ""
ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 2" ""
ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 1" ""
ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 2" ""
ofa-v2-ehca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ehca0 1" ""
ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mthca0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mthca0 1" ""
ofa-v2-mthca0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mthca0 2" ""
ofa-v2-cma-roe-eth2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
ofa-v2-cma-roe-eth3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth3 0" ""
ofa-v2-scm-roe-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-scm-roe-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mcm-1 u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mcm-2 u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-scif0 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "scif0 1" ""
ofa-v2-scif0-u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "scif0 1" ""
ofa-v2-mic0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "mic0:ib 1" ""
ofa-v2-mlx4_0-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mlx4_1-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_1 1" ""
ofa-v2-mlx4_1-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_1 2" ""
ofa-v2-mlx4_1-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_1 1" ""
ofa-v2-mlx4_1-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_1 2" ""
ofa-v2-mlx4_0-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-mlx4_1-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_1 1" ""
ofa-v2-mlx4_1-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_1 2" ""
ofa-v2-mlx5_0-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2s u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-mlx5_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-mlx5_0-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2m u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx5_1 2" ""

[Loc_N_Intel]

Hi Javier,

After consulting with an MPI expert, the recommendation is not to use I_MPI_DEVICE, as it is being deprecated soon, but instead use I_MPI_FABRICS starting from Intel MPI Libraries version 4.0

# export I_MPI_FABRICS=shm:dapl

Also please set the MPI environment variables explicitly:

# source /opt/intel/impi/4.1.3.048/bin64/mpivars.sh

And there is no need to source the MIC-specific script (source /opt/intel/impi/4.1.3.048/bin/mpivars.sh

In phi.vars)

Please let me know how it goes. Thank you.

[Javier_M_2]

Hi, I tried what you just tell me but still fails.

Thxs.

[Loc_N_Intel]

Could you please try this:

# export I_MPI_DYNAMIC_CONNECTION=1

What Intel compiler version are you using?

[Javier_M_2]

Hi, I tried what you just tell me but still fails.

Versions:

Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.144 Build 20140120

Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.144 Build 20140120

MPSS 3.4 

OFED 3.5.2-MIC

Meteo-Xeon-Phi-mic0:SCM:172e:3b7fd500: 199523 us(199523 us):  modify_qp_state: ERR type 2 qpn 0x8 gid 0x2ab91c0209ec (1) lid 0x3ea port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic0:SCM:172e:3b7fd500: 199705 us(182 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic0:SCM:172e:3b7fd500: 199745 us(40 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.2
[13:mic0][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 13
[14:mic1] unexpected DAPL connection event 0x4006 from 13
Assertion failed in file ../../dapl_poll_rc.c at line 1679: 0
internal ABORT - process 14
Meteo-Xeon-Phi-mic1:SCM:172c:6a008500: 240973 us(240973 us):  modify_qp_state: ERR type 2 qpn 0x1d gid 0x2ae0500209ec (1) lid 0x3e9 port 1 state 1 mtu 4 rd 4 rnr 12 sl 0
Meteo-Xeon-Phi-mic1:SCM:172c:6a008500: 241134 us(161 us):  DAPL ERR modify_qp_state Invalid argument
Meteo-Xeon-Phi-mic1:SCM:172c:6a008500: 241174 us(40 us):  ACCEPT_USR: QPS_RTR ERR Invalid argument -> 10.10.10.1
[9:mic0] unexpected DAPL connection event 0x4006 from 17
Assertion failed in file ../../dapl_poll_rc.c at line 1679: 0
internal ABORT - process 9
[17:mic1][../../dapl_conn_rc.c:620] error(0x40000): ofa-v2-scif0: could not accept DAPL connection request: DAT_INTERNAL_ERROR()
Assertion failed in file ../../dapl_conn_rc.c at line 620: 0
internal ABORT - process 17
[1:Meteo-Xeon-Phi] unexpected DAPL connection event 0x4006 from 13
Assertion failed in file ../../dapl_poll_rc.c at line 1679: 0
internal ABORT - process 1
[15:mic1] unexpected DAPL connection event 0x4006 from 13
Assertion failed in file ../../dapl_poll_rc.c at line 1679: 0
internal ABORT - process 15
[3:Meteo-Xeon-Phi] unexpected disconnect completion event from [10:mic0]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 3
[6:Meteo-Xeon-Phi] unexpected disconnect completion event from [7:mic0]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 6
[5:Meteo-Xeon-Phi] unexpected disconnect completion event from [7:mic0]
Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0
internal ABORT - process 5

Thxs.

[Loc_N_Intel]

Hi Javier,

In the first place, you used OFED 1.5.4.1 and lately you mentioned OFED 3.5.2-MIC. Did you change the OFED stack?

What if you set the provider explicitly, e.g., "export I_MPI_PROVIDER=ofa-v2-scif0"

If your program still fails, instead of using WRF can you try a simple program? For example, the test.c program located in <Intel MPI library install>/test/ ?  

[Javier_M_2]

yes, I've recently changed to OFED 3.5.2. but with the same result.

 the provider explicitly is  "export I_MPI_PROVIDER=ofa-v2-scif0"

I'll try to check the configuration of the opensm and IPoIB.
I'm also going to compile the test.c program that you recommend.