and obviously other flags related to queue, number of procs and so on.
But when i run my PBS script i get the follow error:
[bash][5:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[0:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[2:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[4:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[1:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[7:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[6:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0<br>[3:node151ib0] unexpected DAPL event 0x4008<br>Assertion failed in file ../../dapl_module_init.c at line 4045: 0<br>internal ABORT - process 0[/bash]
I have read that this type of error is related to process dynamic connection and using “-genv I_MPI_USE_DYNAMIC_CONNECTIONS 0” can be solve it. But it isn’t true.
Adding -env I_MPI_DEBUG 5 the output is:
[21] MPI startup(): RDMA, shared memory, and socket data transfer modes
[17] MPI startup(): RDMA, shared memory, and socket data transfer modes
…
[27] MPI startup(): DAPL provider OpenIB-cma
[24] MPI startup(): DAPL provider OpenIB-cma
…
[28] MPI startup(): shm and dapl data transfer modes
[31] MPI startup(): shm and dapl data transfer modes
[0] MPI startup(): static connections storm algo
node151:16665: dapl_cma_active: PATH_RECORD_ERR, retries(15) exhausted, DST 10.14.102.68,27804
node151:16664: dapl_cma_active: PATH_RECORD_ERR, retries(15) exhausted, DST 10.14.102.68,27804
node151:16663: dapl_cma_active: PATH_RECORD_ERR, retries(15) exhausted, DST 10.14.102.68,27804
node151:16662: dapl_cma_active: PATH_RECORD_ERR, retries(15) exhausted, DST 10.14.102.68,27804
rank 23 in job 1 node151ib0_36911 caused collective abort of all ranks
exit status of rank 23: return code 1
rank 17 in job 1 node151ib0_36911 caused collective abort of all ranks
exit status of rank 17: return code 1
rank 1 in job 1 node151ib0_36911 caused collective abort of all ranks
exit status of rank 1: return code 1
While it’s true that this issues is sometimes resolved by disabling dynamic connection establishment, the 4008 error code here comes from the OFED layer and means “destination unreachable”. And the causes for that are several.
Looking at your command line, though, the most likely culprit is using the OpenIB-cma DAPL device. The CMA driver is fairly old (in relation to how fast technology moves and does not scale out. We recommend using the newer SCM driver instead. It can be found in later versions of the OFED stack. What version do you have installed right now? You can find it by running the ofed_info utility on your cluster. We recommend you upgrade to the latest OFED 1.5.1 release.
Finally, it’ll be great if you could provide the contents of the your /etc/dat.conf file, and the output of the ibstat utility (usually located in /usr/sbin).
thanks for yout reply! Unfortunately, i can’t install new OFED driver because i’m not a root machine. At the moment, i solved my problem with these settings:
CA ‘mlx4_0’
CA type: MT26428
Number of ports: 1
Firmware version: 2.6.648
Hardware version: a0
Node GUID: 0x0002c9030004b056
System image GUID: 0x0002c9030004b059
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 105
LMC: 0
SM lid: 207
Capability mask: 0x02510868
Port GUID: 0x0002c9030004b057
Running with SKaMPI benchmark, test is very slow and appear locked qith a particular size buffer of communication ( 8192). With OpenMPI works correctly. Do you have an idea of this behaviour?
I’m not an expert on the subject, but after searching a bit I have found some potential alternatives at http://kerneltrap.org/mailarchive/openfabrics-general/2008/2/1/685234. I’ve appending it at the end.Also, your issue seems to be related to OFED which follows a typical open-source process, I would suggest checking at http://www.openfabrics.org/resources_linux.htm for documentation/support resources.Hope it helps.-- AndresWhen using rdma_cm to establish end-to-end connections we incur a 3 step process, each with various tunable knobs. There is ARP, Path Resolution, and CM req/reply. Anyone of these could cause the 4008 timeout error. Here are tunable parameters that may help:1. ARP: ARP cache entries for ib0 can be increased from default of 30: sysctl w net.ipv4.neigh.ib0.base_reachable_time=144002. PATH RESOLUTION: ib_sa.ko provides path record caching, no timer controls, auto refresh with new device notification events from SM/SA, manual refresh control for administrators, default == SA caching is OFF. To enable: add following to /etc/modprobe.conf - options ib_sa paths_per_dest=0x7f or echo 0x7f > /sys/module/ib_sa/paths_per_dest To manually refresh: echo 1 > /sys/module/ib_sa/refresh To monitor: cat /sys/module/ib_sa/lookup_method * 0 round robin 1 round robin cat /sys/module/ib_sa/paths_per_dest You can also increase the uDAPL PR timeout with the following enviroment variable (if you don’t have SA caching): export DAPL_CM_ROUTE_TIMEOUT_MS=20000 (default=4000)3. CM PROTOCOL: OFED 1.2.5 provides the following module parameters to increase the IB cm response timeout from default of 21: To increase timeout: add following to /etc/modprobe.conf - options rdma_cm cma_response_timeout=23 options ib_cm max_timeout=23
I actually think you might be running over the good ol’ sockets interface instead of one of the faster inteconnects. This might explain the slowness. To check, could you provide the output of your mpirun command (since you’re already using I_MPI_DEBUG)?
Also, thanks for providing all that info. Instead of you setting all of those env variables, could you try this:
I’m glad to hear you have it working. Enjoy the Intel MPI Library
Quoting unrueBut what is exactly DAPL provider? An interface to Infiniband driver?
Yes, that’s one way to think of it. If you start at the highest level, the Intel MPI Library supports several interfaces:
sockets - just regular TCP/IP for GigE and 10GigE clusters
shared memory - via the /dev/shm device
TMI - a new interface to allow for native tag matching support - for Myrinet* MX* and Qlogic* PSM* drivers
OFA - direct support of the OFED verbs - allows for multi-rail capabilities
DAPL (Direct Access Programming Library) - formally defined, this is an RDMA (remote direct memory access) API. Informally defined, it’s just a way for the various interconnect vendors out there (e.g. Mellanox, Qlogic, etc) to interact with the higher level libraries (such as Intel MPI) in some standard fashion. And because there are multiple IHVs making network cards (plus different versions of the DAPL interface - 1.2 and 2.0), you need multiple DAPL providers to support them all.
In your case, you were using Mellanox cards, I believe, so you had to pick the one DAPL provided that provided support for those: ofa-v2-mlx4_0-1. The “ofa-v2” is OFED’s way of designating DAPL providers that support the new DAPL 2.0 standard (vs. DAPL 1.2, marked by the OpenIB prefix), the “mlx4_0” was taken from the ibstat output, determining which vendor cards you have, and the “-1” means only your port 1 is active (also from ibstat).
That’s probably a lot more info than you wanted but I hope it was helpful nonetheless.
This is getting a bit outside my expertise, since it’s a question regarding OFED and how they name their devices. My guess would be that you can probably use the ofa-v2-ib0 device, but that would default to the mlx4_0 one. My suggestion would be, go ahead and try it both ways with I_MPI_DEBUG=5. That should print out some information so we can see which providers it’s accessing.
But, just to clarify, you are using InfiniBand if you’re selecting the ofa-v2-mlx4_0-1 provider. That’s the purpose of the /etc/dat.conf file - DAPL providers to access your IB cards.
If you want to switch between IB and Ethernet, you need to do this:
and does not scale out. We recommend using the newer SCM driver instead. It can be found in later versions of the OFED stack. What version do you have installed right now? You can find it by running the ofed_info utility on your cluster. We recommend you upgrade to the latest 