# \[solved\] random problems with MPI + DAPL initialization in RedHat 5.4

**URL:** <https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071>\
**Category:** Intel® MPI Library\
**Created:** [March 9, 2010, 10:34am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071 "2010-03-09T10:34:32Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 9, 2010, 10:34am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/1 "2010-03-09T10:34:32Z")

</div>

```auto
Hi I havesometimes problems with execution of a program with Intel MPI

```

```auto
It happens with an error on stderr (or stdout):

```

> ```auto
> problem with execution of on wn20: [Errno 13] Permission denied
> 
> ```

What could be a problem?

> here is my ulimit -a:core file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedscheduling priority (-e) 0file size (blocks, -f) unlimitedpending signals (-i) 135167max locked memory (kbytes, -l) unlimitedmax memory size (kbytes, -m) unlimitedopen files (-n) 1024pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority (-r) 0stack size (kbytes, -s) unlimitedcpu time (seconds, -t) unlimitedmax user processes (-u) 135167virtual memory (kbytes, -v) unlimitedfile locks (-x) unlimited

> I’ve checked logs on this node (wn20):Mar 8 16:11:52 wn20 mpd: mpd starting; no mpdid yetMar 8 16:11:52 wn20 mpd: mpd has mpdid=wn20\_45723 (port=45723)Mar 8 16:11:53 wn20 mpd: wn20\_45723 (run 1485): Warning: the directory pointed by TMPDIR (/tmp/pbs.2045.mgmt1) does not exist! /tmp will be used.Mar 8 16:11:53 wn20 mpd: wn20\_45723 (\_\_init\_\_ 1045): Warning: the directory pointed by TMPDIR (/tmp/pbs.2045.mgmt1) does not exist! /tmp will be used.Mar 8 16:11:53 wn20 sshd[11867]: pam\_unix(sshd:session): session closed for user routnwpMar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_120Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_121Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_122Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_123Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_124Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_125Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_126Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_127Mar 8 16:12:07 wn20 mpd: mpd ending mpdid=wn20\_45723 (inside cleanup)Mar 8 16:11:52 wn20 mpd: mpd starting; no mpdid yetMar 8 16:11:52 wn20 mpd: mpd has mpdid=wn20\_45723 (port=45723)Mar 8 16:11:53 wn20 mpd: wn20\_45723 (run 1485): Warning: the directory pointed by TMPDIR (/tmp/pbs.2045.mgmt1) does not exist! /tmp will be used.Mar 8 16:11:53 wn20 mpd: wn20\_45723 (\_\_init\_\_ 1045): Warning: the directory pointed by TMPDIR (/tmp/pbs.2045.mgmt1) does not exist! /tmp will be used.Mar 8 16:11:53 wn20 sshd[11867]: pam\_unix(sshd:session): session closed for user routnwpMar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_120Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_121Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_122Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_123Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_124Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_125Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_126Mar 8 16:11:57 wn20 mpdman: mpdman starting new log; wn20\_mpdman\_127Mar 8 16:12:07 wn20 mpd: mpd ending mpdid=wn20\_45723 (inside cleanup)

---

<div class="post-metadata">

**Author:** ![Dmitry\_K\_Intel2](https://avatars.discourse-cdn.com/v4/letter/d/0061B3/32.png) [@Dmitry\_K\_Intel2](https://community.intel.com/u/Dmitry_K_Intel2)\
**Post date:** [March 9, 2010, 12:14pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/2 "2010-03-09T12:14:06Z")

</div>

Hi,

Could you provide command line and output in verbose mode if possible.

Regards!

Dmitry

---

<div class="post-metadata">

**Author:** ![Dmitry\_K\_Intel2](https://avatars.discourse-cdn.com/v4/letter/d/0061B3/32.png) [@Dmitry\_K\_Intel2](https://community.intel.com/u/Dmitry_K_Intel2)\
**Post date:** [March 9, 2010, 1:57pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/3 "2010-03-09T13:57:56Z")

</div>

\>Presumably meaning with environment variable I\_MPI\_DEBUG=9

It seems to me that the issue is related to mpdboot (or mpirun) so this is ‘--verbose’ option for this command.

Regards!

Dmitry

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 17, 2010, 10:18am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/4 "2010-03-17T10:18:45Z")

</div>

Hi Dmitry,thanks for tip. Unfortunately I cannot reproduce the problem.I’ve raised I\_MPI\_DEBUG to 5 as said in documentation, will 9 give more verbosity?The problem is we’ve got now (with I\_MPI\_DEBUG=5):[56] MPI startup(): DAPL provider OpenIB-cma specified in DAPL configuration file /etc/dat.conf[cli\_56]: got unexpected response to get :cmd=get kvsname=kvs\_wn3\_49596\_0\_0 key=DAPL\_MISMATCH:[cli\_56]: got unexpected response to put :cmd=put kvsname=kvs\_wn3\_49596\_0\_0 key=P56-businesscard value=rdma\_port#21114$rdma\_host#2:0:0:192:168:20:10:0:0:0:0:0:0:0:0$:[cli\_56]: aborting job:Fatal error in MPI\_Init: Other MPI error, error stack:MPIR\_Init\_thread(283)…: Initialization failedMPIDD\_Init(98)…: channel initialization failedMPIDI\_CH3\_Init(261)…:MPIDI\_CH3U\_Init\_rdma(64): PMI\_KVS\_Put returned -1[56] MPI startup(): DAPL provider OpenIB-cma specified in DAPL configuration file /etc/dat.conf[cli\_56]: got unexpected response to get :cmd=get kvsname=kvs\_wn3\_49596\_0\_0 key=DAPL\_MISMATCH:[cli\_56]: got unexpected response to put :cmd=put kvsname=kvs\_wn3\_49596\_0\_0 key=P56-businesscard value=rdma\_port#21114$rdma\_host#2:0:0:192:168:20:10:0:0:0:0:0:0:0:0$:[cli\_56]: aborting job:Fatal error in MPI\_Init: Other MPI error, error stack:MPIR\_Init\_thread(283)…: Initialization failedMPIDD\_Init(98)…: channel initialization failedMPIDI\_CH3\_Init(261)…:MPIDI\_CH3U\_Init\_rdma(64): PMI\_KVS\_Put returned -1what could be a possible problem?here is also the output of env from one of mpi processess (I’m running bash script in mpirun to debug it more closely at MPI process level)I\_MPI\_INFO\_LCPU=16I\_MPI\_INFO\_SIGN=67237VT\_MPI=impi3I\_MPI\_INFO\_PACK=1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0I\_MPI\_PIN\_MAP=56 1,57 5,58 3,59 7,60 0,61 4,62 2,63 6I\_MPI\_PIN\_INFO=6I\_MPI\_INFO\_CACHE\_SHARE=2,2,16I\_MPI\_PIN\_UNIT=6I\_MPI\_INFO\_THREAD=0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1I\_MPI\_INFO\_CACHES=3I\_MPI\_INFO\_CORE=0,0,2,2,1,1,3,3,0,0,2,2,1,1,3,3I\_MPI\_DEVICE=rdmaI\_MPI\_RDMA\_EAGER\_THRESHOLD=25972I\_MPI\_INFO\_CACHE\_SIZE=32768,262144,8388608I\_MPI\_DEBUG=5I\_MPI\_INFO\_CACHE1=8,0,10,2,9,1,11,3,8,0,10,2,9,1,11,3I\_MPI\_PIN\_MAP\_SIZE=8I\_MPI\_INFO\_CACHE2=8,0,10,2,9,1,11,3,8,0,10,2,9,1,11,3I\_MPI\_INFO\_CACHE3=1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0I\_MPI\_PERHOST=allcoresMPICH\_INTERFACE\_HOSTNAME=192.168.0.10I\_MPI\_ROOT=/opt/intel/impi/3.2.1.009One more thing, we run it through batch scheduler. After running task I saw that mpd process exist - could it be somewhat connected with the problem:python /opt/intel/impi/3.2.1.009/bin64/mpd.py -h wn9 -p 34585 --ifhn=192.168.0.13 --ncpus=1 --myhost=wn13 --myip=192.168.0.13 -e -d -s 5

---

<div class="post-metadata">

**Author:** ![Dmitry\_K\_Intel2](https://avatars.discourse-cdn.com/v4/letter/d/0061B3/32.png) [@Dmitry\_K\_Intel2](https://community.intel.com/u/Dmitry_K_Intel2)\
**Post date:** [March 17, 2010, 11:17am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/5 "2010-03-17T11:17:21Z")

</div>

This problem is most likely related to configuration of OFED or IP addresses for IPoIB.

Again, I don’t see your command line - it might be useful in some cases.  
What is your DAPL version (run ‘ofed\_info’ command)?  
Could you provide /etc/dat.conf?  
What interconnect cards do you use?

The higher number for I\_MPI\_DEBUG you set the more information you get.

Please try to run you application with I\_MPI\_DEVICE set to ‘sock’.

Regards!  
Dmitry

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 17, 2010, 2:50pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/6 "2010-03-17T14:50:11Z")

</div>

\>This problem is most likely related to configuration of OFED or IP addresses for IPoIB.I’ll check that, thanks. The problem is that it happens randomly only in particular jobs and the configuration is static…  
My command line ismpirun -r ssh -env I\_MPI\_DEBUG 5 -env I\_MPI\_DEVICE rdssm -np 196 /full/path/bin/cm\_w\_00.0.0.2.shwhere cm\_w\_00.0.0.2.sh contains/full/path/bin/cm \> $logbin 2\>&1and other few commands redirected logfiles (like $logbin) with names unique to mpiprocess to gather debugging data like output of ps, ulimit etc. but there is nothing interesting in those logsI’m using stock RHEL5.4 OFEDdapl is dapl-2.0.19-2.el5 from repos, I do not have ofed\_info command/etc/dat.conf is/etc/ofed/dat.conf in RHEL:ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 “ib0 0” ""ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 “ib1 0” ""ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mthca0 1” ""ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mthca0 2” ""ofa-v2-mlx4\_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mlx4\_0 1” ""ofa-v2-mlx4\_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mlx4\_0 2” ""ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “ipath0 1” ""ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “ipath0 2” ""ofa-v2-ehca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “ehca0 1” ""ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 “eth2 0” ""cat /etc/ofed/dat.confofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 “ib0 0” ""ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 “ib1 0” ""ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mthca0 1” ""ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mthca0 2” ""ofa-v2-mlx4\_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mlx4\_0 1” ""ofa-v2-mlx4\_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “mlx4\_0 2” ""ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “ipath0 1” ""ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “ipath0 2” ""ofa-v2-ehca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 “ehca0 1” ""ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 “eth2 0” ""I think about trying sock device but this will rather avoid problem which happens randomly - do you think that’s a good idea?I’m using Mellanox ConnectX:ibv\_devinfohca\_id: mlx4\_0fw\_ver: 2.6.100node\_guid: 0023:7dff:ff94:4518sys\_image\_guid: 0023:7dff:ff94:451bvendor\_id: 0x02c9vendor\_part\_id: 26428hw\_ver: 0xA0board\_id: HP\_0120000009phys\_port\_cnt: 2port: 1state: active (4)max\_mtu: 2048 (4)active\_mtu: 2048 (4)sm\_lid: 6port\_lid: 4port\_lmc: 0x00port: 2state: active (4)max\_mtu: 2048 (4)active\_mtu: 2048 (4)sm\_lid: 6port\_lid: 5port\_lmc: 0x00

---

<div class="post-metadata">

**Author:** ![Dmitry\_K\_Intel2](https://avatars.discourse-cdn.com/v4/letter/d/0061B3/32.png) [@Dmitry\_K\_Intel2](https://community.intel.com/u/Dmitry_K_Intel2)\
**Post date:** [March 17, 2010, 3:04pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/7 "2010-03-17T15:04:58Z")

</div>

Your first output refers to OpenIB-cma provider but there is no such provider in the dat.conf you sent me.  
So you probably need to use DAT\_OVERRIDE=/etc/ofed/dat.conf variable to point to the correct dat.conf file.  
Could you also change DEVICE env variable to:  
-env I\_MPI\_DEVICE rdssm:ofa-v2-mlx4\_0-1  
In this case mlx4\_0 will be used explicitly

\>The problem is that it happens randomly only in particular jobs  
This is very strange. Might be something wrong with cluster configuration or unstable work of some nodes.  
Could you also add:  
-env I\_MPI\_FALLBACK\_DEVICE off  
to your command line.

Let me know the result.

Regards!  
Dmitry

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 19, 2010, 10:05am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/8 "2010-03-19T10:05:18Z")

</div>

I’ve tried your suggestion, it gave me:[0] DAPL provider is not found and fallback device is not enabled[cli\_0]: aborting job:Fatal error in MPI\_Init: Other MPI error, error stack:MPIR\_Init\_thread(283): Initialization failedMPIDD\_Init(98)…: channel initialization failedMPIDI\_CH3\_Init(163)..: generic failure with errno = -1(unknown)():[0] MPI startup(): Intel MPI Library, Version 3.2.1 Build 20090312[0] MPI startup(): Copyright (C) 2003-2009 Intel Corporation. All rights reserved.rank 0 in job 1 wn1\_33304 caused collective abort of all ranksexit status of rank 0: return code 13I’ve checked what is dapl library default dat.conf:wn3 ~]$ dapltestDapltest: Service Point Ready - ofa-v2-ib0wn3 ~]$ dapltestDapltest: Service Point Ready - ofa-v2-ib0I’ve tried to use the same name with Intel mpirun with the same result -[0] DAPL provider is not found and fallback device is not enabledThe weird thing is when running with just I\_MPI\_DEVICE=rdssm:[0] MPI startup(): DAPL provider OpenIB-cma specified in DAPL configuration file /etc/dat.conf[0] MPI startup(): RDMA, shared memory, and socket data transfer modes[0] MPI startup(): Intel MPI Library, Version 3.2.1 Build 20090312[0] MPI startup(): Copyright (C) 2003-2009 Intel Corporation. All rights reserved.so it’s trying to use OpenIB-cma, but it’s not definied anywhere, weird thing is - it’s working but not always…So - Intel MPI is not using dat.conf which dat itself is using?I’ll also try to link dat.conf from /etc/ofed to /etc

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 19, 2010, 10:45am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/9 "2010-03-19T10:45:01Z")

</div>

\> I’ll also try to link dat.conf from /etc/ofed to /etc  
nope, linking won’t work 😕

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 19, 2010, 12:09pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/10 "2010-03-19T12:09:33Z")

</div>

I realized that IntelMPI in RHEL 5.4 is not using dapl - it’s using compat-dapl which has different dat.conf (don’t ask me why):# cat /etc/ofed/compat-dapl/dat.confOpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 “ib0 0” ""OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 “ib1 0” ""OpenIB-mthca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 “mthca0 1” ""OpenIB-mthca0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 “mthca0 2” ""OpenIB-mlx4\_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 “mlx4\_0 1” ""OpenIB-mlx4\_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 “mlx4\_0 2” ""OpenIB-ipath0-1 u1.2 nonthreadsafe default libdaplscm.so.2 dapl.1.2 “ipath0 1” ""OpenIB-ipath0-2 u1.2 nonthreadsafe default libdaplscm.so.2 dapl.1.2 “ipath0 2” ""OpenIB-ehca0-2 u1.2 nonthreadsafe default libdaplscm.so.2 dapl.1.2 “ehca0 1” ""OpenIB-iwarp u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 “eth2 0” ""I’ve usedOpenIB-mlx4\_0-1 in I\_MPI\_DEVICE and it runs ok for now - I’m waiting if this error will appear again.  
Do you think this is what you wanted me to do?

---

<div class="post-metadata">

**Author:** ![Andres\_M\_Intel4](https://avatars.discourse-cdn.com/v4/letter/a/0046C8/32.png) [@Andres\_M\_Intel4](https://community.intel.com/u/Andres_M_Intel4)\
**Post date:** [March 19, 2010, 1:36pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/11 "2010-03-19T13:36:26Z")

</div>

Perhaps the following link may be helpful to understand about DAPL providers.

I would recommend not having invalid DAPL entries onyour dat.conf file, you may want to only offer to your cluster users those which are fully functional.

[http://software.intel.com/en-us/articles/intel-mpi-library-for-linux-experience-with-various-interconnects-and-dapl-providers/](http://software.intel.com/en-us/articles/intel-mpi-library-for-linux-experience-with-various-interconnects-and-dapl-providers/)

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 19, 2010, 1:57pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/12 "2010-03-19T13:57:21Z")

</div>

Hi, thanks. I’ve already read that.The problem was the entries were not completely bad - I was using just wrong names (from other dat.conf) but what you wanted is to use fixed name for DAPL provider, right?Could that help in solving this issue?:[94] MPI startup(): DAPL provider OpenIB-cma specified in DAPL configuration file[cli\_94]: got unexpected response to get :cmd=get kvsname=kvs\_wn3\_49596\_0\_0 key=DAPL\_MISMATCH:[cli\_94]: got unexpected response to put :cmd=put kvsname=kvs\_wn3\_49596\_0\_0 key=P94-businesscardvalue=rdma\_port#18839$rdma\_host#2:0:0:192:168:20:14:0:0:0:0:0:0:0:0$[cli\_94]: got unexpected response to get :cmd=get kvsname=kvs\_wn3\_49596\_0\_0 key=DAPL\_MISMATCH:[cli\_94]: got unexpected response to put :cmd=put kvsname=kvs\_wn3\_49596\_0\_0 key=P94-businesscardvalue=rdma\_port#18839$rdma\_host#2:0:0:192:168:20:14:0:0:0:0:0:0:0:0$

It’s an Intel MPI message, could you explain to me what does it mean? I cannot find any docs about it.It looks like DAPL provider has been chosen (it was the same when it was running fine).

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [March 19, 2010, 2:53pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/13 "2010-03-19T14:53:42Z")

</div>

after changing I\_MPI\_DEVICE toOpenIB-mlx4\_0-1, I’ve got **from wn3 from RANK0 process:** [0] MPI startup(): DAPL provider OpenIB-mlx4\_0-1[cli\_0]: got unexpected response to get :cmd=get kvsname=kvs\_wn3\_37604\_0\_0 key=DAPL\_MISMATCH:[cli\_0]: got unexpected response to put :cmd=put kvsname=kvs\_wn3\_37604\_0\_0 key=shm\_name value=2D1921C52957AD9B5645EBCD4BA371D0:[0] MPI startup(): Intel MPI Library, Version 3.2.1 Build 20090312[0] MPI startup(): Copyright (C) 2003-2009 Intel Corporation. All rights reserved.[cli\_0]: aborting job:Fatal error in MPI\_Init: Other MPI error, error stack:MPIR\_Init\_thread(283)…: Initialization failedMPIDD\_Init(98)…: channel initialization failedMPIDI\_CH3\_Init(319)…:MPIDI\_CH3U\_Init\_sshm(239): PMI\_KVS\_Put returned -1(unknown)(): **from other processes:** [14] MPI startup(): DAPL provider OpenIB-mlx4\_0-1[cli\_14]: got unexpected response to get :cmd=get kvsname=kvs\_wn3\_37604\_0\_0 key=DAPL\_MISMATCH:[cli\_14]: PMIU\_parse\_keyvals: unexpected key delimiter at character 1 in ![cli\_14]: expecting cmd=barrier\_out, got ![cli\_14]: aborting job:Fatal error in MPI\_Init: Other MPI error, error stack:MPIR\_Init\_thread(283)…: Initialization failedMPIDD\_Init(98)…: channel initialization failedMPIDI\_CH3\_Init(319)…:MPIDI\_CH3U\_Init\_sshm(257): PMI\_Barrier returned -1(unknown)():

---

<div class="post-metadata">

**Author:** ![Rafał\_Błaszczyk](https://avatars.discourse-cdn.com/v4/letter/r/276EF2/32.png) [@Rafał\_Błaszczyk](https://community.intel.com/u/Rafa%C5%82_B%C5%82aszczyk)\
**Post date:** [April 6, 2010, 10:03pm UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/14 "2010-04-06T22:03:13Z")

</div>

**I’ve found a solution to my problem.**  
I believe it was the same problem as described here:[http://software.intel.com/en-us/articles/random-fabric-errors-on-rhel5U4/](http://software.intel.com/en-us/articles/random-fabric-errors-on-rhel5U4/)(workaround I\_MPI\_RDMA\_CREATE\_CONN\_QUAL = 0seemed to work too)

After upgrading to OFED 1.5 with new DAPL the problem was finally solved.

DAPL version from RedHat 5.4 seems buggy.BTW: If anyone knows why RedHat decided to have two separate dat.conf files for each dapl version (1 and 2) please give me a note.I havesuccessfullyused new UCM interface (v2) with ConnectX (ofa-v2-mlx4\_0-1u in dat.conf) which seems to be much much faster with many-core jobs than the old CMA provider.I believe that when sticking to RH provided OFED it’s good to have one common dat.conf (DAT\_OVERRIDE) with providers from DAPL1 and DAPL2.

---

<div class="post-metadata">

**Author:** ![Dmitry\_K\_Intel2](https://avatars.discourse-cdn.com/v4/letter/d/0061B3/32.png) [@Dmitry\_K\_Intel2](https://community.intel.com/u/Dmitry_K_Intel2)\
**Post date:** [April 7, 2010, 10:43am UTC](https://community.intel.com/t/solved-random-problems-with-mpi-dapl-initialization-in-redhat-5-4/63071/15 "2010-04-07T10:43:49Z")

</div>

Rafal, thanks for sharing this information.