but when I run a MPI code, I get: mpiexec -genv I_MPI_DEVICE rdma -env I_MPI_DEBUG 4 -n 2 ./a.out I_MPI:  my_dlopen(): dlopen failed: libmpi.def.so I_MPI:  set_up_devices(): will use static-default device couldn't open /dev/ts_ua_cm0: No such file or directory
using more debug value, I get something strange: I_MPI:  try_one_device(): trying device: libmpi.rdma.so I_MPI:  my_dlsym(): dlsym for dats_get_ia_handle failed: /usr/lib/libdat.so: undefined symbol: dats_get_ia_handle I_MPI:  can_use_dapl_provider(): returning; DAPL provider not ok to use: ib0 I_MPI:  can_use_dapl_provider(): returning; DAPL provider not ok to use: ib1
Which version of the Mellanox IBGD package are you using?
If it is 1.8.0 or later you may have to enable the DAPL before you can use the Intel MPI.
Install the Mellanox package with everything selected. This makes sure you have the DAPL software installed.
The DAPL driver is not enabled by default on these versions. To enable it you need to make a minor change to a file:
Change the answer to loading UDAPL to YES on the copy on the master node. Do the same thing to all of the other nodes in the cluster. Once you have finished I recommend shutting down all of the compute nodes then rebooting the master node. This will run the openib init correctly and you should see the fabric come up as each node is turned on.
Once all of the nodes are up you should be able to use the Intel MPI with the proper switch to utilize the RDMA driver.