Intel MIC with OFED

[Atul_Y_]

Hi,

I am trying to build hpc cluster with 4 phi card and 2 nodes.

i am using mlx interconnect for mpi communication.

th cluster details given below:

centos 6.4, intel cluster 2013, MPSS 3.1.2,OFED 1.5.4.1

Intel Phi card is configured with external network connection on bridge with eth0 interface

opensmd,mpss,openibd,ofed-mic services are running properly.

Only on all 4  intel phi card is used for running service program

Question:-

 How to identify all 4 phi card communication is running on infiniband not on ethernet.

Please provide solution in detailed description.

[Frances_R_Intel]

Good question. Your bridge is configured the way it is supposed to be. But how do you prove to yourself that the MPI communication is going over the InfiniBand connection even though the bridge points to the ethernet? I will ask around.

In the meantime, if you haven't already read it, you might find section 2.2.9 of the Intel® Xeon Phi™ System Software Developer’s Guide interesting. (You can find the link at http://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-system-software-developers-guide.) You have apparently already read the chapter 2 of the MPSS User's Guide. You might also find Configuring Intel® Xeon PhiTM coprocessors inside a cluster useful. (You can find the link at http://software.intel.com/en-us/articles/configuring-intel-xeon-phi-coprocessors-inside-a-cluster.) It is a bit out of date at this point and due for a refresh, but it might still be useful. 

[Frances_R_Intel]

The answers I got:

If you are using Intel(r) MPI Library, you can set the I_MPI_DEBUG environment variable to 3 or higher. The output will tell you what fabric is being used:

[plain]

$ mpirun –genv I_MPI_DEBUG 3 –n 16 ./exe

[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1

[0] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1

[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1

…

[0] MPI startup(): dapl data transfer mode

[/plain]

Otherwise, you can use the InfiniBand tools. 

[plain]

ibclearcounters                               # clear counters before running a MPI job.

ibdatacounts   < HCA lid #>   <HCA port #>     # check receive data of a specific port

Or ibdatacounters                         # check receive data on all ports

[/plain]

Does this help?