Intel Phi on Centos6.5 with RHEL InfiniBand Stack

[Rafal_O_]

Hi,

I would like to ask whether it is possible to install Intel MPSS with standard (distributed with OS) RHEL Infiniband stack. In manual (http://registrationcenter.intel.com/irc_nas/4042/MPSS_Users_Guide.pdf) in section 2.3 there are steps to install MPSS with OFED, but we have diskless cluster (8 x blade (Mellanox ConnectX-3) with 3 x Intel Phi in each node + many others blades without Intel Phi) with standard RHEL IB (groupinstall infiniband). Is there a chance to make it work with this IB stack?

[Frances_R_Intel]

If you use the RHEL distributed version of OFED, you cannot use the ofed routines that come with the MPSS. This means that communication between the coprocessor and the host will not make use of any OFED features. It won't affect programs using the offload style of programming. However, for programs that run natively on the coprocessor, communication over Infiniband will loose out on some performance advantages.

[Jakub_Rączkowski]

Does Intel have any near future plans for supporting native Xeon Phi applications over InfiniBand on RHEL 6.5 and newer?

RHEL's own OFED stack support would be fantastic, but support for OFED 3.x releases or Mellanox own release (2.1 release seems to support RHEL 6.5) would do it as well.

[Andrey_Vladimirov]

They already support native Xeon Phi applications over InfiniBand (both Mellanox and True Scale) on RHEL 6.5: http://software.intel.com/en-us/forums/topic/506751#comment-1783258

[Frances_R_Intel]

What Andrey says is correct, but currently to get this support, you must either use the Open Fabrics version of OFED plus additional files distributed with the MPSS, or use the True Scale version of OFED plus additional files distributed with the MPSS or the branch of the Open Fabrics source that includes the Intel Xeon Phi coprocessor support (the OFED-3.5-1-mic branch). I am asking around to see what may be coming, but at this time I don't know of any plans to mainline coprocessor support into the Open Fabrics version or RHEL.

[Tommi_T_]

Well, I've to say that OFED-3.5-1-mic BETA1 does not work properly with el6.5. MPI communication between mic and remote CPU node does not work. If I rollback to the older kernel and OFED 1.5.4.1 then communication works out of a box. Unfortunately old kernels have local root exploits etc so it's not option for computing sites which allow users from outside world.

[Frances_R_Intel]

The MPSS User's Guide says that RHEL6.5 can use OFED 1.5.4.1 as long as the 32-bit packages are not installed. Is this why you rolled back to an earlier version of RHEL? Or is the User's Guide missing something?

I'm sorry that the OFED-3.5-1-mic BETA1 does not work for you. On the bright side, support for the coprocessor is being mainlined into the Open Fabrics Alliance OFED and support is being added to the MPSS to allow Mellanox OFED 2.1 to be used.