Use DAPL provider ofa-v2-mlx4_0-1 without an HCA

[Andrey_Vladimirov]

The DAPL provider ofa-v2-mlx4_0-1 gives much better short message latency for communication between Xeon Phi coprocessors within a single system than ofa-v2-scif0 (2-8 microseconds versus 15 microseconds).

Is it because ofa-v2-mlx4_0-1 uses the HCA? If it is not using the HCA, is it possible to enable this provider without installing an HCA into a stand-alone system with Xeon Phi coprocessors?

[Loc_N_Intel]

Hi Andrey,

An expert is looking into this issue. He/she will get back to you soon. Thank you.

[Gergana_S_Intel]

Hi Andrey,

You're correct, we recommend using the ofa-v2-mlx4_0-1 provider (or corresponding DAPL provider for your InfiniBand HCAs) when sending small messages since it does provide better latency.  It does use the underlying IB card do fast message passing.

On the flip side, the ofa-v2-scif0 provider is preferred for larger messages where bandwidth really plays a role.

If you do not have an IB HCA on your machine, the OFED/SCIF layer will still do some software-level optimizations to provide better performance.  Just make sure to install the OFED* software stack that's shipped with the MPSS distribution on your coprocessor.

A lot more details are available in the Intel® Xeon Phi™ Coprocessor Developers Guide.  Scroll down to section 2.2.9 "Intel® Xeon Phi™ Coprocessor Software Stack for MPI Applications" and read up on the "Coprocessor Communication Link (CCL)" and "OFED*/SCIF" parts.

Note that the Intel MPI Library will do this provider selection for you automatically.  We select the proper DAPL provider to get better small latency and the correct SCIF/CCL provider for better large message bandwidth.  Just make sure you have the latest Intel MPI Library 4.1 Update 3 installed.

I hope this helps.

Regards,
~Gergana