topic Hi Simon, in Intel® MPI Library

Performance issue with multi-rail support in Intel MPI 5.0

Simon_H_2 — Wed, 17 Dec 2014 09:42:36 GMT

Hi,

I am experiencing a severy performance loss when using multiple rails in Intel MPI 5.0 and the KNC and an mlx5 adapter (which has 2 ports). With Intel MPI 4.1 it was much better.

Let me give an example of the performance of our application (per KNC):

Intel MPI 4.1, single-rail (I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx5_0-1u): 220 Gflop/s
Intel MPI 4.1, dual-rail (-IB I_MPI_OFA_ADAPTER_NAME=mlx5_0 I_MPI_OFA_NUM_PORTS=2): 270 Gflop/s
Intel MPI 5.0, single-rail (I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx5_0-1u): 220 Gflop/s
Intel MPI 5.0, dual-rail (-IB I_MPI_OFA_ADAPTER_NAME=mlx5_0 I_MPI_OFA_NUM_PORTS=2): 150 Gflop/s
Intel MPI 5.0, single-rail (-IB I_MPI_OFA_ADAPTER_NAME=mlx5_0 I_MPI_OFA_NUM_PORTS=1): 150 Gflop/s

With DAPL the performance is unchanged, but apparently there is no way to use it with dual-rail support. With OFA I got the best performance in v4.1, but with v5.0 it is extremely low. In particular it is the same for 1 or 2 ports.

Is there anything I am overlooking in the documentation?

Thanks,

Simon

The HCA card in my system has

Loc_N_Intel — Fri, 19 Dec 2014 18:39:58 GMT

The HCA card in my system has one port only, I cannot reproduce the dual-rail issue that you saw. But let me ask experts around here and get back to you. Thank you.

Hi Simon,

Loc_N_Intel — Fri, 19 Dec 2014 20:52:51 GMT

Hi Simon,

I just contacted an expert. Could you describe in details how to reproduce the issue, please? Thank you.

Hi,

Simon_H_2 — Mon, 22 Dec 2014 07:40:52 GMT

Hi,

thanks for your reply. To reproduce, you can use, for example, the OSU bandwidth benchmark: http://mvapich.cse.ohio-state.edu/benchmarks/. My original tests were done on the KNC, but the same problem shows up on the Xeon (Haswell) host.

You can see the result in the attached figure. You can see that for message sizes around 100 kB and above Intel MPI 4.1 with "dual rail" is by far the best (blue solid squares). Intel MPI 5.0 is much much worse.

444821

Are there any new

Simon_H_2 — Mon, 19 Jan 2015 07:03:00 GMT

Are there any new developments on this? Can you confirm the issue now? Thanks, Simon

Hi Simon,

Loc_N_Intel — Wed, 21 Jan 2015 19:41:32 GMT

Hi Simon,

Sorry for the delayed answer. This issue was forwarded to the development team for investigation. I will let you know when I have an update.

Thank you.

Hi,

Simon_H_2 — Wed, 04 Mar 2015 10:17:25 GMT

Hi, Are there any updates regarding this issue? Thanks, Simon

Hi Simon,

Artem_R_Intel1 — Tue, 10 Mar 2015 09:28:55 GMT

Hi Simon,

Could you please specify the exact versions of Intel MPI Library (4.x, 5.x) and OS/MPSS/OFED/DAPL.
Also could you please provide test scenarios you used. Which compute nodes were involved in each run (MPI ranks only on HOST, or only on KNC, or both on HOST and KNC).

Regarding to DAPL - try to run the same scenarios with default DAPL provider (without I_MPI_DAPL_PROVIDER_LIST).

Hi Artem,

Simon_H_2 — Tue, 17 Mar 2015 08:18:00 GMT

Hi Artem,

I used two scenarios, the issue shows up in both cases:

HOST <-> HOST
KNC <-> KNC

Versions:

Intel MPI 4.1.3.045 and 5.0.2.044
OS is Linux (CentOS)
OFED 3.5.2
DAPL 2.1.2
MPSS 3.3.3 (I guess this is irrelevant, since the issue shows up also if only HOSTS are involved)

I think I had tried in the past to run without I_MPI_DAPL_PROVIDER_LIST, but Intel MPI tried to default to an mlx4 device (which does not exist on our system), and would not use the mlx5 device, so using I_MPI_DAPL_PROVIDER_LIST was mandatory. I will try again.

Remark: Should this topic be moved to the general forum, since by now we know that it is not MIC-specific?

Simon

Hi Simon,

Loc_N_Intel — Fri, 27 Mar 2015 23:06:26 GMT

Hi Simon,

FYI, we submitted an internal bug report (DPD200368369) a while ago but there is no update yet.

Hi Simon,

Gergana_S_Intel — Mon, 30 Mar 2015 15:51:09 GMT

Hi Simon,

Loc and I have been communicating internally about this since you initially submitted it.

Just as an FYI, I'm moving this issue over to the regular Intel® Clusters and HPC Technology forum since it's not Phi-specific. That way I can keep track of the internal bug I submitted and update you on current status.

Thanks,
~Gergana

Hey Simon,

Gergana_S_Intel — Mon, 23 Nov 2015 17:06:46 GMT

Hey Simon,

We've made several fixes to the Intel MPI Library in regards to multi-rail support. Just wondering if you've tried the latest Intel MPI 5.1.2 with any better success regarding performance?

~Gergana