Intel MPI support for Amazon's EFA libfabric provider

[areid2]

Hi,

Is anyone working on adding support for Amazon's EFA libfabric provider? Described here:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html

They currently have a build of libfabric that supports their 100Gb/s network with OS bypass. I tried getting Intel MPI to use their libfabric, but it doesn't seem to work. I guess Intel MPI needs to be aware of the specific provider? That surprised me a bit since I thought the point of libfabric would be to provide abstracted access to the fabric.

If support for the EFA provider is in progress, I've got an HPC app that could benefit and would be interested in doing some testing.

[Mikhail_S_Intel]

Hello,

Yes, we are working on adding support for EFA in Intel MPI. For now there is no out-of-box support for EFA provider but it can be enabled with few manual steps. We plan to hide these steps with future IMPI release.

The steps:

1. Install AWS/EFA software stack on c5n.18xlarge instances (section "Step 3: Install EFA Software Components" from link you mentioned above).
2. Install Intel MPI using attached bash script.
3. Build and install OFI/EFA from sources as they include the recent fixes for IMPI.

After installation it should be enough to perform "source /op/intel/impi/.../intel64/bin/mpivars.sh" and "export LD_LIBRARY_PATH=<ofi_install_path>/lib:$LD_LIBRARY_PATH" to enable IMPI environment.

To launch MPI job check that you have passwordless access between instances. And use for example this command:

I_MPI_DEBUG=1 mpiexec.hydra -n 2 -ppn 1 -f hostfile IMB-MPI1 pingpong

In logs you should see "MPI startup(): libfabric provider: efa" - this means that expected provider is used by IMPI.

[areid2]

Thanks Mikhail. I'll give that a shot. I've already got a baseline of PingPong benchmark data for Intel MPI without EFA support, so will run that again with the EFA support.

[areid2]

I followed the steps above and everything seemed to install fine, but isn't quite working. When I run I can see it is using the libfabric I built, but it uses the tcp provider. The performance is not as good as I was seeing with the production Intel MPI 2019.

I tried forcing it to use the efa provider, but then it errors out:

[ec2-user@ip-172-31-23-250 ~]$ export FI_PROVIDER=efa
[ec2-user@ip-172-31-23-250 ~]$ mpiexec.hydra -n 2 -ppn 1 -hosts 172.31.23.250,172.31.25.29 ./benchmarks/imb_impi/IMB-MP
I1 PingPong
[0] MPI startup(): libfabric version: 1.8.0rc1

[0] MPI startup(): libfabric provider: efa;ofi_rxd

Abort(1094799) on node 1 (rank 1 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(666)......:
MPID_Init(922).............:
MPIDI_NM_mpi_init_hook(987): OFI address vector open failed (ofi_init.h:987:MPIDI_NM_mpi_init_hook:Invalid argument)
Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(666)......:
MPID_Init(922).............:
MPIDI_NM_mpi_init_hook(987): OFI address vector open failed (ofi_init.h:987:MPIDI_NM_mpi_init_hook:Invalid argument)

I built the OFI libfabric off master, is that the right branch? I already had Intel MPI 2019.u4 installed, so when I ran that install script it just seemed to patch the install I already had. Should I have removed the install before running your install script?

[Mikhail_S_Intel]

> I built the OFI libfabric off master, is that the right branch?

For now use 1.8.0rc1 tag (git checkout v1.8.0rc1). There is work in progress on libfabric side to fix IMPI/EFA initialization.
I verified 1.8.0rc1 and it works well on IMB with expected "efa" provider in log.

> I already had Intel MPI 2019.u4 installed, so when I ran that install script it just seemed to patch the install I already had. Should I have removed the install before running your install script?

In this case install script should patch existing mpivars.sh. Make sure that mpivars.sh has new "export ..." lines in the beginning, from section 3 of install script, for example MPIR_CVAR_CH4_OFI_ENABLE_ATOMICS, I_MPI_TUNING_BIN.

[areid2]

Thanks, I got it working. The problem on my first attempt is that I had Intel MPI 2018 and 2019 installed and the script decided to patch mpivars.sh from the 2018 version by accident. Once I fixed that it all worked as expected. The latency looks better now, ~16us instead of ~25us with sockets. The bandwidth looks a little bit lower with EFA though, ~10% less than I measured with sockets in the pingpong test.

Looks like I get about the same pingpong results with Intel MPI+EFA as I get with OpenMPI+EFA using the Amazon stack, which I guess isn't surprising. The big advantage to Intel MPI for me is that it supports shared mem for intra node, which apparently Amazon's OpenMPI+EFA does not yet.

Thanks for the help. I'll see if my HPC app runs faster now!

[Jeyaraj]

 HI Team,

1.We have set up an AWS parallel cluster, created the compute and viz nodes, and the instance head node. We are utilizing the Alair PBS Pro job scheduler to schedule jobs on demand using the OnDemand instance.

2.The cluster for computation and visualization has been set up, and on the compute node, the drivers for the AWS EFA, Fsx Lustre file system, and packages for Ansys and Altair requirements have been installed.

3.The job is running and finished while the ANSYS Mechanical batch job is being run on Altair access and submit the compute node with 1 host and 48 cores. I receive the following intel MPI error when I use Multinode.(2 host and 48 core) The user's bashrc file now has the EFA IB parameters added to it.however the mistake persists.I kindly ask for your assistance.

export I_MPI_OFI_PROVIDER=efa
export I_MPI_DEBUG=5
export I_MPI_FABRICS=ofi
export FI_EFA_FORK_SAFE=1

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-enable

4. once submit the Ansys Mechanical job after the line libfabric provider: efa hang

*************output file*****************************

/nfscratch/ansys_inc/v231/ansys/bin/ansys231 -p ansys -dir /fsxscratch/PBS_RUN/pbs.487.ip-10-90-96-108.x8z -np 12 -s read -l en-us -i round_relief_bracket.dat -o round_relief_bracket.out -mpi intelmpi -dis -machines 10.90.96.210:12:10.90.96.229:12: -usessh

NOTE: The -usessh command line option is no longer necessary as SSH is the default protocol
for Distributed ANSYS. The -usessh option is ignored.

[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric version: 1.13.2rc1-impi
[0] MPI startup(): libfabric provider: efa

*********************ERROR****************************

 + /nfscratch/ansys_inc/v231/ansys/bin/ansys231 -p ansys -dir /fsxscratch/PBS_RUN/pbs.487.ip-10-90-96-108.x8z -np 12 -s read -l en-us -i round_relief_bracket.dat -o round_relief_bracket.out -mpi intelmpi -dis -machines 10.90.96.210:12:10.90.96.229:12: -usessh
Warning: Permanently added '10.90.96.210' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.90.96.229' (ECDSA) to the list of known hosts.
Abort(5809551) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(178).......................:
MPID_Init(1532).............................:
MPIDI_OFI_mpi_init_hook(1602)...............:
MPIDU_bc_allgather(194).....................: MPI_Allgather failed
MPIR_Allgather_intra_auto(66)...............: Failure during collective
MPIR_Allgather_intra_auto(62)...............:
MPIR_Allgather_intra_recursive_doubling(117):
MPIC_Sendrecv(349)..........................:
MPIC_Wait(65)...............................:
MPIR_Wait_impl(43)..........................:
MPIDI_Progress_test(95).....................:
MPIDI_OFI_handle_cq_error(1100).............: OFI poll failed (ofi_events.c:1100:MPIDI_OFI_handle_cq_error:Not a directory)
Abort(5809551) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(178).......................:
MPID_Init(1532).............................:
MPIDI_OFI_mpi_init_hook(1602)...............:
MPIDU_bc_allgather(194).....................: MPI_Allgather failed
MPIR_Allgather_intra_auto(66)...............: Failure during collective

Regards,

Jeyaraj.M

+91-9738479698

Bangalore

INDIA

[TobiasK]

@Jeyaraj 
please don't reopen such old posts, post a new thread for a new issue.

Intel MPI 2021.6 is very old, please use the latest 2021.12 release.