Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
Announcements
FPGA community forums and blogs on community.intel.com are migrating to the new Altera Community and are read-only. For urgent support needs during this transition, please visit the FPGA Design Resources page or contact an Altera Authorized Distributor.
2276 Discussions

MPI 4.1 fails to end gracefully when ranks > 2000

Beaver6675
Novice
‎07-29-2014 07:03 AM
1,215 Views

I am testing Intel MPI 4.1 with test.c (the provided test program).

Whenever I run > 2000 ranks the program executes correctly but fails to end gracefully.

Running:

mpiexec.hydra -n 2001 -genv I_MPI_FABRICS shm:ofa -f hostfile ./testc

It stalls at 

...

....

Hello World: Rank 2000 running on host xxxx

##<stalls here; does not return to command prompt>

(If I use -n 2000 or less, it runs perfectly.)

I have testing 3000 ranks using OpenMPI, so it doesn't seem to be a cluster/network issue.

 

0 Kudos

Link Copied

3 Replies
James_T_Intel
Moderator
‎08-01-2014 12:26 PM
1,215 Views

Does it work with I_MPI_FABRICS=shm:dapl or I_MPI_FABRICS=shm:tcp?  Please attach output with I_MPI_DEBUG=5.

0 Kudos
Copy link
TimP
Honored Contributor III
‎08-03-2014 04:59 AM
1,215 Views

Are you running more ranks than slots available?  If so, were you enabling pinning on OpenMPI, and does it help to turn it off for Intel MPI?

0 Kudos
Copy link
Beaver6675
Novice
‎08-04-2014 04:45 AM
1,215 Views

1. DAPL UD works with > 2000 ranks.

2. Attached is the output from I_MPI_FABRICS shm:ofa - stalls after rank 0 receives a single message from ranks 1-2000

3. The cluster has more than 2000 slots: for OpenMPI/OFA I use --map-by socket, with no oversubscription to force the MPI to go across all the nodes.

I am using Mellanox OFED 2.2-1.0.1 on a mlx4 card.

The problem seems to be a MPI -> OFED interaction.

 

Preview file
418 KB
0 Kudos
Copy link
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.