INTEL-MPI-5.0: Bug in MPI-3 shared-memory allocation (MPI_WIN_ALLOCATE_SHARED, MPI_WIN_SHARED_QUERY)

[Michael_R_2]

Dear developers of Intel-MPI,

First of all:   Congratulations, that INTEL-MPI now supports also MPI-3 !

However, I found a bug  in INTEL-MPI-5.0 when running the MPI-3 shared memory feature (calling MPI_WIN_ALLOCATE_SHARED, MPI_WIN_SHARED_QUERY) on a Linux Cluster (NEC Nehalem)  by a  Fortran95 CFD-code.

I isolated the problem into a small Ftn95 example program, which allocates shared an integer*4-array of array dimension N , then uses it by the MPI-processes (on the same node), and then repeats the same for the next shared allocation. So, the number of shared windows do accumulate in the run, because I do not free the shared windows allocated so far. This allocation of shared windows works, but only until the total number of allocated memory exceeds a limit of ~30 millions of Integer*4 numbers (~120 MB).

When that limit is reached, the next call of MPI_WIN_ALLOCATE_SHARED, MPI_WIN_SHARED_QUERY  to allocated one more shared window do not give an error message, but the 1^st attempt to use that allocated shared array results in a bus error (because the shared array has not been allocated correctly).

The problem is independent of the number of MPI-processes started by mpirun on the node (I used only 1 node)

   Example:     N=      100 000   à  bus error occurred at iwin=288   (i.e. the allocation of the 288-th shared window had failed)

                         N=   1 000 000   à  bus error occurred at iwin=  30

                         N=   5 000 000   à  bus error occurred at iwin=    6

                         N= 28 000 000   à  bus error occurred at iwin=    2

                         N= 30 000 000   à  bus error occurred at iwin=    1   (i.e. already the 1^st allocation failed)

The node on the cluster has 8 Nehalem cores, and had a free memory of 10 GB, and I was the only user on it. I used the INTEL-13 and also the INTEL-14 compiler for compiling the example program.

       mpiifort -O0 -debug -traceback -check -fpe0 sharedmemtest.f90

       mpirun -binding -prepend-rank -ordered-output -np 4 ./a.out

If it is helpful for you, I could send you the source code of the program.

It seems to me, that there is an internal storage limitation in the implementation of the MPI-3 shared memory feature in INTEL-MPI-5.0 . I cannot use INTEL-MPI in my real CFD-code with that limitation, because in case of very large grids the total storage allocated simultaneously by the shared windows can exceed 10 GB.

Greetings to you all

 Michael R.

[James_T_Intel]

This is a known bug at this time, and we are working to correct it.

[Qinghua_W_]

James,

Is there a limit on the size of MPI-3 shared memory? In my tests, it seems that the maximum size is about 4GB for each shared window. The total number of windows cannot exceed 16381. Please confirm. I used intel MPI 2017.1.143, and all the tests were performed on HP workstation with 16 cores and 64GB RAM.

Thanks,

Qinghua 

[James_T_Intel]

I'm checking with our engineering team regarding any limits set.

[James_T_Intel]

We do not have any specific limits set for the size of a window.  There is a limit of 16381 for the number of windows, due to a limitation on the number of communicators.

[Rafael_N_]

James,

Sorry for insisting on the topic of shared windows' individual size limit, raised by Qinghua W., but I've been facing similar difficulties. According to my tests, when using MPI_Win_allocate_shared, I'm not able to allocate anything beyond the limit of 4GB for a single MPI shared window, just like what was previously reported by Qinghua W.

In my test code, I'm just trying to allocate an array of (mySize x nProcs) floats in total, where "mySize" is the number of floats allocated by each MPI process and nProcs is the number processes.

    float* myPtrA;    
    MPI_Win_allocate_shared( mySize*sizeof(float), sizeof(float), MPI_INFO_NULL, MPI_COMM_WORLD, &myPtrA, &winA );   

No mattering how many MPI processes I have, the total amount of allocated memory in the window "winA" cannot exceed the limit of 4GB, or i'll get an error. In this example, I can allocate up to 1073741823 floats (i.e. mySize x nProcs = 1073741823). When trying to allocate 1073741824 floats, I get the following error:

Fatal error in PMPI_Win_allocate_shared: Other MPI error, error stack:
PMPI_Win_allocate_shared(168).........: MPI_Win_allocate_shared(size=536870912, MPI_INFO_NULL, MPI_COMM_WORLD, baseptr=00000000002DFA00, win=00000000002DFA10) failed
MPID_Win_allocate_shared(232).........:
MPIDI_CH3U_Win_allocate(230)..........:
MPIDI_CH3I_Win_allocate_shm(217)......:
MPIU_SHMW_Seg_create_and_attach(902)..:
MPIU_SHMW_Seg_create_attach_templ(783): unable to attach to shared memory - MapViewOfFile (...).

I'm using Intel(R) MPI Library for Windows* OS, Version 5.0 Update 1 Build 20140709.

Thanks in advance.
 

[James_T_Intel]

Using Intel® MPI Library 2017 Update 2, I am able to allocate up to the maximum memory my system has available.  Please try with this version.