MPI_File_write_shared problem

[stefano_t_]

Hi,

there seems to be a problem with some versions of intelmpi and file access with mpi shared file pointers. The files are not written correctly. We are using intelmpi 2017 on a cluster with gpfs filesystem. The linux kernel version is 3.10.0-327.36.3.el7.x86_64

Here is a code that reproduce the problem.

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char *argv[])
{
  char string[256];
  char file_name[] = "output";
  int count, slength;
  int open_error;
  int rank;
  MPI_File fh;
  MPI_Status status;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  sprintf(string,"Rank : %d\n",rank);
  slength=strlen(string);

  open_error = MPI_File_open(MPI_COMM_WORLD, file_name, MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &fh);
  if(open_error!=MPI_SUCCESS)
  {
    fprintf(stderr,"Error opening file\n");
    MPI_Abort(MPI_COMM_WORLD,open_error);
  }
  MPI_File_write_shared(fh, string, slength, MPI_CHAR, &status);
  MPI_Get_count(&status,MPI_CHAR,&count);
  if(slength!=count)
  {
    fprintf(stderr,"rank %d: slength=%d , count=%d \n", rank, slength, count);
  }

  MPI_File_close(&fh);
  MPI_Finalize();
  return 0;
}

One example of output is:

$ mpirun -np 10 ./mpi_shared ; cat output | sort -n -k 3
Rank : 2
Rank : 8
Rank : 9

i.e files are truncated.

Instead with the options:

I_MPI_EXTRA_FILESYSTEM=on
I_MPI_EXTRA_FILESYSTEM_LIST=gpfs

the mpi api seems to work, at least with this use case.

I also tested this issue with intelmpi 5.0.3, 5.1.1 and 5.1.3 obtaining the same results as of intel 2017.

It seems that, if the filesystem is not explicitly specified by means of the I_MPI_EXTRA_FILESYSTEM variables the semantic of MPI_File_write_shared is not compliant with the mpi standard.

Is it correct for you ?

Thanks.

Stefano

[Mark_L_Intel]

So what you are reporting is that  the portable mechanism for synchronizing access to a shared file pointer is not implemented. And the shared file pointer access only works properly if one loads GPFS drivers. I would need to check on this.

[Mark_L_Intel]

Indeed, I confirmed your findings with our engineering team. They might be looking to implement a more portable solution in future releases and your input was very valuable.

[stefano_t_]

Hi Mark,

many thanks you for your attention.

Stefano

[Mark_L_Intel]

Hi Stefano,

   I filed internal ticket to track this issue.

Thanks,

Mark

[Cheng__Di]

Hi, @Mark L.

I also encounterred the same problem on the HPC with GPFS and intel MPI. I tried to use 

I_MPI_EXTRA_FILESYSTEM=on
I_MPI_EXTRA_FILESYSTEM_LIST=gpfs

It shows the following error:

[dic17007@cn03 utils]$ srun -n 24 -p debug -t 30 python utils_di.py
srun: job 1443052 queued and waiting for resources

srun: job 1443052 has been allocated resources
[2] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[7] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[8] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[3] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[14] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[22] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[15] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[10] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[12] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[16] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[17] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[6] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[5] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[13] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[23] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[1] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[4] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[0] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[20] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[19] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[11] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[18] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[21] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[9] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func

Do you have any suggestions to circumvent this problem?