topic Indeed, I confirmed your in Intel® MPI Library

MPI_File_write_shared problem

stefano_t_ — Wed, 12 Apr 2017 22:41:50 GMT

Hi,

there seems to be a problem with some versions of intelmpi and file access with mpi shared file pointers. The files are not written correctly. We are using intelmpi 2017 on a cluster with gpfs filesystem. The linux kernel version is 3.10.0-327.36.3.el7.x86_64

Here is a code that reproduce the problem.

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char *argv[])
{
  char string[256];
  char file_name[] = "output";
  int count, slength;
  int open_error;
  int rank;
  MPI_File fh;
  MPI_Status status;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  sprintf(string,"Rank : %d\n",rank);
  slength=strlen(string);

  open_error = MPI_File_open(MPI_COMM_WORLD, file_name, MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &fh);
  if(open_error!=MPI_SUCCESS)
  {
    fprintf(stderr,"Error opening file\n");
    MPI_Abort(MPI_COMM_WORLD,open_error);
  }
  MPI_File_write_shared(fh, string, slength, MPI_CHAR, &status);
  MPI_Get_count(&status,MPI_CHAR,&count);
  if(slength!=count)
  {
    fprintf(stderr,"rank %d: slength=%d , count=%d \n", rank, slength, count);
  }

  MPI_File_close(&fh);
  MPI_Finalize();
  return 0;
}

One example of output is:

$ mpirun -np 10 ./mpi_shared ; cat output | sort -n -k 3
Rank : 2
Rank : 8
Rank : 9

i.e files are truncated.

Instead with the options:

I_MPI_EXTRA_FILESYSTEM=on
I_MPI_EXTRA_FILESYSTEM_LIST=gpfs

the mpi api seems to work, at least with this use case.

I also tested this issue with intelmpi 5.0.3, 5.1.1 and 5.1.3 obtaining the same results as of intel 2017.

It seems that, if the filesystem is not explicitly specified by means of the I_MPI_EXTRA_FILESYSTEM variables the semantic of MPI_File_write_shared is not compliant with the mpi standard.

Is it correct for you ?

Thanks.

Stefano

So what you are reporting is

Mark_L_Intel — Sat, 15 Apr 2017 00:21:17 GMT

So what you are reporting is that the portable mechanism for synchronizing access to a shared file pointer is not implemented. And the shared file pointer access only works properly if one loads GPFS drivers. I would need to check on this.

Indeed, I confirmed your

Mark_L_Intel — Tue, 18 Apr 2017 00:10:21 GMT

Indeed, I confirmed your findings with our engineering team. They might be looking to implement a more portable solution in future releases and your input was very valuable.

Hi Mark,

stefano_t_ — Tue, 18 Apr 2017 08:24:33 GMT

Hi Mark,

many thanks you for your attention.

Stefano

Hi Stefano,

Mark_L_Intel — Wed, 19 Apr 2017 06:23:42 GMT

Hi Stefano,

I filed internal ticket to track this issue.

Thanks,

Mark

Hi, @Mark L.

Cheng__Di — Wed, 21 Feb 2018 16:31:54 GMT

Hi, @Mark L.

I also encounterred the same problem on the HPC with GPFS and intel MPI. I tried to use

I_MPI_EXTRA_FILESYSTEM=on
I_MPI_EXTRA_FILESYSTEM_LIST=gpfs

It shows the following error:

[dic17007@cn03 utils]$ srun -n 24 -p debug -t 30 python utils_di.py
srun: job 1443052 queued and waiting for resources

srun: job 1443052 has been allocated resources
[2] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[7] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[8] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[3] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[14] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[22] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[15] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[10] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[12] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[16] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[17] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[6] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[5] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[13] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[23] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[1] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[4] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[0] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[20] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[19] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[11] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[18] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[21] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func
[9] ERROR - ADIO_Init(): Can't load libmpi_gpfs.so library: /apps2/intelics/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpi_gpfs.so: undefined symbol: ADIOI_IO_Thread_Func

Do you have any suggestions to circumvent this problem?