MPI mpiifort netcdf parallel Error write file

[Christophe__Yohia]

Hello

I work with Intel Compiler version parallel_studio_xe_2019.3.062 

I  compile the version of NetCDF with parallel option. 
The version use

hdf5-1.10.5
Netcdf C 4-6-3 
Netcdf F90 4.4.5



The configuration of Netcdf:
--has-dap       -> yes
  --has-dap2      -> yes
  --has-dap4      -> yes
  --has-nc2       -> yes
  --has-nc4       -> yes
  --has-hdf5      -> yes
  --has-hdf4      -> no
  --has-logging   -> no
  --has-pnetcdf   -> no
  --has-szlib     -> no
  --has-cdf5      -> yes
  --has-parallel4 -> yes
  --has-parallel  -> yes



I run my code on HPC (800cores ,  60nodes) with SLURM for the job soumission. (1 node=24cores), OMNI-PATH network.

The storage disk is mounted with NFS on each node .

The option of NFS are nfs vers=3,soft,nolock,noacl 0 0

I check the permission on filesystem and user.

The error message indicate Permission Denied when I attempt to use nf90_create on several node (>=2).

For some test, I write a little program very simple to write Netcdf File with parallel option. (at the end of this message)

When I run the program on 1 node (24cores), the program create a file and write data.
When I run the program on 2 node or more (>=48),  a file Netcdf is created but the size is 0ko, then the program failed with ERROR 13 Permission denied.
If I keep this file, I run again the program without deleting the file, the program run with successfully.

The error is mentioned on function nf90_create.
nf90_create(filename ,IOR(NF90_NETCDF4,NF90_MPIIO),ncid,comm=MPI_COMM_WORLD, info=MPI_INFO_NULL)

I try several option of cmode (NF90_CLOBBER, NF90_NOCLOBBER, NF90_SHARE) .

It seems all processors don’t have a permission to write file at the same time or the file doesn't exist on some processus.

I compile the program with this option
mpiifort testwrite.f90 -I${PATH_NETCDF}/include/ -L${PATH_NETCDF}lib/ -lnetcdff -lnetcdf -lcurl -lhdf5_hl -lhdf5 -lz -o testwrite

With SLURM, I submit my job like this 
mpirun -bootstrap slurm -genv I_MPI_FABRICS shm:ofi -genv I_MPI_TMI_PROVIDER -psm2 -np $SLURM_NTASKS ${path}/testwrite
Or
mpirun -np $SLURM_NTASKS ${path}/testwrite


Thank you for helping or ideas to solve the problem
Christophe

My program like this
 

program testwrite
USE netcdf
USE mpi

integer :: p, my_rank, ierr
integer :: old_mode,ncid
integer :: ntx_dim_id,nty_dim_id,nvar_dim_id
integer :: ntx_id,nty_id,u_id
integer,dimension(3) :: dims,starts
integer, parameter :: NX = 6, NY = 12
character(len=11) :: filename
character(len=11) ::path

integer, parameter :: NDIMS = 2
integer :: varid, dimids(NDIMS)
integer :: x_dimid, y_dimid, t_dimid
integer :: start(NDIMS), count(NDIMS)
integer :: data_out(NY,NX)
logical :: file_exists


 do x = 1, NX
     do y = 1, NY
        data_out(y, x) = (x - 1) * NY + (y - 1)
     end do
  end do


!Create file
filename="savetest.nc"
call MPI_Init(ierr)
if (ierr /= 0) print *, 'Error in MPI_Init'

call MPI_Comm_rank(MPI_COMM_WORLD, my_rank, ierr)
if (ierr /= 0) print *, 'Error in MPI_Comm_size'

call MPI_Comm_size(MPI_COMM_WORLD, p, ierr)
if (ierr /= 0) print *, 'Error in MPI_Comm_size'
 
 
 status = nf90_create(filename ,IOR(NF90_NETCDF4,NF90_MPIIO),ncid,comm=MPI_COMM_WORLD, info=MPI_INFO_NULL)
 status=nf90_def_dim(ncid, "x", NX, x_dimid)

 status=nf90_def_dim(ncid, "y", NY, y_dimid)
 
  dimids = (/ y_dimid, x_dimid /)
  
  status=nf90_def_var(ncid, "data", NF90_INT, dimids, varid)

  status=nf90_put_var(ncid, varid, data_out)


 status=nf90_ENDDEF(ncid)
 status=nf90_close(ncid)
 call MPI_Finalize(ierr)

end