Slow parallel IO with Intel MPI

[spg]

I wrote a piece code to add take advantage of the parallel file system on HPC, using the MPI-IO.

The code performs well (parallel version of the I/O faster than the sequential version) for GNU compiler with MPT, PGI compiler with MPT.

With Intel compiler andintel compiler/MPI 2020, the parallel version of the IO is 50 times slower than the sequential counter part (gather and save by few processes).

Previously with intel compiler/MPI 2018, by setting some environmental variables (I_MPI_EXTRA_FILESYSTEM="1" and I_MPI_EXTRA_FILESYSTEM_FORCE="lustre" ), I was able to get Intel compiler and intel MPI to perform slightly better on parallel I/O. With intel 2020.2.254, It doeas not work.

Any idea of what to do to get the paralelel I/O to be faster than the serial I/O (collect and save by few processes)?

[ShivaniK_Intel]

Hi,

Thanks for reaching out to us.

Could you please provide the reproducer code so that we can investigate more on your issue?

From Intel MPI Library 2019, Parallel file systems (GPFS, Lustre, Panfs) are supported natively, no need to explicitly set these environment variables(I_MPI_EXTRA_FILESYSTEM*, I_MPI_LUSTRE* variables).

For more details please refer to the below link.

https://software.intel.com/content/www/us/en/develop/articles/intel-mpi-library-release-notes-linux.html#inpage-nav-3-1

Thanks & Regards

Shivani

[spg]

Thanks Shivani for looking into this.

As I try to build a minimal code to reproduce the issue, I found one part of the problem. A data processing performed before sending data to disk that is automatically optimized gnu and pgi regardless of optimization on, is not optimized by INTEL unless optimization is turned on.

When I turned the optimization on for INTEL, it reduced the factor of 50 significantly to about 2. I will double check everything.

Thanks again.

[ShivaniK_Intel]

Hi,

Could you please provide the reproducer code so that we can investigate more on your issue?

Thanks & Regards

Shivani

[ShivaniK_Intel]

Hi,

As we didn't hear back from you, Is your issue resolved? If not, please provide the details that have been asked in my previous post.

Thanks & Regards,

Shivani

[spg · Accepted solution]

@ShivaniK_Intel , Thank you for your continuous interest in this issue. I got it down to similar time as other compiler.

Thank you again.

[ShivaniK_Intel]

Hi,

We are glad that your issue has been resolved. We will no longer respond to this thread. If you require any additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only.

Have a good day

Thanks & Regards

Shivani