topic Hi. in Intel® MPI Library

MPI-IO error when running on lustre with a high number of stripes and processes

Wencan_W_ — Wed, 27 Feb 2013 14:59:09 GMT

Hi,

I'm trying to run pNetCDF on lustre. The test code and pNetCDF library are both compiled with intel mpi library v4.0.2. Our lustre file system has 40 OSTs.

When running with stripes = 1 or processes = 32, the test codes works well and can output data correctly.

However, when I set stripe = 40 and run with 64 processes, the test code crashed as :

rank 19 in job 1 c25b09_39645 caused collective abort of all ranks
exit status of rank 19: killed by signal 9

The test code is attacted. Thank you in advance.

I use GDB to debug and got

Wencan_W_ — Thu, 28 Feb 2013 03:22:56 GMT

I use GDB to debug and got the error message :

Program received signal SIGFPE, Arithmetic exception.
34: 0x00002aaac33327e0 in ADIOI_LUSTRE_Get_striping_info ()
34: from /apps/intel/impi/4.0.2.003/intel64/lib/libmpi_lustre.so

It seems to be the same with http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-September/007947.html.

Is it a bug in Intel mpi library v4.0.2, And has it been fixed in new version?

Hi Wencan,

James_T_Intel — Thu, 28 Feb 2013 20:30:10 GMT

Hi Wencan,

I can't find any indication of this being a known issue. There is an issue related to Lustre in the latest version that might cause a problem for you (undefined symbol in one of our libraries). I would recommend trying version 4.0.3 first, and 4.1.0.030 if 4.0.3 does not work. Please let me know if you try any and what the results are.

Can you please attach your test code? It did not get properly attached to the first post.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Thanke you for your help.

Wencan_W_ — Fri, 01 Mar 2013 02:24:04 GMT

Thanke you for your help.

Hi Wencan,

James_T_Intel — Tue, 05 Mar 2013 19:38:44 GMT

Hi Wencan,

I am only able to set striping up to 18 on the cluster I am using. At 18 stripes, I am unable to reproduce this behavior. Please run with I_MPI_DEBUG=5 and send the output.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Hi.

Wencan_W_ — Wed, 06 Mar 2013 05:50:43 GMT

Hi.

I ran with 64 processes and got the output, which is attacted.

The error occurs when the program runs in loop 1 and outputs the 2nd netCDF file.　

Thank you for your help.

Hi Wencan,

James_T_Intel — Thu, 14 Mar 2013 15:41:41 GMT

Hi Wencan,

It appears you are using LSF* as your job scheduler. We do have some known issues with LSF*. I don't think they're related to this case, but can you try a few things just in case? First, try running in an interactive job. Due to one of the known issues, you will probably need to add

[plain]-genv LD_LIBRARY_PATH $LD_LIBRARY_PATH[/plain]

to your mpirun command. Also, please try running completely outside of LSF*.

Could you also send the output from stderr (for a failing job)? It would be best if you have stdout and stderr in the same file.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Hi,

Wencan_W_ — Mon, 18 Mar 2013 02:08:00 GMT

Hi,

I ran with mpiexec comman as:

[bash]mpiexec -genv I_MPI_EXTRA_FILESYSTEM on -genv I_MPI_EXTRA_FILESYSTEM_LIST lustre -genv I_MPI_DEBUG 5 -genv LD_LIBRARY_PATH $LD_LIBRARY_PATH -n $num ./perform_test_pnetcdf $x_proc $y_proc $output &> $out[/bash]

and got the new output.