topic Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern in Intel® oneAPI Math Kernel Library

MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or internal MPI fault?

Filippo_Spiga — Tue, 08 Sep 2009 20:14:58 GMT

Hi all, I have a problem with MKL FFT Cluster routines. I have followed the code example "1D In-place Cluster FFT Computations" and I have tried to run a simple parallel job on two different clusters equipped with different processors (Intel Xeon CPU X5570 @ 2.93GHz and Dual-Core AMD Opteron Processor 2218), different resource managers (SLURM and LSF) and different versions of the MKL library (10.0.010 and 10.0.2). The underline MPI runtime environment is Open MPI 1.3.2, compiled using INTEL Compiler 10.1 . On both cluster, the test program crashes at the same point with the same error.

This is the output
$ cat out
I'm 3 and I have passed STEP 1
I'm 0 and I have passed STEP 1
I'm 1 and I have passed STEP 1
I'm 2 and I have passed STEP 1

TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME
===== ========== ================ ======================= ===================
00000 node0027 ./x.fft_mpi_mkl Exit (5) 09/08/2009 21:53:25
00001 node0028 ./x.fft_mpi_mkl Exit (5) 09/08/2009 21:53:25
00002 node0023 ./x.fft_mpi_mkl Exit (5) 09/08/2009 21:53:25
00003 node0021 ./x.fft_mpi_mkl Exit (5) 09/08/2009 21:53:25

and this--------------------------------------------------------------------------
[node0021:05836] *** An error occurred in MPI_comm_size
[node0021:05836] *** on communicator MPI_COMM_WORLD
[node0021:05836] *** MPI_ERR_COMM: invalid communicator
[node0021:05836] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
[node0023:25571] *** An error occurred in MPI_comm_size
[node0027:30416] *** An error occurred in MPI_comm_size
[node0023:25571] *** on communicator MPI_COMM_WORLD
[node0023:25571] *** MPI_ERR_COMM: invalid communicator
[node0023:25571] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
[node0027:30416] *** on communicator MPI_COMM_WORLD
[node0027:30416] *** MPI_ERR_COMM: invalid communicator
[node0027:30416] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
[node0028:15920] *** An error occurred in MPI_comm_size
[node0028:15920] *** on communicator MPI_COMM_WORLD
[node0028:15920] *** MPI_ERR_COMM: invalid communicator
[node0028:15920] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------

I have compiled both programs using this command
$ mpicc -openmp -w test.c -Wl,--start-group $MKL_INCLUDE $MKL_LIB/libmkl_cdft_core.a $MKL_LIB/libmkl_blacs_openmpi_lp64.a $MKL_LIB/libmkl_intel_lp64.a $MKL_LIB/libmkl_intel_thread.a $MKL_LIB/libmkl_core.a -Wl,--end-group -L$MKL_LIB -liomp5 -lpthread -lm -o x.fft_mpi_mkl ("lib/em64t" in both case)

The program, with stupid additions to print debug informations, is attached to this post.
What's wrong? Can the problem be related to Open MPI?

Thank you very much in advance ,
Cheers!

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Dmitry_B_Intel — Wed, 09 Sep 2009 08:44:07 GMT

Hi Filippo,
Have you checked that compiler doesn't complaint on undeclared malloc? If undeclared it is assumed to return int, possibly cutting pointers to 32bit. This might be the cause of the failure you see.
Thanks
Dima

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Filippo_Spiga — Wed, 09 Sep 2009 09:04:54 GMT

Quoting - Dmitry Baksheev (Intel)

Hi Dmitry,
are you suggesting to compile using "-m64" flag explicitly?

I have just tried to add "-m64" flag but nothing changes. I think that the problem can be related to OpenMPI. I switch from Open MPI 1.3.2 to OpenMPI 1.2.6 but the problem persist!

>$ mpicc -openmp -m64 test.c -Wl,--start-group $MKL_INCLUDE $MKL_LIB/libmkl_cdft_core.a $MKL_LIB/libmkl_blacs_openmpi_lp64.a $MKL_LIB/libmkl_intel_lp64.a $MKL_LIB/libmkl_intel_thread.a $MKL_LIB/libmkl_core.a -Wl,--end-group -L$MKL_LIB -liomp5 -lpthread -lm -o x.fft_mpi_mkl

>$ ldd x.fft_mpi_mkl
libiomp5.so => /opt/MKL/10.0.2/intel--10.1/lib/em64t/libiomp5.so (0x00002aaaaaac7000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaacc6000)
libimf.so => /opt/intel/fce/10.1.011/lib/libimf.so (0x00002aaaaaee0000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab243000)
libmpi.so.0 => /opt/openmpi/1.2.6/intel--10.1/lib/libmpi.so.0 (0x00002aaaab4c6000)
libopen-rte.so.0 => /opt/openmpi/1.2.6/intel--10.1/lib/libopen-rte.so.0 (0x00002aaaab854000)
libopen-pal.so.0 => /opt/openmpi/1.2.6/intel--10.1/lib/libopen-pal.so.0 (0x00002aaaabb61000)
libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002aaaabdd5000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaabfe0000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaac1ea000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaac3ee000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaac606000)
libguide.so => opt/MKL/10.0.2/intel--10.1/lib/em64t/libguide.so (0x00002aaaac80a000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003081c00000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaac971000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
libsvml.so => /opt/intel/fce/10.1.011/lib/libsvml.so (0x00002aaaaccc2000)
libintlc.so.5 => /opt/intel/fce/10.1.011/lib/libintlc.so.5 (0x00002aaaace49000)

[node0020:22164] *** An error occurred in MPI_comm_size
[node0020:22164] *** on communicator MPI_COMM_WORLD
[node0020:22164] *** MPI_ERR_COMM: invalid communicator
[node0020:22164] *** MPI_ERRORS_ARE_FATAL (goodbye)
[node0027:04801] *** An error occurred in MPI_comm_size
[node0027:04801] *** on communicator MPI_COMM_WORLD
[node0027:04801] *** MPI_ERR_COMM: invalid communicator
[node0027:04801] *** MPI_ERRORS_ARE_FATAL (goodbye)
[node0019:09289] *** An error occurred in MPI_comm_size
[node0019:09289] *** on communicator MPI_COMM_WORLD
[node0019:09289] *** MPI_ERR_COMM: invalid communicator
[node0019:09289] *** MPI_ERRORS_ARE_FATAL (goodbye)
[node0022:32707] *** An error occurred in MPI_comm_size
[node0022:32707] *** on communicator MPI_COMM_WORLD
[node0022:32707] *** MPI_ERR_COMM: invalid communicator
[node0022:32707] *** MPI_ERRORS_ARE_FATAL (goodbye)

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Dmitry_B_Intel — Wed, 09 Sep 2009 09:16:17 GMT

Filippo,

I am sorry I didn't put it clearly. I noticed that #include is missing in your example, and so I suspect that compiler uses implicit declaration for it, which is 'int malloc()' instead of 'void *malloc(size_t)'.

Thanks
Dima

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Vladimir_Petrov__Int — Wed, 09 Sep 2009 09:54:54 GMT

Hi Filippo,

According to the release notes, MKL currently supports only the versions 1.2.x of OpenMPI.
Do you have a possibility to try this older version?

Best regards,
-Vladimir

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Filippo_Spiga — Wed, 09 Sep 2009 10:26:49 GMT

Dear all,
I have tried using "#include " and different versions of OpenMPI (1.2.5, 1.2.6 and 1.2.7) compiled with icc 10.1. The problem remains the same as above. Only on one of the two cluster I have the possibility to modify or change the OpenMPI version.

If it can be useful, OpenMPI 1.2.7 was compiled using these flags...

export CC=icc
export CXX=icc
export F77=ifort
export F90=ifort
export FC=$F90
export CFLAGS="-O2"
export CXXFLAGS="$CFLAGS -lstdc++"
export FFLAGS="-O2"
export FCFLAGS="-O2"
export LDFLAGS="-O2"
export F77FLAGS="-02"

./configure --prefix=... --disable-ipv6 --enable-static --with-openib=/usr/local/ofed --with-openib-libdir=/usr/local/ofed/lib64 --with-mpi-f90-size=medium --with-io-romio-flags="--with-filesystems=ufs" --enable-mpi-threads --enable-cxx-exceptions

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Vladimir_Petrov__Int — Thu, 10 Sep 2009 09:29:31 GMT

Hi Filippo,

While I am trying to reproduce your issue, could you please check the following two potential problems with compiling your test:
1. The MKL_INCLUDE directory does not seem to be placed correctly on the compile-link line - please prepend it with "-I" and place outside the group-clauses;
2. ldd reports dependecies on two Intel threading libraries - both libguide.so and libiomp5.so. I would strongly recommend that you use exactly one - that which was used to build Open MPI.

Best regards,
-Vladimir

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Filippo_Spiga — Thu, 10 Sep 2009 10:08:04 GMT

Quoting - Vladimir Petrov (Intel)

1. The MKL_INCLUDE directory does not seem to be placed correctly on the compile-link line - please prepend it with "-I" and place outside the group-clauses;
2. ldd reports dependecies on two Intel threading libraries - both libguide.so and libiomp5.so. I would strongly recommend that you use exactly one - that which was used to build Open MPI.

1. MKL_INCLUDE environment variable includes "-I"
$ echo $MKL_INCLUDE
-I/opt/MKL/10.0.2/intel--10.1/include
$ ls -1 /opt/MKL/10.0.2/intel--10.1/includefftw
i_malloc.h
mkl_blas.f90
mkl_blas.fi
mkl_blas.h
mkl_cblas.h
[...and so on...]

2. I have tried "-lguide" instead of "-liomp5" ...

$ ldd x.fft_mpi_mkl
libguide.so => /opt/MKL/10.0.2/intel--10.1/lib/em64t/libguide.so (0x00002aaaaaac7000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaacc6000)
libmpi.so.0 => /opt/openmpi/1.2.7/intel--10.1/lib/libmpi.so.0 (0x00002aaaaaee0000)
libopen-rte.so.0 => /opt/openmpi/1.2.7/intel--10.1/lib/libopen-rte.so.0 (0x00002aaaab285000)
libopen-pal.so.0 => /opt/openmpi/1.2.7/intel--10.1/lib/libopen-pal.so.0 (0x00002aaaab59a000)
libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002aaaab80f000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaaba1b000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaabc24000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaabe28000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaac041000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaac244000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003081c00000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaac4c8000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
libimf.so => /opt/intel/fce/10.1.011/lib/libimf.so (0x00002aaaac818000)
libsvml.so => /opt/intel/fce/10.1.011/lib/libsvml.so (0x00002aaaacb7a000)
libintlc.so.5 => /opt/intel/fce/10.1.011/lib/libintlc.so.5 (0x00002aaaacd02000)

and also with Open MPI 1.2.5

[cin8310a@node1310 test_mkl_fft]$ ldd x.fft_mpi_mkl
libguide.so => /opt/MKL/10.0.2/intel--10.1/lib/em64t/libguide.so (0x00002aaaaaac7000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaacc6000)
libimf.so => /opt/intel/fce/10.1.011/lib/libimf.so (0x00002aaaaaee0000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab243000)
libmpi.so.0 => /opt/openmpi/1.2.5/intel/10.1/lib/libmpi.so.0 (0x00002aaaab4c6000)
libopen-rte.so.0 => /opt/openmpi/1.2.5/intel/10.1/lib/libopen-rte.so.0 (0x00002aaaab854000)
libopen-pal.so.0 => /opt/openmpi/1.2.5/intel/10.1/lib/libopen-pal.so.0 (0x00002aaaabb61000)
libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002aaaabdd5000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaabfe0000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaac1ea000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaac3ee000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaac606000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003081c00000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaac80a000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
libsvml.so => /opt/intel/fce/10.1.011/lib/libsvml.so (0x00002aaaacb5a000)
libintlc.so.5 => /opt/intel/fce/10.1.011/lib/libintlc.so.5 (0x00002aaaacce2000)

No changes :-(

Thank you for the support

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Vladimir_Petrov__Int — Fri, 11 Sep 2009 02:55:53 GMT

Hi Filippo,

I managed to reproduce your issue on my local machine. It looks like it is caused by incompatibility between MKL BLACS and the options you use to build Open MPI.

BTW, is F77FLAGS really set to "-02" (where "0" is zero)?

Best regards,
-Vladimir

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Filippo_Spiga — Fri, 11 Sep 2009 07:28:42 GMT

Quoting - Vladimir Petrov (Intel)

BTW, is F77FLAGS really set to "-02" (where "0" is zero)?

Yes. If you are able to suggest me the flags to have full compatibility with MKL I will try to recompile Open MPI and make other tests

Thanks a lot!

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Vladimir_Petrov__Int — Fri, 11 Sep 2009 08:59:04 GMT

Hi Filippo,

I finally figured out what the problem is.
Since Open MPI considers MPI_COMM_WORLD to be a pointer it turns out to be 64-bit long. Whereas Cluster FFT was designed in times where sizeof(MPI_Comm) used to be 32-bit. In order to work correctly with Open MPI you just need to wrap the communicator as follows:

DftiCreateDescriptorDM(MPI_Comm_c2f(MPI_COMM_WORLD),&desc,DFTI_DOUBLE,DFTI_COMPLEX,1,len);

Best regards,
-Vladimir

P.S. I hope you will agree that later is better than never.

Re: MKL FFT Cluster: real DFTI_BAD_DESCRIPTOR problem or intern

Filippo_Spiga — Fri, 11 Sep 2009 09:06:55 GMT

Quoting - Vladimir Petrov (Intel)

Since Open MPI considers MPI_COMM_WORLD to be a pointer it turns out to be 64-bit long. Whereas Cluster FFT was designed in times where sizeof(MPI_Comm) used to be 32-bit. In order to work correctly with Open MPI you just need to wrap the communicator as follows:

DftiCreateDescriptorDM(MPI_Comm_c2f(MPI_COMM_WORLD),&desc,DFTI_DOUBLE,DFTI_COMPLEX,1,len);

Great, it works!

Quoting - Vladimir Petrov (Intel)

P.S. I hope you will agree that later is better than never.

I agree with you (-: Thank you very much again for your support!

Regards