How to correctly compile Quantum Espresso with intel MKL especially intel FFTW

[balabi_b_]

I wish to exploit the power of intel mkl in Quantum Espresso, which is a popular open-source first-principle calculation suite.

I found this tutorial named "Quantum ESPRESSO* for Intel® Xeon Phi™ Coprocessor", in the intel document, which is quite in detail, illustrate step by step the setup flow for the software using intel compiler and mkl. Though I don't have Intel® Xeon Phi, But I guess the setup also works for ordinary multicore computer, am I right?

But I encountered problems following that installation tutorial. This maybe due to that the version of my parallel studio is different from that in the tutorial. My version of parallel studio is 2016.2.181.  After completely follow the steps, I got this when running "make pw"

ifort: command line warning #10006: ignoring unknown option '-/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/include/fftw'

 I notice the FCFLAGS is exported as

export FCFLAGS="-O3 -xAVX -fno-alias -ansi-alias -g -mkl -$MKLROOT/include/fftw"

here -$MKLROOT/include/fftw is not right, because apparently the ifort doesn't know this option.

Investigate further, I found the FFT_LIBS="-L$MKLROOT/intel64" is also wrong, because in version 2016.2.181, there is no intel64 folder under $MKLROOT. So I changed it to $MKLROOT/lib/intel64_lin. However, this still not working. Also after running ./configure --enable-openmp --enable-parallel, there is actually no FFT_LIBS detected. The configure finally shows

The following libraries have been found:
  BLAS_LIBS=   -lmkl_intel_thread -lmkl_core
  LAPACK_LIBS=-lmkl_blacs_intelmpi_lp64
  SCALAPACK_LIBS=-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
  FFT_LIBS=
Please check if this is what you expect.

Why FFT is not being detected? What is wrong? How to compile it correctly?

[TimP]

An include path is specified by -I  examples are given by mkl link advisor.

[balabi_b_]

Tim P. wrote:

An include path is specified by -I  examples are given by mkl link advisor.

Thank you! Change "-$MKLROOT/include/fftwto "-I$MKLROOT/include/fftw" seems working. When doing "make", the compiler recognized the option. However in the output info of "./configure", there is still no FFT_LIBS detected.

Also the new problem emerge. I got a huge list of undefined reference, which seems to mean all kinds of library is not properly set

mpiifort -static-intel  -qopenmp -o pw.x \
       pwscf.o  libpw.a ../../Modules/libqemod.a ../../FFTXlib/libqefft.a ../../flib/ptools.a ../../flib/flib.a ../../clib/clib.a ../../iotk/src/libiotk.a -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lmkl_blacs_intelmpi_lp64  -lmkl_intel_thread -lmkl_core   
libpw.a(update_pot.o): In function `extrapolate_wfcs':
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/PW/src/update_pot.f90:716: undefined reference to `zgesvd_'
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/PW/src/update_pot.f90:782: undefined reference to `zgesvd_'
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/PW/src/update_pot.f90:791: undefined reference to `zgemm_'

/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/FFTXlib/./fft_scalar.FFTW3.f90:128: undefined reference to `dfftw_execute_dft_'
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/FFTXlib/./fft_scalar.FFTW3.f90:124: undefined reference to `dfftw_execute_dft_'
../../FFTXlib/libqefft.a(fft_scalar.o): In function `fft_scalarcft_1z_mp_init_plan_':
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/FFTXlib/./fft_scalar.FFTW3.f90:153: undefined reference to `dfftw_cleanup_threads_'
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/FFTXlib/./fft_scalar.FFTW3.f90:154: undefined reference to `fftw_init_threads'

../../flib/flib.a(dylmr2.o): In function `dylmr2':
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/flib/dylmr2.f90:57: undefined reference to `dcopy_'
/fs10/home/qhw_wang/HPC-nj/quantum_espresso/espresso-5.3.0/flib/dylmr2.f90:81: undefined reference to `daxpy_'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_real'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `ssteqr_'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `dsyr2k_'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `cscal_'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_op_sum'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `caxpy_'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_byte'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_dblprec'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `dsteqr_'
/fs00/software/intel/ps2016u2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `cbdsqr_'

What should I do?

[Evgueni_P_Intel]

Could you please change -lmkl_blacs_openmpi_lp64 to -lmkl_intel_lp64 ?

[TimP]

If you don't care to study the recommendations of mkl link advisor, you must avoid static link.

[balabi_b_]

Evgueni Petrov aka espetrov (Intel) wrote:

Could you please change -lmkl_blacs_openmpi_lp64 to -lmkl_intel_lp64 ?

Thank you very much!

Now I use

./configure MPIF90=mpiifort F90=mpiifort F77=mpiifort --enable-openmp --enable-parallel --with-scalapack=intel

And the make process is successful now.

But the problem is I still can't correctly invoke MKL's FFT. Quantum espresso still use its own FFT library. I tried several ways to set the library, all failed

[Ying_H_Intel]

Hi 

I think you are right,   the setup also works for ordinary multicore computer. 

I notice in that article  use two place to control  the FFT 

export FFT_LIBS="-L$MKLROOT/intel64"

  

14	./configure --enable-openmp --enable-parallel

4. Edit the make.sys file in the top package directory and modify the DFLAGS variable in order to enable hybrid MPI+openmp mode, fftw3, and scalapack:

iew source

print?

1	DFLAGS = -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -D__OPENMP

2	BLAS_LIBS = -lmkl_intel_thread -lmkl_core

3	LAPACK_LIBS = -lmkl_blacs_intelmpi_lp64

[Ying_H_Intel]

So about the FFTW switch may include some trick.

generally,

export FFT_LIBS="-L$MKLROOT/intel64"   it is only mkl path 

Then It should be normal after configure, then FFT_LIB would be found. 

Maybe you can try 

FFT_LIBS="-L$MKLROOT/intel64 -lmkl"

and add 

 Edit the make.sys file in the top package directory and modify the DFLAGS variable in order to enable hybrid MPI+openmp mode, fftw3, and scalapack:

iew source

print?

1
DFLAGS = -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -D__OPENMP

...

and see if it can work?

Best Regards,

Ying

[Hans_P_Intel]

Please have a look at https://software.intel.com/en-us/articles/quantum-espresso-for-the-intel-xeon-phi-processor, which not only focuses on the Intel® Xeon Phi™ x200 processor family (previously code-named "Knights Landing" or "KNL") but seamlessly applies to any Intel processor. The aforementioned article helps to build an optimized binary, which also yields optimized performance.

[Vahid_A_2]

I have successfully compiled QE-6.0 using the following modules:

1) intel/14.0.2.144   2) intel/ompi/1.6.5

To compile against the MKL FFT library (DFTI), I use the following:

./configure --prefix=/home/$username/bin/qe-6.0  --enable-shared  --enable-parallel  --with-scalapack  CC=mpicc  F77=mpif90   FC=mpif90

If I do not include the CC, F77 and FC options, the code compiles against the native QE FFTW library.

Vahid