Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

[yehonatan123f]

Hello,

I am trying to build q-e-7.2-omp5 (https://gitlab.com/QEF/q-e/-/tree/qe-7.2-omp5?ref_type=tags) specifically for Intel GPUs using cmake.

I am running a SPR+4 PVC node on Developer Cloud.  

This is my cmake command (that I run in build dir):

cmake -DCMAKE_C_COMPILER=mpiicx -DCMAKE_Fortran_COMPILER=mpiifx -DCMAKE_C_FLAGS:STRING="-xHOST -qmkl=parallel -fiopenmp -fopenmp-targets=spir64 -I$MKLROOT/include/fftw" -DCMAKE_Fortran_FLAGS:STRING="-xHOST -qmkl=parallel -fiopenmp -fopenmp-targets=spir64 -I$MKLROOT/include/fftw"  -DQE_ENABLE_OPENMP=ON -DQE_ENABLE_OFFLOAD=ON -DQE_FFTW_VENDOR=Intel_DFTI ..

This is the output:

-- The Fortran compiler identification is IntelLLVM 2023.2.0
-- The C compiler identification is IntelLLVM 2023.2.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/intel/oneapi/mpi/2021.10.0/bin/mpiifx - skipped
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/intel/oneapi/mpi/2021.10.0/bin/mpiicx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Setting build type to 'Release' as none was specified
-- Looking for mallinfo
-- Looking for mallinfo - found
-- Enable sanitizer QE_ENABLE_SANITIZER=none
-- C preprocessor used by qe_preprocess_source in qeHelpers.cmake: /usr/bin/cpp
-- Performing Test Fortran_ISYSTEM_SUPPORTED
-- Performing Test Fortran_ISYSTEM_SUPPORTED - Success
-- Found OpenMP_Fortran: -fiopenmp (found version "5.0")
-- Found OpenMP_C: -fiopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0") found components: Fortran C
-- Found MPI_Fortran: /opt/intel/oneapi/mpi/2021.10.0/bin/mpiifx (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: Fortran
-- MPI settings used by CTest
-- Found Git: /usr/bin/git (found suitable version "2.34.1", minimum required is "2.13")
-- Source files are cloned from a git repository.
-- Trying to find LAPACK from Intel MKL
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- Found LAPACK: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;-lm;-ldl
-- Found LAPACK: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;-lm;-ldl
-- Looking for Fortran zhpev
-- Looking for Fortran zhpev - found
-- Installing Wannier90 via submodule
-- Installing MBD via submodule
-- Installing DeviceXlib via submodule
-- Found VendorFFTW: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;-lm;-ldl
-- Found Intel_DFTI library
-- Enabling tests in test-suite
-- generating tests in pw category
-- generating tests in cp category
-- generating tests in ph category
-- generating tests in epw category
-- generating tests in tddfpt category
-- generating tests in hp category
-- Configuring done
-- Generating done
-- Build files have been written to: /home/sdp/q-e/build

The error comes when I run "make": 

[ 5%] Building Fortran object FFTXlib/src/CMakeFiles/qe_fftx.dir/fft_scalar.f90.o
fft_scalar.f90(46): #error: No fft_scalar backend selected for __OPENMP_GPU!

This error is issued from fft_scalar.f90(46) where __ONEMKL is not defined, as you can see (fft_scalar.f90): 

#if defined(__CUDA)
USE fft_scalar_cuFFT
#elif defined(__OPENMP_GPU) && defined(__ONEMKL)
USE fft_scalar_dfti_omp
#elif defined(__OPENMP_GPU) && defined(__HIP)
USE fft_scalar_hipfft
#elif defined(__OPENMP_GPU)
#error No fft_scalar backend selected for __OPENMP_GPU!
#endif

I have tried multiple cmake configurations, nothing seem to solve this issue. Nothing helps to make  __ONEMKL be recognized. 

Just to note, /opt/intel/oneapi/mkl/2023.2.0/include and /opt/intel/oneapi/mkl/2023.2.0/lib/intel64 seem to contain all the needed math files (especially fftw files and mkl_dfti_omp_offload.f90). 

What can be the problem with how I am building the project? 

Thank you. 

[ReemHarel]

I've been working on Quantum espresso for Ponte-vecchio as well, and i also cannot compile it,

[yehonatan123f]

Hey all,

Any idea how to solve this issue?

I am still struggeling in this regard.

Thank you!

[Mark_L_Intel]

I confirmed the issue on internal machine. Let me look into what can cause it.

[Mark_L_Intel]

@yehonatan123f ,

please try the following from the build directory, e.g., <your-directory>/q-e/build

cmake -DQE_FFTW_VENDOR=Intel_DFTI ..

  you should see something like this:

...
-- Found VendorFFTW: /opt/intel/oneapi/mkl/2024.0/lib/libmkl_gf_lp64.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_sequential.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_core.so;-lpthread;-lm;-ldl;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_gf_lp64.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_sequential.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_core.so;-lpthread;-lm;-ldl;-lpthread;-lm;-ldl
-- Found Intel_DFTI library
-- Enabling tests in test-suite

and notice "Found Intel_DFTI library" message above. Then "make" should work.

[Mark_L_Intel]

Hello @yehonatan123f , Have you tried above suggestion? 

[yehonatan123f]

Hey mark,

It still not working. 

sdp@gpunode:~/q-e-qe-7.2-omp5/build$ ll /usr/include/mkl/fftw
total 216
drwxr-xr-x 2 root root  4096 Feb 15 06:58 ./
drwxr-xr-x 5 root root  4096 Feb 15 06:58 ../
-rw-r--r-- 1 root root 11900 Sep 17  2020 fftw.h
-rw-r--r-- 1 root root 37021 Sep 17  2020 fftw3-mpi.f03
-rw-r--r-- 1 root root  9621 Sep 17  2020 fftw3-mpi.h
-rw-r--r-- 1 root root  1902 Sep 17  2020 fftw3-mpi_mkl.h
-rw-r--r-- 1 root root  2777 Sep 17  2020 fftw3.f
-rw-r--r-- 1 root root 54077 Sep 17  2020 fftw3.f03
-rw-r--r-- 1 root root 17945 Sep 17  2020 fftw3.h
-rw-r--r-- 1 root root  3484 Sep 17  2020 fftw3_mkl.f
-rw-r--r-- 1 root root  2023 Sep 17  2020 fftw3_mkl.h
-rw-r--r-- 1 root root 21606 Sep 17  2020 fftw3_mkl_f77.h
-rw-r--r-- 1 root root  1436 Sep 17  2020 fftw_f77.i
-rw-r--r-- 1 root root  6462 Sep 17  2020 fftw_mpi.h
-rw-r--r-- 1 root root  1201 Sep 17  2020 fftw_threads.h
-rw-r--r-- 1 root root  3280 Sep 17  2020 rfftw.h
-rw-r--r-- 1 root root  2966 Sep 17  2020 rfftw_mpi.h
-rw-r--r-- 1 root root  1585 Sep 17  2020 rfftw_threads.h
sdp@gpunode:~/q-e-qe-7.2-omp5/build$ cmake -DQE_FFTW_VENDOR=Intel_DFTI -DFFTW_INCLUDE_DIRS=/usr/include/mkl/fftw ..
-- Enable sanitizer QE_ENABLE_SANITIZER=none
-- C preprocessor used by qe_preprocess_source in qeHelpers.cmake: /usr/bin/cpp
-- MPI settings used by CTest
     MPIEXEC_EXECUTABLE : /opt/intel/oneapi/mpi/2021.10.0/bin/mpiexec
     MPIEXEC_NUMPROC_FLAG : -n
     MPIEXEC_PREFLAGS :
   Tests run as : /opt/intel/oneapi/mpi/2021.10.0/bin/mpiexec -n <NUM_PROCS>  <EXECUTABLE>
-- Source files are not cloned from a git repository.
-- Trying to find LAPACK from Intel MKL
-- Found LAPACK: /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so;/usr/lib/x86_64-linux-gnu/libmkl_sequential.so;/usr/lib/x86_64-linux-gnu/libmkl_core.so;-lm;-ldl;/usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so;/usr/lib/x86_64-linux-gnu/libmkl_sequential.so;/usr/lib/x86_64-linux-gnu/libmkl_core.so;-lm;-ldl;-lm;-ldl
-- Installing Wannier90 via submodule
-- Previous clone found at /home/sdp/q-e-qe-7.2-omp5/external/wannier90.
-- Installing MBD via submodule
-- Previous clone found at /home/sdp/q-e-qe-7.2-omp5/external/mbd.
-- Installing DeviceXlib via submodule
-- Previous clone found at /home/sdp/q-e-qe-7.2-omp5/external/devxlib.
CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find VendorFFTW (missing: VendorFFTW_INCLUDE_DIRS)
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindVendorFFTW.cmake:139 (find_package_handle_standard_args)
  FFTXlib/CMakeLists.txt:63 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/sdp/q-e-qe-7.2-omp5/build/CMakeFiles/CMakeOutput.log".
See also "/home/sdp/q-e-qe-7.2-omp5/build/CMakeFiles/CMakeError.log".

with or without DFFTW_INCLUDE_DIRS. 

Does the solution you suggest work on your machine? 

Thanks!

[Mark_L_Intel]

yes, it worked on my machine. 

[yehonatan123f]

Can you maybe in Intel try your solution on our machine? It could be fantastic.

Resource ID: 27cbe775-965b-43c9-8e58-b3e7b4506540 on Developer Cloud.

If you need more details I would like to continue via emails or open a private support request. 

Thanks!

[Mark_L_Intel]

Could you try on another machine? It should work. I've not worked on Developer cloud for some time and I may need to reestablish my account.