topic Re:Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL in Intel® oneAPI Math Kernel Library

Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

yehonatan123f — Mon, 15 Jan 2024 09:17:19 GMT

Hello,

I am trying to build q-e-7.2-omp5 (https://gitlab.com/QEF/q-e/-/tree/qe-7.2-omp5?ref_type=tags) specifically for Intel GPUs using cmake.

I am running a SPR+4 PVC node on Developer Cloud.

This is my cmake command (that I run in build dir):

cmake -DCMAKE_C_COMPILER=mpiicx -DCMAKE_Fortran_COMPILER=mpiifx -DCMAKE_C_FLAGS:STRING="-xHOST -qmkl=parallel -fiopenmp -fopenmp-targets=spir64 -I$MKLROOT/include/fftw" -DCMAKE_Fortran_FLAGS:STRING="-xHOST -qmkl=parallel -fiopenmp -fopenmp-targets=spir64 -I$MKLROOT/include/fftw" -DQE_ENABLE_OPENMP=ON -DQE_ENABLE_OFFLOAD=ON -DQE_FFTW_VENDOR=Intel_DFTI ..

This is the output:

-- The Fortran compiler identification is IntelLLVM 2023.2.0
-- The C compiler identification is IntelLLVM 2023.2.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/intel/oneapi/mpi/2021.10.0/bin/mpiifx - skipped
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/intel/oneapi/mpi/2021.10.0/bin/mpiicx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Setting build type to 'Release' as none was specified
-- Looking for mallinfo
-- Looking for mallinfo - found
-- Enable sanitizer QE_ENABLE_SANITIZER=none
-- C preprocessor used by qe_preprocess_source in qeHelpers.cmake: /usr/bin/cpp
-- Performing Test Fortran_ISYSTEM_SUPPORTED
-- Performing Test Fortran_ISYSTEM_SUPPORTED - Success
-- Found OpenMP_Fortran: -fiopenmp (found version "5.0")
-- Found OpenMP_C: -fiopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0") found components: Fortran C
-- Found MPI_Fortran: /opt/intel/oneapi/mpi/2021.10.0/bin/mpiifx (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: Fortran
-- MPI settings used by CTest
-- Found Git: /usr/bin/git (found suitable version "2.34.1", minimum required is "2.13")
-- Source files are cloned from a git repository.
-- Trying to find LAPACK from Intel MKL
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- Found LAPACK: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;-lm;-ldl
-- Found LAPACK: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;-lm;-ldl
-- Looking for Fortran zhpev
-- Looking for Fortran zhpev - found
-- Installing Wannier90 via submodule
-- Installing MBD via submodule
-- Installing DeviceXlib via submodule
-- Found VendorFFTW: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl;-lm;-ldl
-- Found Intel_DFTI library
-- Enabling tests in test-suite
-- generating tests in pw category
-- generating tests in cp category
-- generating tests in ph category
-- generating tests in epw category
-- generating tests in tddfpt category
-- generating tests in hp category
-- Configuring done
-- Generating done
-- Build files have been written to: /home/sdp/q-e/build

The error comes when I run "make":

[ 5%] Building Fortran object FFTXlib/src/CMakeFiles/qe_fftx.dir/fft_scalar.f90.o
fft_scalar.f90(46): #error: No fft_scalar backend selected for __OPENMP_GPU!

This error is issued from fft_scalar.f90(46) where __ONEMKL is not defined, as you can see (fft_scalar.f90):

#if defined(__CUDA)
USE fft_scalar_cuFFT
#elif defined(__OPENMP_GPU) && defined(__ONEMKL)
USE fft_scalar_dfti_omp
#elif defined(__OPENMP_GPU) && defined(__HIP)
USE fft_scalar_hipfft
#elif defined(__OPENMP_GPU)
#error No fft_scalar backend selected for __OPENMP_GPU!
#endif

I have tried multiple cmake configurations, nothing seem to solve this issue. Nothing helps to make __ONEMKL be recognized.

Just to note, /opt/intel/oneapi/mkl/2023.2.0/include and /opt/intel/oneapi/mkl/2023.2.0/lib/intel64 seem to contain all the needed math files (especially fftw files and mkl_dfti_omp_offload.f90).

What can be the problem with how I am building the project?

Thank you.

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

ReemHarel — Mon, 15 Jan 2024 09:33:45 GMT

I've been working on Quantum espresso for Ponte-vecchio as well, and i also cannot compile it,

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

yehonatan123f — Thu, 01 Feb 2024 09:34:33 GMT

Hey all,

Any idea how to solve this issue?

I am still struggeling in this regard.

Thank you!

Re:Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

Mark_L_Intel — Tue, 06 Feb 2024 05:20:37 GMT

I confirmed the issue on internal machine. Let me look into what can cause it.

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

Mark_L_Intel — Wed, 07 Feb 2024 22:36:55 GMT

@yehonatan123f ,

please try the following from the build directory, e.g., <your-directory>/q-e/build

cmake -DQE_FFTW_VENDOR=Intel_DFTI ..

you should see something like this:

... -- Found VendorFFTW: /opt/intel/oneapi/mkl/2024.0/lib/libmkl_gf_lp64.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_sequential.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_core.so;-lpthread;-lm;-ldl;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_gf_lp64.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_sequential.so;/opt/intel/oneapi/mkl/2024.0/lib/libmkl_core.so;-lpthread;-lm;-ldl;-lpthread;-lm;-ldl -- Found Intel_DFTI library -- Enabling tests in test-suite

and notice "Found Intel_DFTI library" message above. Then "make" should work.

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

Mark_L_Intel — Tue, 13 Feb 2024 20:17:17 GMT

Hello @yehonatan123f , Have you tried above suggestion?

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

yehonatan123f — Thu, 15 Feb 2024 17:42:20 GMT

Hey mark,

It still not working.

sdp@gpunode:~/q-e-qe-7.2-omp5/build$ ll /usr/include/mkl/fftw total 216 drwxr-xr-x 2 root root 4096 Feb 15 06:58 ./ drwxr-xr-x 5 root root 4096 Feb 15 06:58 ../ -rw-r--r-- 1 root root 11900 Sep 17 2020 fftw.h -rw-r--r-- 1 root root 37021 Sep 17 2020 fftw3-mpi.f03 -rw-r--r-- 1 root root 9621 Sep 17 2020 fftw3-mpi.h -rw-r--r-- 1 root root 1902 Sep 17 2020 fftw3-mpi_mkl.h -rw-r--r-- 1 root root 2777 Sep 17 2020 fftw3.f -rw-r--r-- 1 root root 54077 Sep 17 2020 fftw3.f03 -rw-r--r-- 1 root root 17945 Sep 17 2020 fftw3.h -rw-r--r-- 1 root root 3484 Sep 17 2020 fftw3_mkl.f -rw-r--r-- 1 root root 2023 Sep 17 2020 fftw3_mkl.h -rw-r--r-- 1 root root 21606 Sep 17 2020 fftw3_mkl_f77.h -rw-r--r-- 1 root root 1436 Sep 17 2020 fftw_f77.i -rw-r--r-- 1 root root 6462 Sep 17 2020 fftw_mpi.h -rw-r--r-- 1 root root 1201 Sep 17 2020 fftw_threads.h -rw-r--r-- 1 root root 3280 Sep 17 2020 rfftw.h -rw-r--r-- 1 root root 2966 Sep 17 2020 rfftw_mpi.h -rw-r--r-- 1 root root 1585 Sep 17 2020 rfftw_threads.h sdp@gpunode:~/q-e-qe-7.2-omp5/build$ cmake -DQE_FFTW_VENDOR=Intel_DFTI -DFFTW_INCLUDE_DIRS=/usr/include/mkl/fftw .. -- Enable sanitizer QE_ENABLE_SANITIZER=none -- C preprocessor used by qe_preprocess_source in qeHelpers.cmake: /usr/bin/cpp -- MPI settings used by CTest MPIEXEC_EXECUTABLE : /opt/intel/oneapi/mpi/2021.10.0/bin/mpiexec MPIEXEC_NUMPROC_FLAG : -n MPIEXEC_PREFLAGS : Tests run as : /opt/intel/oneapi/mpi/2021.10.0/bin/mpiexec -n <NUM_PROCS> <EXECUTABLE> -- Source files are not cloned from a git repository. -- Trying to find LAPACK from Intel MKL -- Found LAPACK: /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so;/usr/lib/x86_64-linux-gnu/libmkl_sequential.so;/usr/lib/x86_64-linux-gnu/libmkl_core.so;-lm;-ldl;/usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so;/usr/lib/x86_64-linux-gnu/libmkl_sequential.so;/usr/lib/x86_64-linux-gnu/libmkl_core.so;-lm;-ldl;-lm;-ldl -- Installing Wannier90 via submodule -- Previous clone found at /home/sdp/q-e-qe-7.2-omp5/external/wannier90. -- Installing MBD via submodule -- Previous clone found at /home/sdp/q-e-qe-7.2-omp5/external/mbd. -- Installing DeviceXlib via submodule -- Previous clone found at /home/sdp/q-e-qe-7.2-omp5/external/devxlib. CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find VendorFFTW (missing: VendorFFTW_INCLUDE_DIRS) Call Stack (most recent call first): /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/FindVendorFFTW.cmake:139 (find_package_handle_standard_args) FFTXlib/CMakeLists.txt:63 (find_package) -- Configuring incomplete, errors occurred! See also "/home/sdp/q-e-qe-7.2-omp5/build/CMakeFiles/CMakeOutput.log". See also "/home/sdp/q-e-qe-7.2-omp5/build/CMakeFiles/CMakeError.log".

with or without DFFTW_INCLUDE_DIRS.

Does the solution you suggest work on your machine?

Thanks!

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

Mark_L_Intel — Thu, 15 Feb 2024 18:40:34 GMT

yes, it worked on my machine.

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

yehonatan123f — Sun, 18 Feb 2024 08:18:49 GMT

Can you maybe in Intel try your solution on our machine? It could be fantastic.

Resource ID: 27cbe775-965b-43c9-8e58-b3e7b4506540 on Developer Cloud.

If you need more details I would like to continue via emails or open a private support request.

Thanks!

Re: Compiling Quantum Espresso 7.2 with OpenMP5 offloading using Intel MKL

Mark_L_Intel — Fri, 08 Mar 2024 00:27:54 GMT

Could you try on another machine? It should work. I've not worked on Developer cloud for some time and I may need to reestablish my account.