Intel MKL fails to load libfftw3

[AbdulMuhaymin]

To run a program in a cluster, I am loading intel and impi modules. But it gives me an error of:
"error while loading shared libraries: libfftw3.so.3: cannot open shared object file: No such file or directory". 

When I did `ldd my_program`, it gave me the following output: 

...
libfftw3.so.3 => not found

....

For all other shared libraries, it correctly configures except libfftw3 even though this file is in the same directory as others. What could be the reason? 

[ShanmukhS_Intel]

Hi Abdul,

Could you please confirm if the libfftw3 libraries were installed in the corresponding path? If yes, could you please confirm the name of the .so which was available? If it was not available, you could soft link it with the appropriate name, and try running the source code once again and let us know if the issue persists.

Best Regards,

Shanmukh.SS

[AbdulMuhaymin]

Hello,

Yes, the libfftw3 libraries are installed in the path. When I load the intel and impi module in the head node and then run `ldd my_program.x`, then it says that no problem. All the paths are found correctly. The problem appears when I run the program in the cluster using sbatch. For example, this is the output I get in the head node: 

linux-vdso.so.1 => (0x00007fffd0bc9000)
libmkl_scalapack_ilp64.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_scalapack_ilp64.so (0x00007efc77014000)
libmkl_intel_ilp64.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_intel_ilp64.so (0x00007efc76552000)
libmkl_intel_thread.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so (0x00007efc74076000)
libmkl_core.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_core.so (0x00007efc6fda1000)
libmkl_blacs_intelmpi_ilp64.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_blacs_intelmpi_ilp64.so (0x00007efc6fb32000)
libiomp5.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libiomp5.so (0x00007efc6f748000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efc6f52c000)
libm.so.6 => /lib64/libm.so.6 (0x00007efc6f22a000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007efc6f026000)
libfftw3.so.3 => /lib64/libfftw3.so.3 (0x00007efc6eca1000)
libmpifort.so.12 => /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/libmpifort.so.12 (0x00007efc6e8e2000)
libmpi.so.12 => /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/release/libmpi.so.12 (0x00007efc6d9f0000)
librt.so.1 => /lib64/librt.so.1 (0x00007efc6d7e8000)
libc.so.6 => /lib64/libc.so.6 (0x00007efc6d41a000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007efc6d204000)
/lib64/ld-linux-x86-64.so.2 (0x00007efc77924000)
libfabric.so.1 => /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/lib/libfabric.so.1 (0x00007efc6cfcc000)

But this is the output I get in the computing node when I submit it as a job. The output of the `ldd my_program.x` is:

linux-vdso.so.1 => (0x00007fff59ecc000)
libmkl_scalapack_ilp64.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_scalapack_ilp64.so (0x00007fc4fa739000)
libmkl_intel_ilp64.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_intel_ilp64.so (0x00007fc4f9c77000)
libmkl_intel_thread.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so (0x00007fc4f779b000)
libmkl_core.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_core.so (0x00007fc4f34c6000) 
libmkl_blacs_intelmpi_ilp64.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/libmkl_blacs_intelmpi_ilp64.so (0x00007fc4f3257000) 
libiomp5.so => /opt/intel/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libiomp5.so (0x00007fc4f2e6d000) 
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc4f2c51000)
libm.so.6 => /lib64/libm.so.6 (0x00007fc4f294f000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc4f274b000)
libfftw3.so.3 => not found
libmpifort.so.12 => /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/libmpifort.so.12 (0x00007fc4f238c000) 
libmpi.so.12 => /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/release/libmpi.so.12 (0x00007fc4f149a000) 
librt.so.1 => /lib64/librt.so.1 (0x00007fc4f1292000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc4f0ec5000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc4f0caf000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc4fb049000)
libfabric.so.1 => /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/lib/libfabric.so.1 (0x00007fc4f0a77000)

See line 11 for the difference. The soft links are also there in /lib64/ directory. I don't understand why it is failing for libfftw3 only. In the same directory, there are other libraries too and the program can correctly find those libraries. 

[ShanmukhS_Intel]

Hi Abdul,

We see that you are using an older toolkit for your program execution as per the log shared by you. (Correct us if we are wrong)

Please refer to the below link for supported versions of oneAPI toolkits. 

https://www.intel.com/content/www/us/en/developer/articles/release-notes/intel-parallel-studio-xe-supported-and-unsupported-product-versions.html

Could you please try executing the code with the latest version and let us know if the issue persists? 

In addition, Please get back to us with the below-mentioned environment details.

Slurm file, OS details, Hardware details, Sample reproducer, Libfabric being used, commands for program execution and oneAPI Toolkit version being used.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi Abdul,

A gentle reminder:

Has the information provided helped? If not, Could you please let us know if the issue persists with newer versions of oneAPI?

Best Regards,

Shanmukh.SS

[AbdulMuhaymin]

Hello.

The information you provided didn't help since we have no sudo control over the cluster. That means we cannot execute the code with the upgraded version. Alternatively, we are trying to load non-shared versions of libraries (ending with .a) instead of shared ones (ending with .so) but so far with no success. 

This is the slurm file we use:

#!/bin/bash
#SBATCH -p normal
#SBATCH --account=MyName
#SBATCH --job-name=test
#SBATCH --nodes 1
#SBATCH --ntasks-per-node 40
#SBATCH --time=05:00:00
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err

module unload gnu8
module unload openmpi
module load intel
module load impi
module load quantum-espresso

mpirun -np 40 pw.x -inp h2_relax.in > h2_relax.out

OS details:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

We use `mpirun` to execute the program inside the slurm file and we use `sbatch` command to submit the job to the cluster. 

[ShanmukhS_Intel]

Hi Abdul,

It seems dependency files were missing and as a result, we are facing issues related to the execution of your code. It would be a great help if you share the files ( h2_relax.in) and share the steps to reproduce(if any).

Best Regards,

Shanmukh.SS

[AbdulMuhaymin]

The h2_relax.in file is given below:

&CONTROL
    calculation   = "relax",
    prefix        = "h2",
    pseudo_dir    = "./"
/
&SYSTEM
    a           = 15.00,
    ecutrho     = 46.00,
    ecutwfc     = 221.00,
    ibrav       = 1,
    nat         = 2,
    ntyp        = 1,
    occupations = "smearing",
    smearing    = "gaussian",
    degauss     =  0.01
/
&ELECTRONS
/
&IONS
/
K_POINTS {automatic}
 1 1 1 0 0 0
ATOMIC_SPECIES
 H       1.00784  H.pbe-rrkjus_psl.1.0.0.UPF
ATOMIC_POSITIONS {angstrom}
 H       0.00   0.00   0.00
 H       1.00   0.00   0.00

The pw.x is the program that we run and h2_relax.in is the input. To obtain the pw.x binary, it is compiled by using official quantum espresso software: https://github.com/QEF/q-e/archive/refs/tags/qe-6.4.1.tar.gz. After downloading the source code, first `./configure` and then `make pw` will generate a pw.x binary inside the bin folder. 

Also, we need a H.pbe-rrkjus_psl.1.0.0.UPF file which should be in the same folder. It is uploaded as a zip file since we cannot upload a .UPF file in this forum. Please unzip it before using it. The zip file contains both the h2_relax.in and the upf file. If you put the upf file in some other directory, then change the pseudo_dir variable in line 4 of h2_relax.in file accordingly.

[ShanmukhS_Intel]

Hi Abdul,

A gentle reminder:

As informed earlier, please refer to the below link for the list of supported product versions. 

https://www.intel.com/content/www/us/en/developer/articles/release-notes/intel-parallel-studio-xe-supported-and-unsupported-product-versions.html

Thanks for sharing the files. However, we could not help you with an unsupported version. Maybe, your admin team could help you in getting sudo access and installing the latest version of oneAPI so that you could use the latest version of Intel compilers based on which we could help you. 

Please find the below link for step by step guide to installing the oneAPI toolkit for various environments and let us know if the issue persists.

https://www.intel.com/content/www/us/en/developer/articles/guide/installation-guide-for-oneapi-toolkits.html

We would like to request you download the latest product to get further assistance

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi Abdul,

We have not heard back from you. If you need further assistance, please post a new question as this thread would no longer be monitored by Intel.

Best Regards,

Shanmukh.SS