undefined symbol: __libm_feature_flag

[Thorsten_B_]

Dear colleagues,

I'm compiling WRF model with icc and ifort from parallel_studio_xe_2016.1.056 on a AMD64 with Debian 8 (Jessie). During compilation there is no error. During runtime there appears the same error for almost any executable of the model (example is for real.exe) that is:

./real.exe: symbol lookup error: ./real.exe: undefined symbol: __libm_feature_flag

It is independent of compiling for serial or parallel execution.

Because of this problem my LD_LIBRARY_PATH is probably a bit exaggerated:  

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib:/opt/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/opt/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin:/opt/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64:/opt/compilers_and_libraries_2016.1.150/linux/ipp/lib/mic:/opt/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:/opt/compilers_and_libraries_2016.1.150/linux/mkl/lib/mic:/opt/compilers_and_libraries_2016.1.150/linux/mpi/mic/lib:/opt/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib:/opt/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64:/opt/compilers_and_libraries_2016.1.150/linux/tbb/lib/mic:/opt/compilers_and_libraries_2016.1.150/linux/mkl/benchmarks/hpcg/bin/lib/intel64:/opt/compilers_and_libraries_2016.1.150/linux/mkl/benchmarks/hpcg/bin/lib/mic:/opt/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin:/opt/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin_mic/:/opt/advisor_xe_2016.1.10.435553/lib64/pinruntime/:/opt/inspector_xe_2016.1.1.435552/lib64/pinruntime/:/opt/vtune_amplifier_xe_2016.1.1.434111/lib64/pinruntime/

export CPATH=${CPATH}:/opt/compilers_and_libraries_2016.1.150/linux/daal/include/:/opt/compilers_and_libraries_2016.1.150/linux/daal/include/algorithms:/opt/compilers_and_libraries_2016.1.150/linux/daal/include/services:/opt/compilers_and_libraries_2016.1.150/linux/compiler/include:/opt/compilers_and_libraries_2016.1.150/linux/ipp/include:/opt/compilers_and_libraries_2016.1.150/linux/ipp/tools/intel64/staticlib//opt/compilers_and_libraries_2016.1.150/linux/mkl/benchmarks/mp_linpack/include:/opt/compilers_and_libraries_2016.1.150/linux/mkl/include:/opt/compilers_and_libraries_2016.1.150/linux/mkl/include/fftw/:/opt/compilers_and_libraries_2016.1.150/linux/mkl/interfaces/:/opt/compilers_and_libraries_2016.1.150/linux/mpi/mic/include:/opt/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include:/opt/compilers_and_libraries_2016.1.150/linux/tbb/include:/opt/compilers_and_libraries_2016.1.150/linux/tbb/include/tbb:/opt/compilers_and_libraries_2016.1.150/linux/tbb/include/tbb/internal/:/opt/compilers_and_libraries_2016.1.150/linux/tbb/include/tbb/machine/:/opt/compilers_and_libraries_2016.1.150/linux/tbb/include/serial/tbb/

I copied libm.so.6 and  libm-2.3.4.so from the glibc directory to the next higher one, because libc6 en glibc caused problems with some GNU functions. I still don't know in which library contains the libm_feature_flag. Suppose it should be libm.so.6 or libimf.so. I see this error for he first time. It didn't appear in my former compilations of WRF for example with the ifort and icc from composer_xe_2013.

Any suggestion to fix this problem will be welcome.

Many thanks

Thorsten

[TimP]

As libimf which comes with ifort requires this entry point in the linux libraries, you must assure that you have a working g++ (matching your choice of Intel64 or 32-bit mode) on path, including the libm on the path which can be found by g++ -print-search-dirs.

If you really need to know which libraries are linked, you can add -# option in your link step and compare a good and a bad installation.

These seem typically to include -lm -lc -lgcc -lgcc_s -ldl -lc  (libc linked twice).

I don't know whether you may run into a problem when you append all those libraries to LD_LIBRARY_PATH rather than putting the needed additional ones ahead of the default ones.  Perhaps you have such a long path that it doesn't find all the necessary ones.

[Thorsten_B_]

Dear Tim,

you are quite right: replacing the long LD_LIBRARY_PATH in profile by 'source /opt/bin/compilervars.sh' gave the solution. Although running /opt/bin/compilervars.sh by hand gives errors like: /opt/compilers_and_libraries_2016.1.150/linux/daal/bin/daalvars.sh: Bad substitution

Anyway WRF is running! Many thanks

Thorsten

[Jerome_B_4]

Hi,

I reopen this issue because I am facing the same behavior with another program. No errors at compilation but when execution I get :

exec/em64t_M/charmm: symbol lookup error: exec/em64t_M/charmm: undefined symbol: __libm_feature_flag

I have ran source /opt/intel/.../compilervars.sh intel64 successfully prior compiling the charmm programm.

What is wrong on my system ?

[Kevin_D_Intel]

Perhaps libm is not installed on your system or is but just not on any path in LD_LIBRARY_PATH. Check that the complete GNU gcc/g++/glibc development packages were installed.

On my particular RedHat system, glibc-2.17-55.el7.i686, contributes libm.so.6:

$ rpm -qf /lib/libm.so.6
glibc-2.17-55.el7.i686

[TimP]

From the looks of Jerome's post, he would need the 64-bit libm, which would be part of the basic g++ development system option for 64-bit linux, and would require using the intel64 compiler.

[Adelman__Zac]

Hi All

I'm running into this same issue.  I'm compiling with parallel_studio_xe_2019_professional on Ubuntu Server 18.04 LTS (it's an AWS EC2 instance). I got the software to compile and run with gfortran-7 just fine.  It compiles with Intel, but when I try to run the exec, I get this same error:

./wrf.exe: symbol lookup error: ./wrf.exe: undefined symbol: __libm_feature_flag

I've tried setting my LD_LIBRARY path several different ways, augmenting with directory paths with every instance of the libm* on my system, moving the order of the directories around, no luck. I'm getting this same error with every version of the software. I've tried WRF 3.8, 3.9.1, and 4.0.  They all compiled, they just won't run. 

I started with a bare metal instance and built it up to get WRF to compile. During the setup, for the Intel compiler, I didn't do the full install.  I tried a custom install where I just included the compilers, math libs, and GDB.  I installed gcc-7, gfortran-7, and g++-7 with apt-get. I'm wondering if maybe I left out something that is breaking this install? 

Here are some details of my system. Please let me know if you see something that I'm missing. 

ubuntu>env
USER=ubuntu
LOGNAME=ubuntu
HOME=/home/ubuntu
PATH=/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64:/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/bin:/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/bin:/opt/intel/debugger_2019/gdb/intel64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
MAIL=/var/mail/ubuntu
SHELL=/usr/bin/tcsh
SSH_TTY=/dev/pts/0
TERM=xterm
XDG_SESSION_ID=107
XDG_RUNTIME_DIR=/run/user/1000
LANG=C.UTF-8
HOSTTYPE=x86_64-linux
VENDOR=unknown
OSTYPE=linux
MACHTYPE=x86_64
SHLVL=1
PWD=/apps/WRFV3.9.1/WRFV3_intel_mpich2/main
GROUP=ubuntu
WRFIO_NCD_LARGE_FILE_SUPPORT=1
INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2019.0.117/linux/licenses:/opt/intel/licenses:/home/ubuntu/intel/licenses:/opt/licenses/NCOM_L__CPPFOR_NHKS-B7WTZDZF.lic
DAALROOT=/opt/intel/compilers_and_libraries_2019.0.117/linux/daal
CPATH=/opt/intel/compilers_and_libraries_2019.0.117/linux/ipp/include:/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/include:/opt/intel/compilers_and_libraries_2019.0.117/linux/pstl/include:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/include:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/include:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/include
LIBRARY_PATH=/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/../tbb/lib/intel64_lin/gcc4.4
LD_LIBRARY_PATH=/usr/local/lib:/usr/lib/x86_64-linux-gnu:/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/lib:/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib/release:/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/opt/intel/debugger_2019/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/apps/MPICH2/Linux2_x86_64ifort/lib
CLASSPATH=/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/lib/daal.jar
NLSPATH=/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin/locale/%l_%t/%N:/opt/intel/debugger_2019/gdb/intel64/share/locale/%l_%t/%N
INFOPATH=/opt/intel/documentation_2019/en/debugger/gdb-ia/info/
MANPATH=/opt/intel/documentation_2019/en/man/common:/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/man:/opt/intel/documentation_2019/en/debugger/gdb-ia/man/:/usr/local/man:/usr/local/share/man:/usr/share/man:
INTEL_PYTHONHOME=/opt/intel/debugger_2019/python/intel64/
TBBROOT=/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb
PSTLROOT=/opt/intel/compilers_and_libraries_2019.0.117/linux/pstl
MKLROOT=/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl
PKG_CONFIG_PATH=/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/bin/pkgconfig
IPPROOT=/opt/intel/compilers_and_libraries_2019.0.117/linux/ipp
arch=intel64
I_MPI_ROOT=/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi
FI_PROVIDER_PATH=/opt/intel//compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/lib/prov

ubuntu>gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.3.0-27ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04) 

ubuntu>sudo updatedb
ubuntu>locate libm.
/lib/x86_64-linux-gnu/libm.so.6
/lib32/libm.so.6
/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/ia32_lin/locale/en_US/libm.cat
/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin/locale/en_US/libm.cat
/snap/core/5328/lib/i386-linux-gnu/libm.so.6
/snap/core/5328/lib/x86_64-linux-gnu/libm.so.6
/snap/core/5548/lib/i386-linux-gnu/libm.so.6
/snap/core/5548/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libm.a
/usr/lib/x86_64-linux-gnu/libm.so

[TimP]

In your efforts with LD_LIBRARY_PATH, you might look at which entries aren't wanted. It seems excessively long; comments I made near the top of the thread may apply. With a plain WRF, you shouldn't require daal.  The order you have seems not normal , but it may not matter, provided that your g++ is working with that path.

It's not good to have both mpich and Intel mpi libraries.  You must keep your various mpi installations separate; for example, when your mpich was built, it needed to be in its own directory tree, as Intel mpi is, so you don't have even the mpich Fortran libraries visible when you run with Intel MPI (as well as no Intel mpi visible when building or running mpich).  Intel MPI and mpich are so similar that there would be many duplicate library entry points.