Link-time optimization issue when using ICX

fatvlad1744 · ‎06-17-2021

Hi,

I'm trying to have two-stage compilation where I create static library first and link it afterwards.

// main.cpp
void foo();

int main() {
    foo();
}

// foo.cpp
#include <iostream>

void foo() {
    std::cout << "Hi foo!" << std::endl;
}

I'm following the guide from Intel documentation: https://software.intel.com/content/www/us/en/develop/documentation/oneapi-dpcpp-cpp-compiler-dev-guide-and-reference/top/optimization-and-programming-guide/interprocedural-optimization-ipo/creating-a-library-from-ipo-objects.html

$ icpx -fast -c -o foo.o foo.cpp
$ xiar rcs libfoo.a foo.o
xiar: executing 'ar'
$ icpx -fast main.cpp -L. -lfoo
/usr/bin/ld: skipping incompatible ./libfoo.a when searching for -lfoo
/usr/bin/ld: cannot find -lfoo
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
$ icpx -fast main.cpp -L. -lfoo -v
Intel(R) oneAPI DPC++ Compiler 2021.2.0 (2021.2.0.20210317)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2021.2.0/linux/bin
Found candidate GCC installation: /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0
Selected GCC installation: /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda, version 11.0
 "/opt/intel/oneapi/compiler/2021.2.0/linux/bin/clang++" -cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -flto -flto-unit -disable-free -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model static -fveclib=SVML -mframe-pointer=none -menable-no-infs -menable-no-nans -menable-unsafe-fp-math -fno-signed-zeros -mreassociate -freciprocal-math -fdenormal-fp-math=preserve-sign,preserve-sign -ffp-contract=fast -fno-rounding-math -ffast-math -ffinite-math-only -mconstructor-aliases -munwind-tables -target-cpu haswell -target-feature +sse2 -target-feature -tsxldtrk -target-feature +cx16 -target-feature +sahf -target-feature -tbm -target-feature -avx512ifma -target-feature -sha -target-feature -gfni -target-feature -fma4 -target-feature -vpclmulqdq -target-feature -prfchw -target-feature +bmi2 -target-feature -cldemote -target-feature +fsgsbase -target-feature -ptwrite -target-feature -amx-tile -target-feature -uintr -target-feature +popcnt -target-feature -widekl -target-feature +aes -target-feature -avx512bitalg -target-feature -movdiri -target-feature -xsaves -target-feature -avx512er -target-feature -avxvnni -target-feature -avx512vnni -target-feature -amx-bf16 -target-feature -avx512vpopcntdq -target-feature -pconfig -target-feature -clwb -target-feature -avx512f -target-feature -xsavec -target-feature -clzero -target-feature -pku -target-feature +mmx -target-feature -lwp -target-feature -rdpid -target-feature -xop -target-feature -rdseed -target-feature -waitpkg -target-feature -kl -target-feature -movdir64b -target-feature -sse4a -target-feature -avx512bw -target-feature -clflushopt -target-feature +xsave -target-feature -avx512vbmi2 -target-feature +64bit -target-feature -avx512vl -target-feature -serialize -target-feature -hreset -target-feature +invpcid -target-feature -avx512cd -target-feature +avx -target-feature -vaes -target-feature -avx512bf16 -target-feature +cx8 -target-feature +fma -target-feature -rtm -target-feature +bmi -target-feature -enqcmd -target-feature +rdrnd -target-feature -mwaitx -target-feature +sse4.1 -target-feature +sse4.2 -target-feature +avx2 -target-feature +fxsr -target-feature -wbnoinvd -target-feature +sse -target-feature +lzcnt -target-feature +pclmul -target-feature -prefetchwt1 -target-feature +f16c -target-feature +ssse3 -target-feature -sgx -target-feature -shstk -target-feature +cmov -target-feature -avx512vbmi -target-feature -amx-int8 -target-feature +movbe -target-feature -avx512vp2intersect -target-feature +xsaveopt -target-feature -avx512dq -target-feature -adx -target-feature -avx512pf -target-feature +sse3 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /opt/intel/oneapi/compiler/2021.2.0/linux/lib/clang/12.0.0 -I/opt/intel/oneapi/vpl/2021.2.2/include -I/opt/intel/oneapi/tbb/2021.2.0/env/../include -I/opt/intel/oneapi/mpi/2021.2.0//include -I/opt/intel/oneapi/mkl/latest/include -I/opt/intel/oneapi/ipp/2021.2.0/include -I/opt/intel/oneapi/ippcp/2021.2.0/include -I/opt/intel/oneapi/ipp/2021.2.0/include -I/opt/intel/oneapi/dpl/2021.2.0/linux/include -I/opt/intel/oneapi/dpcpp-ct/2021.2.0/include -I/opt/intel/oneapi/dnnl/2021.2.0/cpu_dpcpp_gpu_dpcpp/lib -I/opt/intel/oneapi/dev-utilities/2021.2.0/include -I/opt/intel/oneapi/dal/2021.2.0/include -I/opt/intel/oneapi/compiler/2021.2.0/linux/include -I/opt/intel/oneapi/ccl/2021.2.0/include/cpu_gpu_dpcpp -cxx-isystem /opt/intel/oneapi/clck/2021.2.0/include -internal-isystem /opt/intel/oneapi/compiler/2021.2.0/linux/bin/../compiler/include -internal-isystem /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0 -internal-isystem /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu -internal-isystem /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward -internal-isystem /usr/local/include -internal-isystem /opt/intel/oneapi/compiler/2021.2.0/linux/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -fdeprecated-macro -fdebug-compilation-dir /home/vfatenko/source/atom_dev -ferror-limit 19 -disable-cpudispatch-ifuncs -fheinous-gnu-extensions -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -mllvm -enable-gvn-hoist -fcolor-diagnostics -vectorize-loops -vectorize-slp -fintel-compatibility -fintel-advanced-optim -mllvm -disable-hir-generate-mkl-call -mllvm -intel-libirc-allowed -mllvm -loopopt -o /tmp/main-1b97ae.o -x c++ main.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/intel/oneapi/ipp/2021.2.0/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/intel/oneapi/vpl/2021.2.2/include
 /opt/intel/oneapi/tbb/2021.2.0/env/../include
 /opt/intel/oneapi/mpi/2021.2.0//include
 /opt/intel/oneapi/mkl/latest/include
 /opt/intel/oneapi/ipp/2021.2.0/include
 /opt/intel/oneapi/ippcp/2021.2.0/include
 /opt/intel/oneapi/dpl/2021.2.0/linux/include
 /opt/intel/oneapi/dpcpp-ct/2021.2.0/include
 /opt/intel/oneapi/dnnl/2021.2.0/cpu_dpcpp_gpu_dpcpp/lib
 /opt/intel/oneapi/dev-utilities/2021.2.0/include
 /opt/intel/oneapi/dal/2021.2.0/include
 /opt/intel/oneapi/compiler/2021.2.0/linux/include
 /opt/intel/oneapi/ccl/2021.2.0/include/cpu_gpu_dpcpp
 /opt/intel/oneapi/clck/2021.2.0/include
 /opt/intel/oneapi/compiler/2021.2.0/linux/bin/../compiler/include
 /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0
 /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu
 /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward
 /usr/local/include
 /opt/intel/oneapi/compiler/2021.2.0/linux/lib/clang/12.0.0/include
 /usr/include
End of search list.
 "/usr/bin/ld" --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /lib/../lib64/crt1.o /lib/../lib64/crti.o /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtbegin.o /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtfastmath.o -L. -L/opt/intel/oneapi/compiler/2021.2.0/linux/bin/../compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2021.2.0/linux/bin/../lib -L/opt/intel/oneapi/compiler/2021.2.0/linux/bin/../compiler/lib/intel64_lin -L/usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0 -L/usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../.. -L/opt/intel/oneapi/compiler/2021.2.0/linux/bin/../lib -L/lib -L/usr/lib -plugin /opt/intel/oneapi/compiler/2021.2.0/linux/bin/../lib/icx-lto.so -plugin-opt=mcpu=haswell -plugin-opt=O3 -plugin-opt=-vector-library=SVML -plugin-opt=fintel-advanced-optim -plugin-opt=-disable-hir-generate-mkl-call -plugin-opt=-enable-multiversioning -plugin-opt=-intel-libirc-allowed -plugin-opt=-loopopt -L/opt/intel/oneapi/vpl/2021.2.2/lib -L/opt/intel/oneapi/tbb/2021.2.0/env/../lib/intel64/gcc4.8 -L/opt/intel/oneapi/mpi/2021.2.0//libfabric/lib -L/opt/intel/oneapi/mpi/2021.2.0//lib/release -L/opt/intel/oneapi/mpi/2021.2.0//lib -L/opt/intel/oneapi/mkl/latest/lib/intel64 -L/opt/intel/oneapi/ipp/2021.2.0/lib/intel64 -L/opt/intel/oneapi/ippcp/2021.2.0/lib/intel64 -L/opt/intel/oneapi/ipp/2021.2.0/lib/intel64 -L/opt/intel/oneapi/dnnl/2021.2.0/cpu_dpcpp_gpu_dpcpp/lib -L/opt/intel/oneapi/dal/2021.2.0/lib/intel64 -L/opt/intel/oneapi/compiler/2021.2.0/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2021.2.0/linux/lib -L/opt/intel/oneapi/clck/2021.2.0/lib/intel64 -L/opt/intel/oneapi/ccl/2021.2.0/lib/cpu_gpu_dpcpp /tmp/main-1b97ae.o -lfoo -Bstatic -lsvml -Bdynamic -Bstatic -lirng -Bdynamic -lstdc++ -limf -lm -lgcc_s -lgcc -Bstatic -lirc -Bdynamic -ldl -lgcc_s -lgcc -lc -lgcc_s -lgcc -Bstatic -lirc_s -Bdynamic /usr/local/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtend.o /lib/../lib64/crtn.o
/usr/bin/ld: skipping incompatible ./libfoo.a when searching for -lfoo
/usr/bin/ld: cannot find -lfoo
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Seems like it fails to parse static library format when IPO is enabled. When IPO is disabled, everything is fine. Also, classic ICC compiler works fine.

However, when using direct linkage with object file, everything seems to work fine:

$ icpx -fast main.cpp foo.o
$ ./a.out 
Hi foo!

Can you please clarify how to fix that? I'm using CentOS 7:

  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-1127.8.2.el7.x86_64
      Architecture: x86-64

LD output:

GNU ld version 2.27-44.base.el7
ld: supported targets: elf64-x86-64 elf32-i386 elf32-iamcu elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex
ld: supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om

P.S. When I try to check libfoo.a symbols with nm, I'm getting "foo.o: File format not recognized" error. NM version output:

GNU nm version 2.27-44.base.el7
nm: supported targets: elf64-x86-64 elf32-i386 elf32-iamcu elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex