dpcpp and GNU Autotools

[lawmurray]

I'm having a little trouble getting dpcpp and the GNU Autotools to play nicely, so wanted to report here and see if others have experienced this (I could not find similar reports). Currently I don't have a workaround, although a wrapper script around dpcpp may be one approach to resolve this issue.

I've attached a minimal GNU Autotools setup that reproduces the issue on my openSUSE Leap 15.3 system (side note: Leap 15.3 provides gcc 7, dpcpp seems to want at least version 8, so I've installed gcc 8 as well, which dpcpp seems to find).

Running the following:

./bootstrap
./configure CXX=dpcpp
make

produces an error such as:

dpcpp: error: no such file or directory: '/tmp/conftest-0f58ae.o'
dpcpp: error: no such file or directory: '/tmp/a-b8ad38.o'

The link line tries to link in these /tmp/*.o object files, that do not exist. If I copy the link line, remove them, and run, it works. Static library builds seem unaffected by this, whereas shared library builds are affected.

I've done some sleuthing as to how these /tmp/*.o files end up there. After running ./configure, inspecting config.status reveals that these two object files in the /tmp directory have been added to config.status for linking:

grep conftest config.status

predep_objects_CXX='/usr/lib64/gcc/x86_64-suse-linux/8/../../../../lib64/crti.o /usr/lib64/gcc/x86_64-suse-linux/8/crtbeginS.o /tmp/conftest-0f58ae.o /tmp/a-b8ad38.o /usr/lib64/gcc/x86_64-suse-linux/8/crtendS.o /usr/lib64/gcc/x86_64-suse-linux/8/../../../../lib64/crtn.o'

Notice the '/tmp/conftest-0f58ae.o /tmp/a-b8ad38.o'. The rest are standard C library things.

Looking into the 'configure' script as to where these come from, find the variable 'output_verbose_link_cmd'. Above its second use is an explanation of what's going on:

# Commands to make compiler produce verbose output that lists
# what "hidden" libraries, object files and flags are used when
# linking a shared library.

To reproduce what the script then does (no need for the attached package, can just run this), try:

touch conftest.cpp
dpcpp -o conftest.o -c conftest.cpp
dpcpp -v conftest.o 2>&1 | grep -v "^Configured with:" | grep "\-L"

On my system I get:

"/usr/lib64/gcc/x86_64-suse-linux/8/../../../../x86_64-suse-linux/bin/ld" --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib64/gcc/x86_64-suse-linux/8/../../../../lib64/crt1.o /usr/lib64/gcc/x86_64-suse-linux/8/../../../../lib64/crti.o /usr/lib64/gcc/x86_64-suse-linux/8/crtbegin.o -L/opt/intel/oneapi/compiler/2021.3.0/linux/bin/../compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2021.3.0/linux/bin/../lib -L/opt/intel/oneapi/compiler/2021.3.0/linux/bin/../compiler/lib/intel64_lin -L/usr/lib64/gcc/x86_64-suse-linux/8 -L/usr/lib64/gcc/x86_64-suse-linux/8/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib64/gcc/x86_64-suse-linux/8/../../../../x86_64-suse-linux/lib -L/usr/lib64/gcc/x86_64-suse-linux/8/../../.. -L/opt/intel/oneapi/compiler/2021.3.0/linux/bin/../lib -L/lib-L/usr/lib -L/opt/intel/oneapi/vpl/2021.4.0/lib -L/opt/intel/oneapi/tbb/2021.3.0/env/../lib/intel64/gcc4.8 -L/opt/intel/oneapi/mpi/2021.3.0//libfabric/lib -L/opt/intel/oneapi/mpi/2021.3.0//lib/release -L/opt/intel/oneapi/mpi/2021.3.0//lib -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/ipp/2021.3.0/lib/intel64 -L/opt/intel/oneapi/ippcp/2021.3.0/lib/intel64 -L/opt/intel/oneapi/ipp/2021.3.0/lib/intel64 -L/opt/intel/oneapi/dnnl/2021.3.0/cpu_dpcpp_gpu_dpcpp/lib -L/opt/intel/oneapi/dal/2021.3.0/lib/intel64 -L/opt/intel/oneapi/compiler/2021.3.0/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2021.3.0/linux/lib -L/opt/intel/oneapi/ccl/2021.3.0/lib/cpu_gpu_dpcpp -L. /tmp/conftest-9bfe87.o /tmp/a-ce4b84.o -Bstatic -lsvml -Bdynamic -Bstatic -lirng -Bdynamic -lstdc++ -limf -lm -lgcc_s -lgcc -Bstatic -lirc -Bdynamic -ldl -lgcc_s -lgcc -lsycl -lc -lgcc_s -lgcc -Bstatic -lirc_s -Bdynamic /usr/lib64/gcc/x86_64-suse-linux/8/crtend.o /usr/lib64/gcc/x86_64-suse-linux/8/../../../../lib64/crtn.o

The script eventually parses all the *.o files from this to add to the link line. This include the /tmp/*.o object files. I suppose these are only intended for this particular invocation of dpcpp and are then removed, causing the final link error during make above.

I understand that this is an interaction effect between dpcpp and the GNU Autotools, but a patch to the latter will take rather a few years to propagate, so I wonder if there is a workaround on the dpcpp side.

As a workaround for my own purposes, I might try writing a dpcpp_wrapper script around dpcpp that captures and removes these /tmp/*.o files for the purposes of configure, then passing CXX=dpcpp_wrapper to configure instead.

[NoorjahanSk_Intel]

Hi,

 Thanks for reaching out to us.

We tried it from our end and  able to reproduce the same issue.

we are working on your issue internally. we will get back to you soon.

Meanwhile, could you please try using icx and icc compilers.

Thanks & Regards

Noorjahan.

[lawmurray]

Thanks for the help Noorjahan.

It seems to me that icpx has the same issue here. If, using the minimal package I attached above, I use:

./configure CXX=icpx

then it appears to work, but this is because SYCL is not used in this case. If instead I do:

./configure CXX="icpx" CXXFLAGS="-fsycl"

then I'm back to the same error as for dpcpp.

In the meantime, as a workaround, I've hacked up a wrapper script that just strips the problematic args:

eval dpcpp `echo $@ | sed -E 's:/tmp/[^[:space:]]+\.o::g'`

 If I name that dpcpp_wrapper and use:

./configure CXX=./dpcpp_wrapper

then it at least links without error.

Eventually I get a runtime error on the first kernel launch though:

INTEL MKL ERROR: /opt/intel/oneapi/mkl/2021.3.0/lib/intel64/libmkl_avx2.so.1: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so.1 or libmkl_def.so.1.

But I see mkl_sparse_optimize_bsr_trsm_i8 defined in /opt/intel/oneapi/mkl/2021.3.0/lib/intel64/libmkl_sequential.so.1, which is definitely linked in. I'll dig some more. This new error may not be related. I'll open a new issue if it persists, and I see others have reported similar errors (although the suggested fixes there haven't worked for me).

(Edit: The secondary INTEL_MKL_ERROR was fixed with LD_PRELOAD="/opt/intel/oneapi/mkl/2021.3.0/lib/intel64/libmkl_tbb_thread.so /opt/intel/oneapi/mkl/2021.3.0/lib/intel64/libmkl_core.so"... seems I have a linking issue to resolve.)

[NoorjahanSk_Intel]

Hi,

We are working on your issue internally. We will get back to you soon.

Thanks & Regards

Noorjahan.

[Sravani_K_Intel · Accepted solution]

The temporary files that are being included here for the link command line are as follows:

• conftest*.o // host object part of conftest.o fat object
• a*.o // generated device object (wrapped SPIR-V)

The original object is not used during the link due to the fact that the original object is a fat object which contains host and device binary parts. Typically, objects and libraries as passed in from the command line are sent directly to the link step so in general the expectation is for the link command line to be static in nature when constructed.

There are no existing mechanisms to retain these binaries beyond using a more general option like -save-temps which will save all of the temporaries.

[lawmurray]

Thanks for the explanation of what's going on here. It seems then that the temporary objects are only associated with that particular test run by the configure script, so stripping them out with a wrapper script as I suggested is a sensible solution. That's still working for me.

[cmacmackin]

Is there any work being done to resolve this issue in some way? Lots of packages still use autotools and hacks need to be used to work around the issue.