topic Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking in Intel® oneAPI DPC++/C++ Compiler

dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Thu, 09 Sep 2021 12:30:10 GMT

I'm getting a linking error with dpcpp 2021.3.0:

InvalidModule: Invalid SPIR-V module: Casts from private/local/global address space are allowed only to generic

when trying to link in an external library with DPCPP. This appears to be an identical issue to https://community.intel.com/t5/Intel-oneAPI-Base-Toolkit/Dpcpp-link-error-after-upgrade-from-2021-1-to-2021-2, but not with upgrading, with the difference in versions, being an old thread and the note about closing it I thought it would be better to start a new thread.

I can reproduce with this minimal example. Cloning the external dependency:

$ git clone https://github.com/fmtlib/fmt.git -b 8.0.1

and with CMakeLists.txt:

cmake_minimum_required(VERSION 3.20 FATAL_ERROR) set(CMAKE_CXX_COMPILER dpcpp) set(CMAKE_CXX_STANDARD 17) project(dpcpp CXX) add_subdirectory(fmt) add_executable(try try.cc) target_link_libraries(try fmt::fmt)

And try.cc:

#include <fmt/core.h> int main(int argc, char** argv) { fmt::print("{:35}", "Result from kernel:"); }

I get (mkdir _build && cd _build && cmake .. && ninja

[5/5] Linking CXX executable try FAILED: try : && /opt/intel/oneapi/compiler/2021.3.0/linux/bin/dpcpp CMakeFiles/try.dir/try.cc.o -o try fmt/libfmt.a && : InvalidModule: Invalid SPIR-V module: Casts from private/local/global address space are allowed only to generic llvm-foreach: dpcpp: error: llvm-spirv command failed with exit code 1 (use -v to see invocation) ninja: build stopped: subcommand failed.

Interestingly enough, if I add the -fintelfpga flag to the linker, this actually completes (apparently) successfully. I'm guessing this is something to do with what dpcpp assumes the standard targets are?

Am I not supposed to be compiling other libraries with dpcpp and trying to isolate it (in which case, CMake becomes tricker, as multiple compilers for the same language isn't so easy, and care needs to be taken to sync up ABI versions, etc.).

Otherwise, is there some way I can work around or understand what is causing this issue, preferably without abandoning other libraries?

Re: dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Thu, 09 Sep 2021 13:02:16 GMT

Possibly (dpcpp-specific) related: https://github.com/intel/llvm/issues/3172

It looks like I can bypass the specific issue by using the header-only version of the library here, but it'd be good to understand if it's me or the compiler being dumb here.

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ShivaniK_Intel — Mon, 13 Sep 2021 12:05:01 GMT

Hi,

Thanks for reaching out to us.

Thanks for providing the details. We are able to reproduce the issue. We are working on it and will get back to you soon.

Thanks & Regards

Shivani

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

Alex_Y_Intel — Sat, 18 Sep 2021 09:44:35 GMT

Please help me clarify these questions:

Does the problem occur when you try to compile the files in "source" directory first to be static libraries and link them later?
For the content of CMakeList.txt you've provided, are those supposed to be additional to the original CMakeList.txt or as a new CMakeList.txt consists of only these lines?
Can I suppose that it has nothing to do with cmake version?
When you mentioned about adding -fintelfpgaflag to the linker, can you demonstrate how you add it?
Can you provide a smaller reproducer to show the problem? (trim off the unnecessary files and compilation from the original package)

Thanks.

Re: Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Sat, 18 Sep 2021 11:18:29 GMT

Hi,

5) I'm not sure how to reduce the example further since it's 10 lines of CMake and 5 lines of C++, beyond removing the external library. I'm not sure how to further narrow down the source of the issue since the external library successfully compiles, just fails to link it in. I have split out the demonstration into a standalone repository: https://github.com/ndevenish/demonstrate_invalid_module . Starting from a devcloud login node:

$ qsub -I $ git clone --recursive https://github.com/ndevenish/demonstrate_invalid_module.git $ cd demonstrate_invalid_module/ $ mkdir _build && cd _build $ cmake .. $ make -i [ 11%] Building CXX object fmt/CMakeFiles/fmt.dir/src/format.cc.o [ 22%] Building CXX object fmt/CMakeFiles/fmt.dir/src/os.cc.o [ 33%] Linking CXX static library libfmt.a [ 33%] Built target fmt [ 44%] Building CXX object CMakeFiles/try.dir/try.cc.o [ 55%] Linking CXX executable try InvalidModule: Invalid SPIR-V module: Casts from private/local/global address space are allowed only to generic llvm-foreach: dpcpp: error: llvm-spirv command failed with exit code 1 (use -v to see invocation) [ 55%] Built target try [ 66%] Building CXX object CMakeFiles/try_intelfpga.dir/try.cc.o [ 77%] Linking CXX executable try_intelfpga [ 77%] Built target try_intelfpga [ 88%] Building CXX object CMakeFiles/try_header_only.dir/try.cc.o [100%] Linking CXX executable try_header_only [100%] Built target try_header_only

1) To try what I think you are asking, I tried separating out the call to `fmt::print` into a `call_fmt.a` static library, that privately linked `libfmt.a`. To avoid making the simple example messy, I made these changes in a separate branch: https://github.com/ndevenish/demonstrate_invalid_module/tree/try_separate_lib . The issue still happened on the linking of this intermediate library into the target executable.

2) This CMakeLists.txt was an entirely new standalone CMakeList.txt to demonstrate this issue.

3) The only possible way I can think this would affect things was in the choice of the C++ standard, as it chooses -std=gnu++17. I checked the build with -std=c++17 and the same error occurs. Otherwise, the actual compile/link instructions emitted look perfectly normal. Setting C++20 as the minimum also has this issue.

4) I added a second target to the CMakeLists.txt in the example repository that adds the -fintelfpga flag. This applies the flag to the link stage, and successfully compiles and runs.

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

Alex_Y_Intel — Tue, 21 Sep 2021 09:35:12 GMT

Does the problem occur when you change the CMakeLists.txt to the new shorter version you provided, or does it also exist with the original CMakeLists.txt?

I've tried your new shorter version of CMakeLists.txt but it shows error as there's no fmt subdirectory. Are you trying to add the subdirectory "include/fmt" instead?

If I just run with the original CMakeLists.txt, but instead of using "ninja," I use "make," the project actually builds successfully with unicode-test created under the _build/bin directory.

Re: Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Tue, 21 Sep 2021 11:38:02 GMT

Hi Ayu1,

@Alex_Y_Intel wrote:

Does the problem occur when you change the CMakeLists.txt to the new shorter version you provided, or does it also exist with the original CMakeLists.txt?

The second version in the repo is actually longer, because it includes a demonstration of the -fintelfpga flag. And yes, the issue happens with both.

I've tried your new shorter version of CMakeLists.txt but it shows error as there's no fmt subdirectory. Are you trying to add the subdirectory "include/fmt" instead?

This sounds like you didn't git clone --recursive. I included fmt as a git submodule in the example so that it would be easier to get an identical environment, but you need to either tell git to clone recursively or git submodule init[update] if you just cloned normally. Otherwise, just cloning the repository into the fmt/ subdirectory should be good enough.

If I just run with the original CMakeLists.txt, but instead of using "ninja," I use "make," the project actually builds successfully with unicode-test created under the _build/bin directory.

Apologies for the confusion, I used "Ninja" in the original example, because I have that set as my CMake default generator.

However, using Make gives identical results for me on devcloud; unicode-test isn't actually the output from this build, and isn't built by fmt unless you explicitly enable testing for fmt (with -DFMT_TEST=on), or possibly build fmt separately. Did you accidentally run your build from within the fmt/ subfolder? That would pass the build if you don't tell it to use dpcpp explicitly (e.g. with mkdir _build_fmt && cd _build_fmt && CXX=dpcpp cmake ../fmt). If there isn't a failure or a try executable in the root of the build folder then it hasn't built the correct CMakeLists.txt. Additionally, the start of the CMake configure should look like:

-- The CXX compiler identification is IntelLLVM 2021.3.0

if you are properly configured to use dpcpp.

To avoid all uncertainty, here is a Dockerfile that replicates the issue cleanly:

FROM intel/oneapi-basekit RUN apt-get update && apt-get install -y git RUN cd /opt && git clone --recursive https://github.com/ndevenish/demonstrate_invalid_module RUN mkdir /opt/demonstrate_invalid_module/build WORKDIR /opt/demonstrate_invalid_module/build RUN cmake .. RUN make

And running "docker build ." gives:

#10 31.33 [ 55%] Linking CXX executable try #10 32.68 InvalidModule: Invalid SPIR-V module: Casts from private/local/global address space are allowed only to generic #10 32.68 #10 32.69 llvm-foreach: #10 32.69 dpcpp: error: llvm-spirv command failed with exit code 1 (use -v to see invocation)

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

Alex_Y_Intel — Wed, 22 Sep 2021 07:42:33 GMT

I've tried your new source code but with cmake version 3.16:

I can reproduce your problem now but with intelfpga option, I still see the error:

cmake_minimum_required(VERSION 3.10 FATAL_ERROR)

set(CMAKE_CXX_COMPILER dpcpp)

set(CMAKE_CXX_STANDARD 17)

project(dpcpp CXX)

add_subdirectory(fmt)

# Standard executable, linking external library

#add_executable(try try.cc)

#target_link_libraries(try fmt::fmt)

# Linking to the header-only version of this library works

#add_executable(try_header_only try.cc)

#target_link_libraries(try_header_only fmt::fmt-header-only)

# As does adding the -fintelfpga flag

add_executable(try_intelfpga try.cc)

target_link_options(try_intelfpga PRIVATE -fintelfpga)

target_link_libraries(try_intelfpga fmt::fmt)

Scanning dependencies of target fmt

[ 20%] Building CXX object fmt/CMakeFiles/fmt.dir/src/format.cc.o

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format.h:1783:33: warning: comparison with infinity always evaluates to false in fast floating point modes [-Wtautological-constant-compare]

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format.h:1989:20: note: in instantiation of function template specialization 'fmt::detail::write<char, fmt::appender, float, 0>' requested here

return detail::write(out, value, specs, locale);

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format.h:2653:10: note: in instantiation of member function 'fmt::detail::vformat_to(buffer<char> &, basic_string_view<char>, basic_format_args<basic_format_context<detail::buffer_appender<type_identity_t<char>>, type_identity_t<char>>>, fmt::detail::locale_ref)::format_handler::on_format_specs' requested here

struct format_handler : error_handler {

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:53:31: note: in instantiation of function template specialization 'fmt::detail::vformat_to<char>' requested here

template FMT_API void detail::vformat_to(

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return detail::write(out, value, specs, locale);

struct format_handler : error_handler {

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:53:31: note: in instantiation of function template specialization 'fmt::detail::vformat_to<char>' requested here

template FMT_API void detail::vformat_to(

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return detail::write(out, value, specs, locale);

struct format_handler : error_handler {

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:53:31: note: in instantiation of function template specialization 'fmt::detail::vformat_to<char>' requested here

template FMT_API void detail::vformat_to(

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format.h:1836:33: warning: comparison with infinity always evaluates to false in fast floating point modes [-Wtautological-constant-compare]

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format.h:1969:12: note: in instantiation of function template specialization 'fmt::detail::write<char, fmt::appender, float, 0>' requested here

return write<Char>(out, value);

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/core.h:1469:12: note: in instantiation of function template specialization 'fmt::detail::default_arg_formatter<char>::operator()<float>' requested here

return vis(arg.value_.float_value);

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return write<Char>(out, value);

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/core.h:1471:12: note: in instantiation of function template specialization 'fmt::detail::default_arg_formatter<char>::operator()<double>' requested here

return vis(arg.value_.double_value);

5 warnings generated.

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return detail::write(out, value, specs, locale);

struct format_handler : error_handler {

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:53:31: note: in instantiation of function template specialization 'fmt::detail::vformat_to<char>' requested here

template FMT_API void detail::vformat_to(

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return detail::write(out, value, specs, locale);

struct format_handler : error_handler {

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:53:31: note: in instantiation of function template specialization 'fmt::detail::vformat_to<char>' requested here

template FMT_API void detail::vformat_to(

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return detail::write(out, value, specs, locale);

struct format_handler : error_handler {

/nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:53:31: note: in instantiation of function template specialization 'fmt::detail::vformat_to<char>' requested here

template FMT_API void detail::vformat_to(

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return write<Char>(out, value);

return vis(arg.value_.float_value);

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/src/format.cc:8:

In file included from /nfs/pdx/home/ayu1/demonstrate_invalid_module/fmt/include/fmt/format-inl.h:29:

return write_nonfinite(out, std::isinf(value), specs, fspecs);

^~~~~~~~~~~~~~~~~

return write<Char>(out, value);

return vis(arg.value_.double_value);

5 warnings generated.

[ 40%] Building CXX object fmt/CMakeFiles/fmt.dir/src/os.cc.o

[ 60%] Linking CXX static library libfmt.a

[ 60%] Built target fmt

Scanning dependencies of target try_intelfpga

[ 80%] Building CXX object CMakeFiles/try_intelfpga.dir/try.cc.o

[100%] Linking CXX executable try_intelfpga

InvalidModule: Invalid SPIR-V module: Casts from private/local/global address space are allowed only to generic

llvm-foreach:

dpcpp: error: llvm-spirv command failed with exit code 9 (use -v to see invocation)

make[2]: *** [CMakeFiles/try_intelfpga.dir/build.make:85: try_intelfpga] Error 9

make[1]: *** [CMakeFiles/Makefile2:96: CMakeFiles/try_intelfpga.dir/all] Error 2

make: *** [Makefile:84: all] Error 2

I understand that your original CMakeLists.txt states to use cmake version 3.20, is it a must? If you use an older version, will the last case with intelfpga option build successfully?

Re: Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Wed, 22 Sep 2021 08:48:45 GMT

Hi Ayu1,

No, it doesn't require CMake 3.20, but I haven't been building or testing on lower versions. However, using 3.16 it still exhibits the same behaviour for me, and those warnings aren't emitted - so you must have some other settings differently. This Dockerfile reproduces the same failure with CMake 3.16, and demonstrates that try_intelfpga also compiles successfully:

FROM intel/oneapi-basekit RUN apt-get update && apt-get install -y git wget RUN cd /opt && git clone --recursive https://github.com/ndevenish/demonstrate_invalid_module RUN mkdir /opt/demonstrate_invalid_module/build # Get CMake 3.16 RUN cd /opt && \ wget https://github.com/Kitware/CMake/releases/download/v3.16.0/cmake-3.16.0-linux-x86_64.tar.gz && \ mkdir cmake && \ tar -xC cmake --strip-components=1 -f cmake-* ENV PATH=/opt/cmake/bin:$PATH # Rewrite the CMakeList to Match Ayu1's (note: <3.13 will not work due to target_link_options) RUN sed -i s/3.20/3.10/ /opt/demonstrate_invalid_module/CMakeLists.txt WORKDIR /opt/demonstrate_invalid_module/build RUN cmake --version && cmake .. # Explicitly build try_intelfpga first RUN make try_intelfpga &&\ echo &&\ make

And the output from the last stage:

Scanning dependencies of target try_intelfpga [ 80%] Building CXX object CMakeFiles/try_intelfpga.dir/try.cc.o [100%] Linking CXX executable try_intelfpga [100%] Built target try_intelfpga [ 33%] Built target fmt Scanning dependencies of target try [ 44%] Building CXX object CMakeFiles/try.dir/try.cc.o [ 55%] Linking CXX executable try InvalidModule: Invalid SPIR-V module: Casts from private/local/global address space are allowed only to generic

It's puzzling that you get both more warnings and an extra failure - are you sure you are using the same compiler? However, I feel that focussing on CMake is probably chasing a red herring.

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

Alex_Y_Intel — Wed, 22 Sep 2021 09:12:03 GMT

I've done another experiment, please let me know what you think--

I've used the CMakeLists.txt in fmt directory of your original source code to build libfmt.a. Now just for convenience, I put try.cc and newly built libfmt.a both under fmt/include, and I did this:

dpcpp try.cc libfmt.a

which created a.out, and then I ran a.out which gave me the proper result: Output.

By the way, I also modified the try.cc a little bit: I changed <fmt/core.h> to "fmt/core.h"

So, it seems dpcpp compiler has no problem linking the library.

Another thing you can help me clarify is that, since the project itself is small, can you arrange the files to be compiled and linked with dpcpp commands instead of using CMake?

Re: Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Wed, 22 Sep 2021 12:42:21 GMT

Hi Ayu1,

Did you use dpcpp to build libfmt.a, or used a separate compiler? Yes, it works if you don't use dpcpp, but that's besides the point - as far as I am aware, dpcpp is supposed to act like a normal C++ compiler (and cross-linking between compilers potentially raises ABI issues, in any case). Here's a Dockerfile that replicates the issue, but with a separate compilation of libfmt.a. When compiling separately, you have to explicitly request only libfmt be built, because the linking fails with the same SPIR-V error.

FROM intel/oneapi-basekit RUN apt-get update && apt-get install -y git wget RUN cd /opt && git clone --recursive https://github.com/ndevenish/demonstrate_invalid_module RUN mkdir /opt/demonstrate_invalid_module/build WORKDIR /opt/demonstrate_invalid_module/build RUN cmake --version && cmake .. # Build libfmt separately RUN mkdir /opt/build_fmt &&\ cd /opt/build_fmt &&\ CXX=dpcpp cmake /opt/demonstrate_invalid_module/fmt &&\ make fmt VERBOSE=1 RUN dpcpp ../try.cc -I../fmt/include /opt/build_fmt/libfmt.a

I am not sure what you are trying to achieve here?

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

Alex_Y_Intel — Thu, 23 Sep 2021 09:13:07 GMT

We've identified the bug. We'll work on this issue internally and get back to you with update.

Re:dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

Alex_Y_Intel — Wed, 06 Oct 2021 07:24:59 GMT

The problem has been resolved and will be rolled out in future release.

Re: dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Wed, 10 Nov 2021 12:01:22 GMT

Hi ayu1,

Was this supposed to be addressed in 2021.4, or was that too early for this to be fixed? Do you have any information on a public link to a ticket or repo fix for this? Or a suggested workaround? I've tried switching to the 2021.4 docker image and am finding that it's now happening more often, even sometimes with the header-only variants or -fintelfpga (it seems somewhat intermittent and I haven't managed to duplicate it on a reduced example yet).

Re: dpcpp gives "InvalidModule: Invalid SPIR-V..." on linking

ndevenish — Thu, 28 Jul 2022 11:49:58 GMT

This appears to be solved sometime between 2021.4 and 2022.1. On 2022.1 this now compiles.