SPIR-V from clang/llvm-spirv fails or crashes with OpenCL runtime on Windows

[pxcc]

Overview

I'm seeing a range of failures when attempting to execute SPIR-V from clang/llvm-spirv with the Intel OpenCL runtime on Windows. Depending on the driver version and kernel source, the failures include CL_INVALID_KERNEL_NAME when retrieving a kernel, CL_INVALID_KERNEL_ARGS when launching a kernel, or a crash in clBuildProgram.

When kernels are compiled to SPIR-V with the ocloc offline compiler from oneAPI, everything works fine. However, ocloc only compiles C kernels. Support for C++ kernels is rapidly becoming a gating requirement for device support for the project (a consumer desktop application). The Metal, Cuda (driver API), and CPU backends for the project all support C++ kernels, and this is turning out to be indispensible for managing the kernel-side combinatorics of types, dimensions, memory spaces, function blocks, and so on.

Environment

Device:  Intel UHD Graphics 630, i7-10700K processor
Driver:  Intel graphics driver 31.0.101.2125, 30.0.101.1273
OS:  Windows 11 Pro 10.0.22621
LLVM:  17.0.0 from main on July 9, 2023
ocloc:  2023.1.0, from oneAPI installation

Code to reproduce

See attached .zip file. Contents:

main.cpp -- Host code
kernel.cl -- C kernel that results in execution errors
kernel.clcpp -- C++ kernel that results in execution errors
crash_kernel.cl -- C kernel that results in execution crash
build.bat -- Build script for host app & kernels
test.exe -- prebuilt host app binary
*.spv -- prebuilt SPIR-V bitcode
*.spirv-dis -- disassembly of *.spv using spirv-dis

Build instructions

1. Edit paths at top of build.bat for your system
2. Start a Visual Studio command prompt (i.e. with vcvars64.bat loaded)
3. Run build.bat

This should produce the following:

test.exe -> host application
kernel_cl_ocloc.spv -> kernel.cl compiled with ocloc
kernel_cl_clang.spv -> kernel.cl compiled with clang/llvm-spirv
kernel_clcpp_clang.spv -> kernel.clcpp compiled with clang/llvm-spirv
crash_kernel_cl_ocloc.spv -> crash_kernel.cl compiled with ocloc
crash_kernel_cl_clang.spv -> crash_kernel.cl compiled with clang/llvm-spirv

Results

With driver 31.0.101.2125:

test kernel_cl_ocloc.spv -> succeeds
test kernel_cl_clang.spv -> fails with  CL_INVALID_KERNEL_ARGS
test kernel_clcpp_clang.spv -> fails with  CL_INVALID_KERNEL_ARGS 
test crash_kernel_cl_ocloc.spv -> succeeds
test crash_kernel_cl_clang.spv -> clBuildProgram crashes

With driver 30.0.101.1273:

test kernel_cl_ocloc.spv -> succeeds
test kernel_cl_clang.spv -> fails with CL_INVALID_KERNEL_NAME
test kernel_clcpp_clang.spv -> fails with CL_INVALID_KERNEL_NAME
test crash_kernel_cl_ocloc.spv -> succeeds
test crash_kernel_cl_clang.spv -> fails with CL_INVALID_KERNEL_NAME

Success with ocloc and various C kernels establishes that the basic SPIR-V -> device execution chain is sane at least for C kernels, both with the sample code here and with the (much more elaborate) cross-API framework that I mentioned above.  So the problem seems to be either in the Intel OpenCL runtime on Windows, or in the SPIR-V produced by clang/llvm-spirv (but even then it shouldn't crash the driver).  Documentation and slideware from Khronos and Intel imply that the SPIR-V from clang/llvm-spirv should be viable with Intel drivers for both C and C++ kernels, and comments elsewhere indicate some success on Linux. 

Level Zero: I could implement a Level Zero backend if that would make a difference, and it would probably be a better long-term solution. However, if Level Zero relies on the same SPIR-V-to-device path that the OpenCL runtime is using, then I'll just end up in the same spot.

[Anita_Intel]

Dear User,

Thank you for reaching out to us. We are reproducing your issue and get back to you shortly.

Thanks

[Anita_Intel · Accepted solution]

Dear User,

We used below steps to reproduce this issue,

1. set below paths in build.bat
   1. set opencl_hdr_dir="C:\Program Files (x86)\IntelSWTools\system_studio_2020\OpenCL\sdk\include"
   2. set opencl_lib_dir="C:\Program Files (x86)\IntelSWTools\system_studio_2020\OpenCL\sdk\lib\x64"
   3. set ocloc="C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\lib\ocloc\gen9-11\ocloc.exe"
   4. set clang="C:\Program Files (x86)\Intel\oneAPI\compiler\2023.2.0\windows\bin-llvm\clang.exe"
   5. set llvm_spirv="C:\Program Files (x86)\Intel\oneAPI\compiler\2023.2.0\windows\bin-llvm\llvm-spirv.exe"
2. downloaded OpenCL API C++ bindings from the Khronos link: https://codeload.github.com/KhronosGroup/OpenCL-CLHPP/zip/refs/heads/main
   1. copy both header files "cl2.hpp and opencl.hpp" from location OpenCL-CLHPP-main\include\CL to opencl sdk header "C:\Program Files (x86)\IntelSWTools\system_studio_2020\OpenCL\sdk\include\CL"
3. Loaded "c:\Program Files (x86)\Intel\oneAPI\setvars.bat"
4. Run build.bat
5. All opencl kernels compile successfully and test.exe gets created and run on GPU.

Could you please follow same steps and let us know issues?

If you are having some other concerns, please write specifically and send us logs; we can investigate further if any incompatibility in the SPIR-V generated.

Thanks

[pxcc]

Hi Anita,

I did a quick test using the 2023.1 clang/llvm-spirv from my current oneAPI installation, and that was sufficient to make the test cases work.  I didn't need to update the OpenCl headers.

However, my actual application code is still crashing with that change, and I'm in the middle of another task right now. As soon as I get to a stopping point, I'll install the July 20 oneAPI release, update OpenCL headers from Khronos, and go through my code to see if I can resolve any discrepancies with the test code. It might be a couple of days before I can dig into it. I'll report back when I know more.

Thanks for your help with this, I appreciate it.

[pxcc]

Success!  It's working now.  I had forgotten to tickle part of my CMake superbuild and it wasn't picking up the new compiler path.

I switched on cl-std=CLC++2021  for the kernel language and so far templates and lambdas and whatnot are compiling and executing without issue.

This pleases me greatly.

Thanks again for your help.