clBuildProgram infinite spinloop around cmpxchg/Sleep(0) and AV

[doug65536]

I am working on blender GPGPU renderer and I found that Intel OpenCL hangs (NVidia works fine with same code).

It isn't even computing yet, Intel OpenCL 1.5 hangs in the call to clBuildProgram.

intelocl.dll C:\\Program Files (x86)\\Intel\\OpenCL SDK\\1.5\\bin\\x64\\intelocl.dll 000007FEF95F0000-000007FEF9694000

xor ecx,ecx/call qword ptr [7FEF963B068h] is a call to Sleep(0)

This is the infinite spin loop:

000007FEF96140B0 BA 01 00 00 00       mov         edx,1  
000007FEF96140B5 33 C9                xor         ecx,ecx  
000007FEF96140B7 FF 15 AB 6F 02 00    call        qword ptr [7FEF963B068h]  (call to kernel32!Sleep)
000007FEF96140BD 33 C0                xor         eax,eax  
000007FEF96140BF F0 0F B1 BB 8C 00 00 00 lock cmpxchg dword ptr [rbx+8Ch],edi  
000007FEF96140C7 83 F8 05             cmp         eax,5  
000007FEF96140CA 75 E4                jne         000007FEF96140B0  

This pegs the core to 100%.

It never finishes, this thread just eats infinite CPU and clBuildProgram never returns.

This is the call stack:

KernelBase.dll!SleepEx()
intelocl.dll!000007fef96140bd()
[Frames below may be incorrect and/or missing, no symbols loaded for intelocl.dll]
intelocl.dll!000007fef960d0d8()
intelocl.dll!000007fef960cf13()
intelocl.dll!000007fef9609673()
intelocl.dll!000007fef95ff1bf()
intelocl.dll!000007fef95f65ed()
OpenCL.dll!000007fef9d71a9f()
> blender.exe!ccl::OpenCLDevice::build_kernel(const std::basic_string,std::allocator > & kernel_path="") Line 398 + 0x32 bytes C++
blender.exe!ccl::OpenCLDevice::compile_kernel(const std::basic_string,std::allocator > & kernel_path={...}, const std::basic_string,std::allocator > & kernel_md5="pE@
") Line 441 + 0xb bytes C++
blender.exe!ccl::OpenCLDevice::load_kernels(bool experimental=true) Line 498 + 0xa bytes C++
blender.exe!ccl::Session::run() Line 421 + 0x13 bytes C++
blender.exe!ccl::thread::run(void * arg=0x0000000000000000) Line 58 C++

Another thread throws an access violation exception:

000007FEE9800E29 80 7A 08 05 cmp byte ptr [rdx+8],5

rdx+8 = 0x0000000145962fb0

Call stack:

> OclCpuBackEnd.dll!000007fee9800e29()
[Frames below may be incorrect and/or missing, no symbols loaded for OclCpuBackEnd.dll]
OclCpuBackEnd.dll!000007fee9800dcb()
OclCpuBackEnd.dll!000007fee97fe7c4()
OclCpuBackEnd.dll!000007fee939a95d()
OclCpuBackEnd.dll!000007fee939a580()
OclCpuBackEnd.dll!000007fee939a270()
OclCpuBackEnd.dll!000007fee97de441()
OclCpuBackEnd.dll!000007fee939b1c0()
OclCpuBackEnd.dll!000007fee939ba90()
OclCpuBackEnd.dll!000007fee9271248()
OclCpuBackEnd.dll!000007fee926f149()
cpu_device.dll!000007fef12c8d52()
task_executor.dll!000007fef98a1fe3()
task_executor.dll!000007fef98a286d()
tbb.dll!0000000017f4a3b5()
tbb.dll!0000000017f48e8f()
tbb.dll!0000000017f47ba4()
tbb.dll!0000000017f45969()
msvcr90.dll!_endthreadex() + 0x47 bytes
msvcr90.dll!_endthreadex() + 0xe8 bytes
kernel32.dll!BaseThreadInitThunk() + 0xd bytes
ntdll.dll!RtlUserThreadStart() + 0x21 bytes

OclCpuBackEnd.dll C:\\Program Files (x86)\\Intel\\OpenCL SDK\\1.5\\bin\\x64\\OclCpuBackEnd.dll 1, 5, 0, 15293 9/14/2011 7:59 AM 000007FEE9260000-000007FEE9B93000

The program is running with pageheap enabled. (Enabled with debugging tools for windows gflags.exe)

[Shiri_M_Intel]

Hi
Can you send the kernel code?
Which HW/OS are you using?
Thanks, Shiri

[doug65536]

Kernel code is attached.

Hardware is Core I7 990x with 24GB.

OS is Windows 7 Ultimate SP1 64-bit.

Program is 64-bit C++ program built on VS2010.

Note again that the issue with compilation only reproduces when the gflags pageheap option is set for the executable. (In case you aren't familiar, the windows pageheap flag puts guard pages around all memory allocations, and offsets returned allocation pointers so any buffer overrun will AV)

In any case, even if pageheap is off and the buildprogram doesn't AV, the OpenCL kernel does not function correctly, where it works perfectly on NVidia GPU. (Everything is shades of white, as if it isn't shuffling correctly somewhere)

Thanks for looking into it.

[Shiri_M_Intel]

Hi
We will look into that
Thanks, Shiri

[Amjad_A_Intel]

Hi,

The file you attached contains ~21,000 lines. Not easy to debug at all.

Can you run preprocessor and generate a clean file that contains only the OpenCL kernels?

Also, to be sure that I understand the problem, is the error during building the kernel? or during execution?

In case the problem is during build, you might try using the ioc tool provided with the Intel OpenCL SDK to compile the kernel offline and assure that there is a problem during the compilation.

I tried running ioc on the file you sent us, and it seems that it contains syntaxt errors and compilation is failling (no crash though).

You can read more about the ioc in this link:

http://software.intel.com/en-us/articles/inspect-your-code-with-intel-opencl-sdk-offline-compiler/

I will be waiting for more input from you before I investigate this issue further.

Thanks,

Amjad