Application crashes: OMP: Error #136: Cannot create thread

Ying_H_Intel · ‎08-14-2012

It was move from IPP forum. http://redfort-software.intel.com/en-us/forums/showthread.php?t=106951

I have two Windows 7 SP1 x64 dev machines that have reliability problems unless I set KMP_ALL_THREADS environment variable. If I don't set that environment variable, I get application crashes in Windows save (as) file dialogs. When the app crashes, it pops a separate console window to display errors. The errors I see are like this:

OMP: Error #136: Cannot create thread.

OMP: System error #8: Not enough storage is available to process this command.

And then the application crashes. The applications that have crashed like this are Outlook Express, VirtualDub, Visual Studio 2010, and a tiny C++ test ui I wrote that just opens a file save dialog. These machines both have 8GB of RAM, I am not low on RAM according to task manager. I have generally had only Visual Studio running.

When I execute the test ui and point to a directory without a jpeg, it works fine. When I point to a directory with a jpeg (as the above steps do), it crashes and pops a console window with this:

OMP: Error #136: Cannot create thread.

OMP: System error #8: Not enough storage is available to process this command.

OMP: Error #178: Function GetExitCodeThread() failed:

OMP: System error #6: The handle is invalid.

When I run under the visual studio debugger I don't get the crash.

The crash is perfectly repeatable, I haven't seen it not crash given the specified run conditions.

If I set the system environment variable KMP_ALL_THREADS (I tried 0,1,16 and 4096) or OMP_THREAD_LIMIT the crash does not happen. With system environment variable OMP_NUM_THREADS set to 1 or 16 the crash still happens. With system environment variable KMP_STACKSIZE and/or OMP_STACKSIZE set to 4m the crash still happens.

But when I set KMP_ALL_THREADS to 1, 16, or 4096 (probably anything) I get this when I run another app that uses IPP:

OMP: Warning #96: Cannot form a team with 8 threads, using 1 instead.

OMP: Hint: Consider unsetting KMP_ALL_THREADS and OMP_THREAD_LIMIT (if either is set).

So I either have crashes and a console window with errors with windows file save as dialog, or a console window with errors on an app that uses IPP. Neither is good.

What is going on here? Is Windows using Intel OMP? What does "not enough storage" mean if I'm not low on memory or disk? Why can it not form a team of 8 threads with KMP_ALL_THREADS set to 4096?

RyanS · ‎08-14-2012

Some of your questions indicate that you didn't notice this sentence: "The applications that have crashed like this are (Microsoft) Outlook Express, VirtualDub, (Microsoft) Visual Studio 2010, and a tiny C++ test ui I wrote that just opens a file save dialog."

All of these programs are 32-bit, I think.
I don't know that any of these applications are using IPP. However they appear to be using Intel OpenMP because the crash is affected by an Intel-specific environment variable KMP_ALL_THREADS. Is it correct that KMP_ALL_THREADS is Intel-specific?

So it is like the Windows operating system itself has the problem, not any particular application.

One PC is Nehalem and the other is Ivy Bridge. They have different motherboards. So I don't think hardware is the issue.

Ying_H_Intel · ‎08-14-2012

Application crashes: OMP: Error #136: Cannot create thread.

Hi Ryans,

No, we notice the scenario you mentioned. (none of the apllication Outlook Express, VirtualDub, (Microsoft) Visual Studio 2010, and a tiny C++ test ui use IPP, so it should not related to IPP). But as you post the problem to IPP forum, so we need to make sure if it is IPP-related or Intel Compiler related.

But you should be right about KMP_ALL_THREADS is Intel-specific. So the problem appear to be related Intel OpenMP.
As Sergery mentioned, it is impossible to catch the reason without a test case, you may do general check, like check how many threads started in task manager etc.

( i may try to move your post to Intel Compiler forum and see if there are some insight from Compiler experts)
the new post in Intel Compiler Forum is http://redfort-software.intel.com/en-us/forums/showthread.php?t=107255

Best Regards,
Ying

SergeyKostrov · ‎08-19-2012

Quoting Ying H (Intel)

...When the app crashes, it pops a separate console window to display errors. The errors I see are like this:
OMP: Error #136: Cannot create thread.
OMP: System error #8: Not enough storage is available to process this command.
...

Some time ago I had a similar problem with a stress test-case ( based on TBB library ) and it was creating hundreds of threads.
In order to create as many as possible threads I used the following settings for the linker:

...
#pragma message ( "*** New HEAP Commit:Reserve and STACK Commit:Reserve Values Defined ***" )

#ifdef _DEBUG
// Case 1
// #pragma comment( linker, "/HEAP:1069547520,1069547520" )
// #pragma comment( linker, "/STACK:4194304,4194304" )

// Case 2
// #pragma comment( linker, "/HEAP:134217728,134217728" )
// #pragma comment( linker, "/STACK:134217728,134217728" )

// Case 3
// #pragma comment( linker, "/HEAP:268435456,268435456" )
// #pragma comment( linker, "/STACK:268435456,268435456" )

     // Case 4
     #pragma comment( linker, "/HEAP:536870912,536870912" )
     #pragma comment( linker, "/STACK:536870912,536870912" )
#endif

#ifdef NDEBUG
// Case 1
// #pragma comment( linker, "/HEAP:1069547520,1069547520" )
// #pragma comment( linker, "/STACK:4194304,4194304" )

// Case 2
// #pragma comment( linker, "/HEAP:134217728,134217728" )
// #pragma comment( linker, "/STACK:134217728,134217728" )

// Case 3
// #pragma comment( linker, "/HEAP:268435456,268435456" )
// #pragma comment( linker, "/STACK:268435456,268435456" )

     // Case 4
     #pragma comment( linker, "/HEAP:536870912,536870912" )
     #pragma comment( linker, "/STACK:536870912,536870912" )
#endif
...

All my tests were done on a 32-bit Windows XP.

Best regards,
Sergey

SergeyKostrov · ‎08-19-2012

I'd like to answer some of questions asked by Rayn.

Quoting Ying H (Intel)

What is going on here?

     [SergeyK] It looks like some problem with memory allocation. However, it is not clear why some 3rd party applications crashed.

Is Windows using Intel OMP?

     [SergeyK] No. Microsoft has its own version of OpenMP.

What does "not enough storage" mean if I'm not low on memory or disk?

     [SergeyK] When Windows creates a thread it needs some memory to create a stack. If there is no more memory
                        available Windows doesn't create a thread and a user application could crash if there is no an error processing.

                       Error #136 is returned by from OpenMP library
                        System Error #8 is returned from a 'GetLastError' Win32 API function

Why can it not form a team of 8 threads with KMP_ALL_THREADS set to 4096?

     [SergeyK] With Intel OpenMP library you should be able to do it.

SergeyKostrov · ‎08-19-2012

Hi Ryan,

Quoting Sergey Kostrov

...Why can it not form a team of 8 threads with KMP_ALL_THREADS set to 4096?

[SergeyK] With Intel OpenMP library you should be able to do it.

Here is a link to a thread:

Stress testing of Intel OpenMP library - More than 18,600 OpenMP threads created in a parallel region
http://redfort-software.intel.com/en-us/forums/showthread.php?t=105139&o=a&s=lr

Please take a look.

Best regards,
Sergey

JenniferJ · ‎08-29-2012

Can you upload the program that is built with Intel C++ compiler or using the IPP libs? Also how you build and run.

do you have other versions of Intel C++ Compiler installed or IPP installed? make sure the correction openmp dll are use (the path should point to the correct openmp directory).

Jennifer

RyanS · ‎09-12-2012

Some of the comments seem to misunderstand the problem. Note this sentence: "The applications that have crashed like this are (Microsoft) Outlook Express, VirtualDub, (Microsoft) Visual Studio 2010, and a tiny C++ test ui I wrote that just opens a file save dialog." The C++ tiny test UI I wrote doesn't use IPP, it just opens the Windows file save dialog. I am not using the Intel C++ compiler for my test UI, nor is it likely that the other affected applications use it. I do not own the Intel C++ compiler, however I see various Intel products have been installed possibly as part of the IPP installation or the Intel Media SDK installation. I am not sure how to check which openmp dll would be used. I have not done OpenMP development before. From an internet search, I see mention of "vcomp.dll" but that is not in my path.

TimP · ‎09-13-2012

If you include objects built with CL /openmp in your build which also uses ICL or ifort , your link step must be set up so as to use -defaultlib:libiomp5md.lib -nodefaultlib:vcomp.lib -nodefaultlib:vcompd.lib This should happen automatically if you use the Intel compiler with option /Qopenmp for the link step. I'm confused about what all is asked in this thread, but I'm answering the part about choosing the correct OpenMP support version, without knowing whether it's relevant to the other questions. You can verify the OpenMP libraries, e.g. by "dumpbin /dependents your.exe" : ... File Type: EXECUTABLE IMAGE Image has the following dependencies: libiomp5md.dll KERNEL32.dll imagehlp.dll That is, you must not see either of the vcomp .dll when you are using an Intel compiler, even though part of your build is done by CL.

RyanS · ‎09-17-2012

Thanks for the suggestions. dumpbin /dependents gives: user32.dll, comdlg32.dll, msvcr100d.dll, and kernel32.dll (for the debug build). I also tried Dependency Walker. Depends.exe did not show a dependency on libiomp*.dll, nor vcomp.dll. I tried to "profile" using depends.exe but it hung, regardless of settings. So I am still unable to tell why my trivial windows exe is failing with an OpenMP problem that depends on Intel-specific environment variables.

SergeyKostrov · ‎09-18-2012

>>...So I am still unable to tell why my trivial windows exe is failing with an OpenMP problem that depends on Intel-specific environment variables... Hi, Could you upload a VS project that reproduces the problem? If there are some proprietery pieces of codes in your "trivial application" you could remove them. Best regards, Sergey

RyanS · ‎09-20-2012

Because of the list of applications that crash ((Microsoft) Outlook Express, VirtualDub, (Microsoft) Visual Studio 2010, and my tiny ui) I have concluded that this is a platform problem, and that it doesn't have anything to do with my source code. The trivial windows exe is just the default windows app that visual studio creates for you when you create a new Windows GUI project, and then I took over the Help->About menu item and opened a Windows Save File Dialog instead. There is no proprietary code, nor is there anything that is likely to help this investigation, except an easy way to pop a Windows Save File Dialog. But I will upload it anyway. This is a platform problem. The Windows Save File Dialog is crashing when pointed to a directory with a jpeg in it. And the crash depends on an Intel-specific OpenMP environment variable.

RyanS · ‎09-20-2012

Per Ying's suggestion on the original thread, I ran with KMP_AFFINITY=verbose. Here is the output: OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7} OMP: Info #156: KMP_AFFINITY: 8 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores) OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1,2,3,4,5,6,7} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1,2,3,4,5,6,7} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {0,1,2,3,4,5,6,7} OMP: Info #147: KMP_AFFINITY: Internal thread 8 bound to OS proc set {0,1,2,3,4,5,6,7} ..
And then several hundred more of these (up to thread numbers over 500) and then crashes with the errors I listed in my original post.

SergeyKostrov · ‎09-20-2012

>>...There is no proprietary code, nor is there anything that is likely to help this investigation, except an easy way to pop a Windows Save File >>Dialog. But I will upload it anyway... >>... Thank you. I'll take a look at your VS 2010 project. By the way, did you try to verify dependencies ( with MS Depends utility ) of your test application as soon as it is built? Best regards, Sergey

RyanS · ‎09-21-2012

Yes, in my post of 9/17 I was working with a fresh build.

SergeyKostrov · ‎09-25-2012

>>...There is no proprietary code, nor is there anything that is likely to help this investigation, except an easy way to pop a Windows Save File >>Dialog. But I will upload it anyway... >>... . I tested your project and I didn't have any problems. When I selected 'About' menu item the test application opened the 'Save File' dialog, then I selected some file ( under debugger I checked that everything was right ) and then selected 'Exit' menu irem. So, No problems at all. Could you upload a file that is located at a path: . . Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" . I'm interested to look at 'Microsoft.Cpp.Default.props' file. Thanks in advance. Best regards, Sergey

RyanS · ‎09-25-2012

Thanks for your effort. It is not likely that you will repro my crash, or you probably would have seen it yourself before, since it it not only caused by my test program. I noticed you didn't mention pointing the save dialog to a folder with a jpeg in it, so you may have missed that detail in the repro steps. I will upload the props file. Can I ask what you are looking for? Note that since this crash affects other programs not compiled on my machine, it is unlikely that my compiler or build configuration is related.

Ying_H_Intel · ‎09-25-2012

Hi Ryan Agree with you, it should be nothing to do with your code and exe. I just tried your code. It can run without any problem when point to a directory with a jpeg in it. and no any KMP information to show up. (as the exe or my file system doesn't trigger the openm program.) So This is your platform problem. The Windows Save File Dialog is crashing when pointed to a directory with a jpeg in it. And the crash depends on an Intel-specific OpenMP environment variable. The key questionare 1) which application call the libiomp5md.dll on your machine? Could you please search in your machine and see where are the libiomp5md.dll located besides IPP (intel composer XE folder, or if only in IPP folder, you can rename them and see your exe's behavious) not sure if you have tried it before. please set KMP_VERISON=1 and run your exe to see the library version information, and compare with the openmp dll on your machine? 2) It seems jpeg folder should trigger the application which have OpenMP library call. which default program are set for open a jpeg file? for exmaple, my system use windows Photo viewer. Best Regards, Ying

SergeyKostrov · ‎09-26-2012

Hi Ryan, >>...I noticed you didn't mention pointing the save dialog to a folder with a jpeg in it, so you may have missed that detail in the repro steps. . I did it. Sorry for not mentioning it. . >>...I will upload the props file. .. . Thank you and I'll take a look. . Ryan, could you reproduce your problem in Debug configuration of your test project? I'd like to see a complete output from 'Output' window of a Visual Studio. Let me know if you need some help. . Best regards, Sergey

SergeyKostrov · ‎09-26-2012

This is a short follow up... . >>>>...I will upload the props file. .. >>. >>Thank you and I'll take a look. . Hi Ryan, I did not find any problems with the props file. As I already mentioned "...I'd like to see a complete output from 'Output' window of a Visual Studio..." Best regards, Sergey

RyanS · ‎10-01-2012

Thanks Ying for pointing the right direction. The problem was that I have a windows installer that installs UIC dll's. The install builder automatically decided to register wic_uic_codec.dll with COM. So when Windows opened a save dialog on a folder with a jpeg, it was using that UIC dll (for a thumbnail, or just properties or something) and some of the other UIC dll's also (because when I deleted the other dll's the problem went away). The UIC dlls use Intel OpenMP, and somewhere there is a bug or maybe a version incompatibility or maybe something about the COM registration was wrong. I did not nail down the root cause nor try the newer versions of UIC dll's. The UIC dll's are the Intel-built ones and are from version 7.0.7. I told the installer not to register that UIC DLL and that solves my problem.