OMP crash during program exit

Jascha_Wetzel · ‎11-26-2012

On MacOS i'm getting a crash during program exit when using libiomp5.dylib from Composer XE 2013.1.119.
The crash does not occur when using

compiler and libiomp5 from 2013.0.088 or older versions
using libiomp5 from 2013.0.088 or older versions while still compiling with 2013.1.119 and linking aginst it's version of libiomp5

Here is the libiomp5 module info:

185 libiomp5.dylib - 0x111e48000 dyld Y Y /Users/jascha/turbulence/package/TurbulenceFD_C4D_v1-0_1038/TurbulenceFD/libs/libiomp5.dylib at 0x111e48000 (offset 0x111e48000)

and the stack trace:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000114a76f40
0x0000000111ebe612 in __kmp_cleanup_user_locks ()
(gdb) bt
#0 0x0000000111ebe612 in __kmp_cleanup_user_locks ()
#1 0x0000000111ece2ad in __kmp_internal_end ()
#2 0x0000000111eceb30 in __kmp_internal_end_dtor ()
#3 0x00007fff5fc0f15c in __dyld__ZN16ImageLoaderMachO13doTerminationERKN11ImageLoader11LinkContextE ()
#4 0x00007fff5fc03591 in __dyld__ZN4dyld14runTerminatorsEPv ()
#5 0x00007fff935a17c8 in __cxa_finalize ()
#6 0x00007fff935a1652 in exit ()
#7 0x00007fff9256ee5a in -[NSApplication terminate:] ()
#8 0x00007fff8dfad70d in -[NSObject performSelector:withObject:] ()
#9 0x00007fff9266b8ca in -[NSApplication sendAction:to:from:] ()
#10 0x00007fff927587ff in -[NSMenuItem _corePerformAction] ()
#11 0x00007fff92758536 in -[NSCarbonMenuImpl performActionWithHighlightingForItemAtIndex:] ()
#12 0x00007fff926d60d4 in -[NSMenu performKeyEquivalent:] ()
#13 0x00000001009cd3a6 in catch_exception_raise ()
#14 0x00007fff926d500e in -[NSView _performKeyEquivalent:conditionally:] ()
#15 0x00007fff926d5259 in -[NSControl _performKeyEquivalent:conditionally:] ()
#16 0x00007fff926d513c in -[NSView performKeyEquivalent:] ()
#17 0x00007fff926d500e in -[NSView _performKeyEquivalent:conditionally:] ()
#18 0x00007fff926d513c in -[NSView performKeyEquivalent:] ()
#19 0x00007fff926d500e in -[NSView _performKeyEquivalent:conditionally:] ()
#20 0x00007fff926d4f81 in -[NSWindow performKeyEquivalent:] ()
#21 0x00007fff926d4d7a in -[NSApplication _handleKeyEquivalent:] ()
#22 0x00007fff925cde86 in -[NSApplication sendEvent:] ()
#23 0x00007fff92564a0e in -[NSApplication run] ()
#24 0x00000001009d53d0 in catch_exception_raise ()
#25 0x0000000100002194 in ?? ()

jimdempseyatthecove · ‎11-27-2012

>>__kmp_cleanup_user_locks () Are (were) any of the lock objects located on the stack of a thread (which may have departed or poped out of existance)? Note, no crash in different version may indicate you were lucky. Jim Dempsey

Jascha_Wetzel · ‎11-28-2012

Thanks for your reply! There are no explicit lock objects in the program, i.e. no omp_init_lock or omp_init_nest_lock calls, only #pragma omp critical sections. Replacing all of them with custom locks prevents the crash. Why would a #pragma omp critical not clean up the underlying lock properly? At the time of the program exit there are no existing OMP threads except for the sleeping pooled threads. These have a stack trace like this: #0 0x00007fff93510bca in __psynch_cvwait () #1 0x00007fff935b2274 in _pthread_cond_wait () #2 0x0000000110e1c57e in __kmp_suspend () #3 0x0000000110def96c in __kmp_hyper_barrier_release () #4 0x0000000110df9852 in __kmp_fork_barrier () #5 0x0000000110df8362 in __kmp_launch_thread () #6 0x0000000110e17dee in __kmp_launch_worker () #7 0x00007fff935ae8bf in _pthread_start () #8 0x00007fff935b1b75 in thread_start ()

jimdempseyatthecove · ‎11-28-2012

>>Replacing all of them with custom locks prevents the crash. >>Why would a #pragma omp critical not clean up the underlying lock properly? Seems strange. Is it possible that you enter the critical section with one thread... one or more other threads attempt to enter critical section thread holding critical section exits or issues unhandled structured exception normal exit from critical section (never reached by holding thread). Jim Dempsey

Olga_M_Intel · ‎11-29-2012

Hello! We need some more information for further investigation. Please, provide the output from setting KMP_VERSION=1 and KMP_SETTINGS=1 environment variables. Also you mentioned the crash occurs on MacOS. Does it mean the crash dos not occur on Linux/Windows or you just tried only Mac? More information about application itself would be useful as well. Actually we need to know how the application uses OpenMP (constructs, API). It would be really useful to get a reproducer.

SergeyKostrov · ‎11-29-2012

>>...Replacing all of them with custom locks prevents the crash... Hi everybody, >>Is it possible that you enter the critical section with one thread... >>one or more other threads attempt to enter critical section >>thread holding critical section exits or issues unhandled structured exception >>normal exit from critical section (never reached by holding thread). I understand that the crash happens on a MacOS platform and a critical section is somehow involved. So, I simply would like to give an example how I do clean ups for a critical section on a Windows platform ( I know that it could be not applicable for MacOS... ). I hope that it could help: CDataContainer::~CDataContainer( RTvoid ) { ... while( m_cs.RecursionCount > 0 ) ::LeaveCriticalSection( &m_cs ); ::DeleteCriticalSection( &m_cs ); ::SetLastError( _RTCOMPLETED_SUCCESSFULLY ); } As you can see there is a 'while' loop with a call to a Win32 API function 'LeaveCriticalSection' in order to properly release the critical section completely (!).

Jascha_Wetzel · ‎11-30-2012

Olga Malysheva (Intel) wrote:
Please, provide the output from setting KMP_VERSION=1 and KMP_SETTINGS=1 environment variables.

Intel(R) OMP Copyright (C) 1997-2012, Intel Corporation. All Rights Reserved. Intel(R) OMP version: 5.0.20120914 Intel(R) OMP library type: performance Intel(R) OMP link type: dynamic Intel(R) OMP build time: 2012-09-14 06:12:26 UTC Intel(R) OMP build compiler: Intel C++ Compiler 12.1 Intel(R) OMP alternative compiler support: yes Intel(R) OMP API version: 3.1 (201107) Intel(R) OMP tracefile support: no Intel(R) OMP dynamic error checking: no Intel(R) OMP helper thread support: yes Intel(R) OMP thread affinity support: no Intel(R) OMP debugger support version: 1.1 User settings: KMP_SETTINGS=1 KMP_VERSION=1 Effective settings: KMP_ABORT_DELAY=0 KMP_ABORT_IF_NO_IRML=false KMP_ALIGN_ALLOC=64 KMP_ALL_THREADPRIVATE=128 KMP_ALL_THREADS=2147483647 KMP_ASAT_DEC=1 KMP_ASAT_FAVOR=0 KMP_ASAT_INC=4 KMP_ASAT_INTERVAL=5 KMP_ASAT_TRIGGER=5000 KMP_ATOMIC_MODE=2 KMP_BLOCKTIME=200 KMP_DETERMINISTIC_REDUCTION=false KMP_DUPLICATE_LIB_OK=false KMP_FORCE_REDUCTION: value is not defined KMP_FOREIGN_THREADS_THREADPRIVATE=true KMP_FORKJOIN_BARRIER="2,2" KMP_FORKJOIN_BARRIER_PATTERN="hyper,hyper" KMP_GTID_MODE=0 KMP_HANDLE_SIGNALS=false KMP_INIT_AT_FORK=true KMP_INIT_WAIT=2048 KMP_ITT_PREPARE_DELAY=0 KMP_LIBRARY=throughput KMP_LOCK_KIND=queuing KMP_MALLOC_POOL_INCR=1M KMP_MONITOR_STACKSIZE: value is not defined KMP_NEXT_WAIT=1024 KMP_NUM_LOCKS_IN_BLOCK=1 KMP_PLAIN_BARRIER="2,2" KMP_PLAIN_BARRIER_PATTERN="hyper,hyper" KMP_REDUCTION_BARRIER="1,1" KMP_REDUCTION_BARRIER_PATTERN="hyper,hyper" KMP_SCHEDULE="static,balanced;guided,iterative" KMP_SETTINGS=true KMP_SSP_MAX_THREADS=0 KMP_STACKOFFSET=0 KMP_STACKSIZE=4M KMP_STORAGE_MAP=false KMP_TASKING=2 KMP_TASK_STEALING_CONSTRAINT=1 KMP_USE_IRML=false KMP_VERSION=true KMP_WARNINGS=true OMP_DYNAMIC=false OMP_MAX_ACTIVE_LEVELS=2147483647 OMP_NESTED=false OMP_NUM_THREADS: value is not defined OMP_WAIT_POLICY=PASSIVE After program exit has begun, i get another line: Intel(R) OMP Intel(R) RML support: not using

Olga Malysheva (Intel) wrote:
Also you mentioned the crash occurs on MacOS. Does it mean the crash dos not occur on Linux/Windows or you just tried only Mac?

On Windows i don't get the crash. There's no Linux build of the application.

Olga Malysheva (Intel) wrote:
More information about application itself would be useful as well. Actually we need to know how the application uses OpenMP (constructs, API).

It would be really useful to get a reproducer.

Unfortunately, i won't have the time to create an isolated reproducer. I can guide you through setting up the software and making it crash, though.

Olga_M_Intel · ‎12-04-2012

Yes, please, give detailed instructions how to see the crash. Thank you.

SergeyKostrov · ‎12-04-2012

>>...Unfortunately, i won't have the time to create an isolated reproducer... Please take into account that your case could be given a lowest priority and it possible that it won't be resolved for a long time. How an Intel software engineer could reproduce it? What would you personally do in a similar case? Olga Malysheva (Intel) already asked for detailed instructions from you...

Jascha_Wetzel · ‎12-07-2012

Olga Malysheva (Intel) wrote:
Yes, please, give detailed instructions how to see the crash.
Thank you.

You should have received a message with files and detailed instructions.

JenniferJ · ‎05-01-2013

Update: this opm crashing issue is fixed in the 13.0 update 3.

Thanks for reporting it.

Jennifer