Thread local storage in glibc conflicts with tbb memory proxy strategy

[Giggle_L_]

Hi all,

My system is Ubuntu 11.04 and the library is tbb 4.1.

I found a heisenbug (crash sometimes) occured when the application exited. After I narrowed down the issue, the root cause was the combination of thread local storage and TBB library. I had prepared an easy setup for this issue: https://github.com/DemonGiggle/TlsWithTBB

Since it was not easy to crash, I suggested to use valgrind to grasp the error.

I summaried the root cause here:

When a thread touched a TLS variable, it firstly call __tls_get_addr() to obtain the address for this specific thread, which in turn called mmap(). And when a thread was exiting, it deallocated its TLS data, which in turn called free(), hooked by tbb library. The function free() in TBB was not aware whether the input pointer was allocated by TBB, and it crashed when it tried to access what it expected data structure from customized malloc.

[Vladimir_P_1234567890]

Hello Giggle, The memory checker error is an expected behavior of tbbmalloc_proxy. this is runtime dispatcher that checks where the block was allocated by tbbmalloc or libc's malloc and redirects it to correct either scalable_free() or free(). The crash is unexpected behavior:) Do you have a stack after the crash? Or is this crash reproduced on the test case? Test case is straight forward and it is not a problem to run it in the loop until we get crash. Thanks --Vladimir

[Giggle_L_]

Hi Vladimir, The test case indeed has error. Could you help make sure if the tbb library is really linked, that is, ldd a.out. And the crash rate is really low, but it could be detected by valgrind, so I suggest to use valgrind to run the test. The callstack is #0 safer_dereference (object=0x3f0f010, original_free=0x7f53c4f14360 <*__GI___libc_free>) at ../../src/tbbmalloc/frontend.cpp:2144 #1 isSmallObject (object=0x3f0f010, original_free=0x7f53c4f14360 <*__GI___libc_free>) at ../../src/tbbmalloc/frontend.cpp:2173 #2 safer_scalable_free (object=0x3f0f010, original_free=0x7f53c4f14360 <*__GI___libc_free>) at ../../src/tbbmalloc/frontend.cpp:2582 #3 0x00007f53c5bbf159 in free (object=Unhandled dwarf expression opcode 0xf3 ) at ../../src/tbbmalloc/proxy.cpp:96 #4 0x00007f53c9dbdcb9 in *__GI__dl_deallocate_tls (tcb=0x7f53413b8700, dealloc_tcb=false) at dl-tls.c:481 #5 0x00007f53c648e6bd in __free_stacks (limit=41943040) at allocatestack.c:274 #6 0x00007f53c648e8da in queue_stack (pd=0x7f533bc9b700) at allocatestack.c:302 #7 __deallocate_stack (pd=0x7f533bc9b700) at allocatestack.c:740 #8 __free_tcb (pd=0x7f533bc9b700) at pthread_create.c:223 #9 0x00007f53c648ec1d in start_thread (arg=) at pthread_create.c:401 #10 0x00007f53c4f7e16d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #11 0x0000000000000000 in ?? () And in the last frame, the code is: 2138│ static inline BackRefIdx safer_dereference (const BackRefIdx *ptr) 2139│ { 2140│ BackRefIdx id; 2141│ #if _MSC_VER 2142│ __try { 2143│ #endif 2144├> id = *ptr; 2145│ #if _MSC_VER 2146│ } __except( GetExceptionCode() == EXCEPTION_ACCESS_VIOLATION? 2147│ EXCEPTION_EXECUTE_HANDLER : EXCEPTION_CONTINUE_SEARCH ) { 2148│ id = BackRefIdx(); 2149│ } 2150│ #endif 2151│ return id; 2152│ } where (gdb) x ptr 0x3f0c068: Cannot access memory at address 0x3f0c068 From the last frame (isSmallObject), the object is TLS variable address, which is not allocated by TBB malloc, so the pointer passed to the last frame is wrong.

[Vladimir_P_1234567890]

Very nice and clear report, thanks! We are thinking on a fix. --Vladimir