dlopen/malloc failure, Fedora 11 Linux, debug version of TBB only

mayaknife · ‎12-26-2009

I recently started getting the following error message when I try to run my program after building it in debug mode:

malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
Abort

When I build my program in release mode it works fine.

The stack trace leading up to the failure is as follows:

#0 in raise () from /lib64/libc.so.6
#1 in abort () from /lib64/libc.so.6
#2 in _int_malloc () from /lib64/libc.so.6
#3 in malloc () from /lib64/libc.so.6
#4 in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#5 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#7 in _dl_open () from /lib64/ld-linux-x86-64.so.2
#8 in dlopen_doit () from /lib64/libdl.so.2
#9 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 in _dlerror_run () from /lib64/libdl.so.2
#11 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#12 in tbb::internal::dynamic_link(char const*, tbb::internal::dynamic_link_descriptor const*, unsigned long, unsigned long, void**) ()
from /localBuild/lib/libtbb_debug.so.2
#13 in tbb::internal::initialize_cache_aligned_allocator() ()
from /localBuild/lib/libtbb_debug.so.2
#14 in tbb::internal::DoOneTimeInitializations() ()
from /localBuild/lib/libtbb_debug.so.2
#15 in tbb::internal::DummyMalloc(unsigned long) ()
from /localBuild/lib/libtbb_debug.so.2
#16 in tbb::internal::NFS_Allocate(unsigned long, unsigned long, void*) () from /localBuild/lib/libtbb_debug.so.2
#17 in tbb::cache_aligned_allocator<:CONCURRENT_HASH_MAP> > >::segment>::allocate (this=0x7fff847b07bf, n=64, hint=0x0)
at /localBuild/TBB/2.1/include/tbb/cache_aligned_allocator.h:86
#18 in tbb::concurrent_hash_map > >::initialize (
this=0x2249ef8)
at /localBuild/TBB/2.1/include/tbb/concurrent_hash_map.h:709
#19 in tbb::concurrent_hash_map > >::concurrent_hash_map (
this=a=@0x7fff847b0816)
at /localBuild/TBB/2.1/include/tbb/concurrent_hash_map.h:450

...and eventually leads back to __do_global_ctors_aux() in one of my modules.

If I'm reading this correctly, one of my static constructors is creating a concurrent_hash_map. That causes tbb::internal::initialize_cache_aligned_allocator() to try to load a shared library, which fails. dlopen() tries to report the error but that requires a call to the tbb debug memory allocator which has not yet been set up, leading to the assertion.

That sound right?

What I'm trying to figure out is what shared library tbb::internal::initialize_cache_aligned_allocator() is failing to load. When I use 'ldd' on my program all of the shared objects resolve correctly, so it must be something which is hard-coded into tbb itself.

Note that in the listings above copies of the necessary intel libraries are kept in /localBuild/lib and the application's LD_LIBRARY_PATH is set to look for them there.

Thanx.

mayaknife · ‎12-27-2009

This is looking less and less like a problem with TBB.

The library which is failing to load is libtbbmalloc_debug.so.2. The lib itself is being successfully loaded but while _dl_map_object_deps() its resolving its dependencies a call to malloc() fails due to heap corruption.

It's possible that the corruption is occurring as a result of TBB's concurrent_hash_map code, but then I would have expected to see a lot more people reporting problems with it.

The static initializer which creates the concurrent_hash_map is in a third-party library. My current focus is on determining if the problem stems from differences between the version of TBB 2.1 that it was built against versus the version I have installed.

Vladimir_P_1234567890 · ‎12-28-2009

hi,

Offical support for Fedora 11 was added to TBB 2.2. Could you try it or even newer version? BTW is should be binary backward compatible with 2.1

The static initializer which creates the concurrent_hash_map is in a third-party library. My current focus is on determining if the problem stems from differences between the version of TBB 2.1 that it was built against versus the version I have installed.

If you set LD_LIBRARY_PATH to the right place the system tbb libraries should not be a problem

added later: did you rebuild tbbmalloc also? if libtbbmalloc.so.2 is taken from another location it might cause a problem

--Vladimir