Multi-threaded console app crashes on exit with newer version of TBB

[Long_Nguyen]

Hello,

We use TBB only for 'new' and 'delete'. The app only loads tbbmalloc.dll and tbbmalloc_proxy.dll. It runs fine with version 2.2.2009.627. However, when we switch to version 2.2.2009.1101, our application crashes on exit, outside of our code. Specifically, there is an access violation exception in the 'doexit' function of the C++ runtime (crt0dat.c). Does anyone know what changed between these versions that causes the crash? Thank you very much for your help.

Long Nguyen

[Vladimir_P_1234567890]

Quoting - Long Nguyen

Hello,

Does anyone know what changed between these versions that causes the crash? Thank you very much for your help.

Long Nguyen

hi,

there were changes related to realloc() andfree(). and there might be changes related to operator delete().
Which version of visual studio do you use and are there static objects in the your code?

thanks,
Vladimir

[Long_Nguyen]

Quoting - Vladimir Polin (Intel)

hi,

there were changes related to realloc() andfree(). and there might be changes related to operator delete().
Which version of visual studio do you use and are there static objects in the your code?

thanks,
Vladimir

Hi Vladimir,

We use Visual Studio 2008 and we do have static objects in the code. We also use the ACE framework and singletons leveraged from that (ACE).

Thanks,
Long

[Vladimir_P_1234567890]

Quoting - Long Nguyen

Hi Vladimir,

We use Visual Studio 2008 and we do have static objects in the code. We also use the ACE framework and singletons leveraged from that (ACE).

Thanks,
Long

And a question about the package (2.2.2009.1101). Did you use commercial-alligned package TBB 2.2 U1 (which is 2.2.2009.1011 actually) or the latest development package from 11/01?

If you did not use the latest development package, could you try to use the app with it? There were more fixes to the malloc replacement and for doExit() function particulary. In case the problem is still there we would like to fix it in the nearest updates. So your help here is really appreciated


thanks,
Vladimir

[Long_Nguyen]

Quoting - Vladimir Polin (Intel)

And a question about the package (2.2.2009.1101). Did you use commercial-alligned package TBB 2.2 U1 (which is 2.2.2009.1011 actually) or the latest development package from 11/01?

If you did not use the latest development package, could you try to use the app with it? There were more fixes to the malloc replacement and for doExit() function particulary. In case the problem is still there we would like to fix it in the nearest updates. So your help here is really appreciated


thanks,
Vladimir

I downloaded the package 'tbb22_20091101oss' at http://www.threadingbuildingblocks.org/ver.php?fid=145 and compiled the code with VS2008. Same result, our app crashes on exit in the doExit() function.

[Vladimir_P_1234567890]

Quoting - Long Nguyen

I downloaded the package 'tbb22_20091101oss' at http://www.threadingbuildingblocks.org/ver.php?fid=145 and compiled the code with VS2008. Same result, our app crashes on exit in the doExit() function.

I still can't to reproduce this:(
As far as I understand you did not get "CheckOpcodes failed" assertion in tbbmalloc_proxy.dll in the start of application. it is enabled even for release mode for this particular development release

Could you provide the following info:
OS
architecture ia32 or intel64
release or debug config (I guess release according to first post).
the stack trace around doexit() (or line number in crt0dat.c where the crash is)

of course test case would be the best suggestion, but i hope that above info should be enough:)

thanks,
Vladimir

[Long_Nguyen]

Quoting - Vladimir Polin (Intel)

I still can't to reproduce this:(
As far as I understand you did not get "CheckOpcodes failed" assertion in tbbmalloc_proxy.dll in the start of application. it is enabled even for release mode for this particular development release

Could you provide the following info:
OS
architecture ia32 or intel64
release or debug config (I guess release according to first post).
the stack trace around doexit() (or line number in crt0dat.c where the crash is)

of course test case would be the best suggestion, but i hope that above info should be enough:)

thanks,
Vladimir

Hi Vladimir,

Architecture: ia32
Config: release

I did not get the assert you mentioned. I am trying to narrow down the crash scenario and will report back. Thanks for the help.

Long

[john_gil-gomez]

Hi Vladimir,

I tracked the problem down. The problem is due to a mismatch in the size of memory requested by calling code, the size reported by _msize() and the size of memory stored inside TBB.

The following sample demonstrates the problem;

// stop at 1st unspecified character
int countChar(char *p, char c, int size)
{
int count = 0;
for(count =0; count < size; count++)
if( p[count] != c)
break;

return count;
}


int main()
{
char *p = (char *)malloc(1);

// re-alloc to 9K
p = (char *)realloc(p, 9 * 1024);
int reportedSize = _msize(p);
// size reported is 16K

// set the entire block to #
memset( p, '#', reportedSize);

// count the #'s, 16K as expected
int c1 = countChar(p, '#', reportedSize);

// realloc to 16K + 1 to force new memory allocation
p = (char *)realloc(p, reportedSize + 1);

// count the #'s, should be 16K
// but it is 9K
int c2 = countChar(p, '#', _msize(p));
}

In the function reallocAligned() the memory size requested by the user from the previous realloc() is used as the copy size.

From tbb MemoryAllocator.cpp;

if (isLargeObject(ptr)) {
LargeMemoryBlock* lmb = ((LargeObjectHdr *)ptr - 1)->memoryBlock;
copySize = lmb->unalignedSize-((uintptr_t)ptr-(uintptr_t)lmb); // ** copySize here is 16K
if (size <= copySize && (0==alignment || isAligned(ptr, alignment))) {
lmb->objectSize = size;
return ptr;
} else {
copySize = lmb->objectSize; // **** this line is executed. size is 9K
result = alignment ? allocateAligned(size, alignment) : scalable_malloc(size);
}
} else {
....
....
if (result) {
memcpy(result, ptr, copySize scalable_free(ptr);
}


copySize ends up as 9K. The size reported by _msize() was 16K. So only 9K of our 16K is copied into the new block.

The crash comes from the initialization of MS CRT. The CRT initialization sets up a of table of destructors. As the table grows MS uses realloc() to resize and _msize() to determine the size of the block. When the table size hits the 9K/16K boundary (as shown in above example) the table is not copied completely so we end up with uninitialized values in the destructor table. On exit the MS CRT calls each destructor in the table and crashes when it hits an uninitialized entry.

The MS documentation does not specify the exact behavior of realloc() /_msize(). But this is the behavior I see from the MS CRT when I allocate 1K/2K/4K etc..

requestedSize = 256 reportedMsize = 256
requestedSize = 512 reportedMsize = 512
requestedSize = 1024 reportedMsize = 1024
requestedSize = 2048 reportedMsize = 2048
requestedSize = 4096 reportedMsize = 4096
requestedSize = 8192 reportedMsize = 8192
requestedSize = 16384 reportedMsize = 16384
requestedSize = 32768 reportedMsize = 32768
requestedSize = 65536 reportedMsize = 65536
requestedSize = 131072 reportedMsize = 131072

using TBB (and using a function I added to TBB to get the size stored in TBB)

requestedSize = 256 reportedMsize = 256 storedRequestedSize = 256
requestedSize = 512 reportedMsize = 512 storedRequestedSize = 512
requestedSize = 1024 reportedMsize = 1024 storedRequestedSize = 1024
requestedSize = 2048 reportedMsize = 2688 storedRequestedSize = 2688 DIFFERENT
requestedSize = 4096 reportedMsize = 5376 storedRequestedSize = 5376 DIFFERENT
requestedSize = 8192 reportedMsize = 16384 storedRequestedSize = 8192 DIFFERENT
requestedSize = 16384 reportedMsize = 16384 storedRequestedSize = 16384
requestedSize = 32768 reportedMsize = 40960 storedRequestedSize = 32768 DIFFERENT
requestedSize = 65536 reportedMsize = 73728 storedRequestedSize = 65536 DIFFERENT
requestedSize = 131072 reportedMsize = 139264 storedRequestedSize = 131072 DIFFERENT

Offhand, I'd think _msize() should return the size the calling code requested when the block was allocated/reallocated. However, _msize() returning the actual amount of memory would also seem workable. As long as the value returned by _msize() and the value used internally were the same.

Please let me know if you need further clarification.

John

[e4lam]

Hi John,

Just curious, are you running into this because you're using tbbmalloc_proxy? I'm running into some weird crashes using tbbmalloc_proxy, static constructors, and on Win7 machines myself.

[Alexey-Kukanov]

Thanks for the detailed analysis John.

Looks likewe misunderstood the semantics of _msize, and so created the described discrepancy between _msize and realloc. We will fix this, one or the other way.

[john_gil-gomez]

Yes, we are using tbbmalloc_proxy. We are running on XP, but that shouldn't matter.

In our case, the problem showed itself through the creation of numerous const STL strings. But, static constructors would also be a likely candidate for trouble.

If you post more details I'll have a look to see if may be able to determine if you are seeing the same problem.

John

[john_gil-gomez]

Thank you for acknowledging the issue.

I look forward to updated code.

Can you give a time estimate?

Thanks.

John

[Alexey-Kukanov]

The fix and regression test are in our source repository, so the next open-source development release will have it; a rough time estimate would be "within a month".The next commercial release likely will have the fix too. I will add a CHANGES line for the fix. Stay tuned.

[john_gil-gomez]

I have updated to "20100406 open-source release'.

Problem has been resolved.

Thanks.

[Alexey-Kukanov]

Thanks for letting us know; even though we haveregression tests,it's always great to know that a fix works for the customer who asked for it :)