Re: Crash on exit of application

prestos · ‎10-02-2009

hi,

I am experiencing a deterministic crash on the exit of my application. The application has been built as a MSVC2005 solution and one of the projects (the most critical wrt speed) has been compiled with Intel compiler.

The crash takes place during destruction of static objects. It is like these objects are destructed twice (so, the crash takes place when trying to free for 2nd time any memory they have allocated).

Any ideas where to start with?

thanks

jimdempseyatthecove · ‎10-03-2009

There are generally several reasons for this behavior:

a) a second dtor - which is not likely
b) bug in your code that stomped on static object then when dtor called junk was in object
c) ctor of objects did not properly initialize object causing dtor to use junk
d) static objects positioned incorrectly such that sequence of ctor, then later sequence of dtor, are incompatable (i.e. objects have dependencies and dependencies are goofed up).

Try placing print statement in your ctors and dtors with if tests to print only if object (this)indicates a static object (you may have to add a member variable to indicate if object is static or dynamic). Print out object class name and address.

Jim Dempsey

prestos · ‎10-03-2009

Quoting - jimdempseyatthecove

There are generally several reasons for this behavior:

a) a second dtor - which is not likely
b) bug in your code that stomped on static object then when dtor called junk was in object
c) ctor of objects did not properly initialize object causing dtor to use junk
d) static objects positioned incorrectly such that sequence of ctor, then later sequence of dtor, are incompatable (i.e. objects have dependencies and dependencies are goofed up).

Try placing print statement in your ctors and dtors with if tests to print only if object (this)indicates a static object (you may have to add a member variable to indicate if object is static or dynamic). Print out object class name and address.

Jim Dempsey

I will exclude (b) & (c) since these objects are allocated/deallocated several times in the normal execution. It's only at exit that I get the crash.

So, what remains is... (a) & (d). If this gives any hint, note that the crash takes place not in a particular class but rather in all of the classes that envolve a static object.

The static object is created though as a parameter of another template one. The template class is a header file that is included multiple times. Here is a very simplified pseudo code:

//--------------- HEADER FILE START --------------

#ifndef __LINK_OF_CLASS__

#define__LINK_OF_CLASS__

template class LinkOfClass { public: T myLinkedObject; static T globalLinkedObject; };

template T LinkOfClass::globalLinkedObject;

#endif

//--------------- HEADER FILE END --------------

While using the VC compilers I never came to any problem, could that be the case here? That the compiler generates static objects that do not match against the constructors/destructors?

Now that I am thinking about it, I will try a simple remedy, by placing the definition of the static object in a source file and see what I get from that... Any other ideas are welcomed!

thanks

prestos · ‎10-04-2009

Quoting - prestos

Now that I am thinking about it, I will try a simple remedy, by placing the definition of the static object in a source file and see what I get from that... Any other ideas are welcomed!

Scratch that... of course, putting the definition in a source file makes it impossible to link.

prestos · ‎10-04-2009

After putting some logging messages on the constructor/destructor, I found out that the destructor IS called twice.

In fact, the constructor is ALSO called twice with the same address and at the end, the destructor is called twice. In a sense, the problem is before actually invoking the destructor twice and there is one question here:

Under what circumstances, the compiler may decide to create a "double" object (at the same heap address)?? Shouldn't there be any procedure to avoid this in the case of static objects of template classes?

Any help is appreciated.

prestos · ‎10-04-2009

PS. The crash takes place in debug mode as well, so it is irrelevant of any optimizations.

jimdempseyatthecove · ‎10-04-2009

You stated this was a static object, now you say same heap address (its dynamic), which is it?

If on heap, then termination with

ctor @ 0xnnnnn
dtor @ 0xnnnnn
ctor @ 0xnnnnn
dtor @ 0xnnnnn

Where 0xnnnnn is the same is ok, whereas

ctor @ 0xnnnnn
ctor @ 0xnnnnn
dtor @ 0xnnnnn
dtor @ 0xnnnnn

Is not ok.

If the object is in static memory (not on heap, not on stack), then you should not have more than one ctor nor more than one dtor to the same address.

From my experience, the usual culprit for static object ctor/dtor problems is where one static object references a different static object then uses that other static object (member variables or functions)in the dtor. This practice is fine provided that the sequence of dtors of static objects is controlled such that the depencies are not accessed after they have been dtor'd. The sequence of dtors for static objects is by established convention, the reverse of the order of how the ctors were issued.

When staticdependencies are within the same compilation unit, the order is relatively easy to maintain (order of ctor is top down, dtor is bottom up).

However, when static dependencies are in different compilation units, then link order comes in to play. You may crash depending on link order.

Note, I am not stating that your problem is related to ctor/dtor issues. I do not have sufficient information to make that assessment. If you do suspect ctor/dtor order issue then...

In the class (struct) of static objects insert diagnostic code, much like your prints (conditional compiled). The code performs sanity checks. Choose a signature bit pattern that is not likely to be generated at load time or during normal use. Something like

const int was_dtor_ed = 0xDEADBEEF;
const int was_ctor_ed = ~was_dtor_ed;

Thenconditionally add member variable to those class/structthat are statically constructed.

int ctor_dtor_state; // junk, or was_ctor_ed, or was_dtor_ed

In the ctor of those objects

_ASSERT(ctor_dtor_state != was_ctor_ed);
ctor_dtor_state = was_ctor_ed;

In the dtor of those objects

_ASSERT(ctor_dtor_state == was_ctor_ed);
ctor_dtor_state = was_dtor_ed;

In the member functions

_ASSERT(ctor_dtor_state == was_ctor_ed);

In the class referencing other object member variable

_ASSERT(OtherObject->ctor_dtor_state == was_ctor_ed);

Note, when

_ASSERT(ctor_dtor_state == was_ctor_ed);

fails and if (ctor_dtor_state == was_dtor_ed)
then you are referencing a dtor'd object (sequencing error of dtors)

fails and if (ctor_dtor_state != was_dtor_ed) i.e. is junk
then you are referencing an object that was never ctor'd

Jim Dempsey

prestos · ‎10-05-2009

Quoting - jimdempseyatthecove

You stated this was a static object, now you say same heap address (its dynamic), which is it?

If on heap, then termination with

ctor @ 0xnnnnn
dtor @ 0xnnnnn
ctor @ 0xnnnnn
dtor @ 0xnnnnn

Where 0xnnnnn is the same is ok, whereas

ctor @ 0xnnnnn
ctor @ 0xnnnnn
dtor @ 0xnnnnn
dtor @ 0xnnnnn

Is not ok.

Sorry, I was in a hurry and my post was not very clarifying.

The objects ARE static, but they allocate memory (on the heap). The address that I had printed is the static object address (and not the heap address of its allocated memory as perhaps incorrectly stated).

Printing the addresses from ctors/dtors gave me the problematic sequence of 2 ctors first then followed by 2 dtors, all with the same address.

prestos · ‎10-05-2009

Quoting - jimdempseyatthecove

From my experience, the usual culprit for static object ctor/dtor problems is where one static object references a different static object then uses that other static object (member variables or functions) in the dtor. This practice is fine provided that the sequence of dtors of static objects is controlled such that the depencies are not accessed after they have been dtor'd. The sequence of dtors for static objects is by established convention, the reverse of the order of how the ctors were issued.

When staticdependencies are within the same compilation unit, the order is relatively easy to maintain (order of ctor is top down, dtor is bottom up).

However, when static dependencies are in different compilation units, then link order comes in to play. You may crash depending on link order.

I understand. Since the workspace is BIG and there are quite many static objects, my guess is that there might be such a case. BUT, it happens virtually for ALL static objects. If I change the code for one such class, so that it doesn't allocate any memory (and thus avoiding the crash), then I get a crash with another static object of a different class! So, my belief is that we are not talking with a particular bad reference case; it must be something more generic.

About linking order coming into play, that's surely the case. There are 3 projects and the static objects are created by headers of a template class - included by source files in all 3 projects.

thank you for providing a debugging method and putting effort on the problem.

JenniferJ · ‎10-05-2009

Quoting - prestos

About linking order coming into play, that's surely the case. There are 3 projects and the static objects are created by headers of a template class - included by source files in all 3 projects.

Do you mean the same static objects are defined in all 3 projects? Could you provide more detail info about the 3 projects? (dll, static libs, mfc, etc.)

Jennifer

prestos · ‎10-06-2009

I was finally able to re-create the problem on a small test-case. There is definitely something here that could be called "compiler bug" or at least that's my understanding.

You can download the test archive from this location:

http://software.intel.com/file/22782 (the same as the attached file).

The archive contains a Visual C++ 8.0 solution that is a small console application. that can be easily compiled and present the crash (no external dependencies). I used latest Intel C++ compiler evaluation version 11.1.046. The projects are setup so that the library is compiled by Intel compiler and the main project by Visual C++.

You will see that you get a static object double construction (at the same address) and at exit a double destruction (at the same address) which results in a crash!

Making some further tests I found out that when compiling BOTH library and main project by Intel compiler, the bug didn't appear anymore! Only a single static object was being created (as opposed to the crash case where 2 static objects were being created at the same address).

My simple conclusion is; MIXING Intel & Visual C++ compilers for projects of the same solution is unsafe!? But what happens if we have to use an external library that hasn't been compiled by Intel compiler (from a 3rd party)? Perhaps there could be at least a warning about this situation? It all makes me feel quite unsafe...

thanks

JenniferJ · ‎10-06-2009

Quoting - prestos

I was finally able to re-create the problem on a small test-case. There is definitely something here that could be called "compiler bug" or at least that's my understanding.

You can download the test archive from this location:

http://software.intel.com/file/22782 (the same as the attached file).

Making some further tests I found out that when compiling BOTH library and main project by Intel compiler, the bug didn't appear anymore! Only a single static object was being created (as opposed to the crash case where 2 static objects were being created at the same address).

Thank you very much for the testcase. I'll file a bug report to the compiler.

In general it's ok to mix Intel C++ compiler compiled project/files with VC compiled project/files. This is a bug only. When there's a fix, I'll let you know.

Also thanks Jim for the help.

Jennifer

TimP · ‎10-06-2009

Quoting - Jennifer Jiang (Intel)

In general it's ok to mix Intel C++ compiler compiled project/files with VC compiled project/files.

Linking both the vcomp and one of the Intel OpenMP run-time libraries creates problems, with no explicit warning. It may be a latent bug, if none of the VC objects actually activates vcomp. It's necessary to exclude vcomp and vcompd and specify the appropriate libiomp5.

prestos · ‎10-06-2009

Quoting - Jennifer Jiang (Intel)

In general it's ok to mix Intel C++ compiler compiled project/files with VC compiled project/files. This is a bug only. When there's a fix, I'll let you know.

Thank you Jennifer (& Jim) for your help. I will appreciate very much any information/progress related to this issue. In general, I would like to use the Intel compiler only for the most critical project where optimizations are necessary.

Please note that the test case does not make use of any special libraries (no MKL, no OpenMP, etc., just pure Ansi C++). Also, the bug appears already in debug mode, so it is irrelevant of any optimizations. The solution consists of a Win32 console application linked against a static library, both of them created as default projects from Visual Studio (the only change I made is unchecking precompiled headers).

thanks

JenniferJ · ‎10-06-2009

Quoting - tim18

Linking both the vcomp and one of the Intel OpenMP run-time libraries creates problems, with no explicit warning. It may be a latent bug, if none of the VC objects actually activates vcomp. It's necessary to exclude vcomp and vcompd and specify the appropriate libiomp5.

Not sure how you linked the omp libs. did you follow the instructions here
http://software.intel.com/en-us/articles/how-to-use-intelr-compiler-openmp-compatibility-libraries-on-windows/ ?

do you have a testcase or maybesome description about the testcase?

Thanks,
Jennifer

TimP · ‎10-06-2009

Quoting - Jennifer Jiang (Intel)

Not sure how you linked the omp libs. did you follow the instructions here
http://software.intel.com/en-us/articles/how-to-use-intelr-compiler-openmp-compatibility-libraries-on-windows/ ?

I don't know that the original poster had a problem with omp libraries, I simply point that out as a case where a library conflict may occur when icl and cl objects are linked together, and it can't be dismissed so easily.
The reference you point to is in error: for either VS2005 or VS2008, /nodefaultlib:vcomp /nodefaultlib:vcompd is required so as to ensure that all OpenMP functions are linked from libiomp5.
The problem will occur automatically in a build with cl /openmp, linked with MKL thread, unless the /nodefaultlib are added explicitly, and spelled according to the actual VC .lib names. We have run into it even when there is no OpenMP in the MSVC code, only in ifort, MKL sequential is in use, and ifort was in use for linking.
Intel compilers normally supply the necessary /nodefaultlib automatically; one would think the omp lib problem would occur only when CL is used to drive the link (as might happen when a few ICL objects are substituted in a project), or link is invoked directly. It seems, however, the Intel supplied /nodefaultlib doesn't necessarily take effect in a long link command.

JenniferJ · ‎11-25-2014

I have a good news about this bug.

It is not an easy fix, but it is finally fixed in 15.0 compiler.

Thanks for reporting :)

Jennifer