How to get meaningful routine names in call stack, after runtime error?

[chris-sn]

We are having trouble identifying the calling sequence at the time of a crash. For an example, see the attached screen captures, one of which shows the FORTRAN runtime report, and the other the VS2008 stack after we take the break option. We can see only the dll, not the routine called.

I have checked that the FORTRAN project has Configuration>Fortran>Runtime> Traceback set to "yes".

The FORTRAN project is building a static library which is then imported into a C++ dynamic library, would this cause the trouble?

The DLL is running in its own thread as a calculation engine, but I don't have any reason to believe this is the problem?

FWIW this project was auto-converted from the old COMPAQ compiler.

Any suggestions gratefully received...

[mecej4]

Did you clean/rebuild the Fortran library and the C++ DLL after setting the Traceback:yes option?

[chris-sn]

Nice thought, but I just redid that to make sure and no difference.

Another observation is that if I step through the code in the debugger prior to the runtime error, the VS2008 call stack is happily showing me the right function names rightthe way up the stack from Fortran to C code.But then after the runtime error, they are gone.

It looks like the error is actually getting generated by the VC runtime, which I suppose is being called from the FORTRAN runtime, might that be a cause?

This used to work with the old COMPAQ compiler apparently...

[chris-sn]

As an experiment, I tried inserting a deliberate divide by zero into the FORTRAN code just prior to the (array bound check) runtime error discussed above. This behaves quite differently - the runtime error gives no information about the call stack, but on breaking into VS2008 I get the information I need. See attached images.

Anyone shed any light on to why the difference and how to get round it? Is it a question of whether the runtime error occurs in FORTRAN or C++ code?

[jimdempseyatthecove]

Can you open a seperate VS instance (empty solution), the Debug | Attach to Process and browse to and select your errant process. You may have to specify path to sources.

When you make the attachment, you will be attached to the application while it is displaying the Divide by 0 dialog. The stack window will usually show "junk" (hex addresses of unknown sources). Mouse over to the stack window and then double click on the next higher level (next lower line). Keep going up (down in stack window) until you start seeing the Fortran function names.

An alternate means is, without disturbing "crashed" application, start a second debug session of the application (same execuitable immage) with a break point at the beginning of the application (or step into). Once there, open a Dissassembly Window, type Ctrl-G (Goto) enter the hex address obtained from the divide by 0 and see if the source window moves to the source line/file of error. Note, if the divide by 0 occured in a library, you might not have source to the library.

Good luck hunting.

Jim Dempsey

[IanH]

The divide by zero would result in a Windows structured exception being thrown. Because your fortran code is in a DLL the Fortran runtime may not have installed an exception handler in the chain. The message you see is probably from an exception handler installed by runtime of the parent EXE process (or the debugger or the OS). To test call the DLL with the deliberate fault from a fortran parent exe.

In terms of your original problem - I can get tracebacks with a setup similar to that which you describe. Is it possible to confirm at the point of the crash that the stack in your original example is still in reasonable shape? Is the optimisation setting for "omit frame pointer" off? Can you post the relevant compile and link command lines?

[Wendy_Doerner__Intel]

I think it is implied by this thread, but traceback only works for the Fortran code in the stack (not C++).

------

Wendy

Attaching or including files in a post

[chris-sn]

Jim, just in case we are at cross purposes - I am running the program from VS under the debugger already, and in the case of the divide by zero error the VS call stack is just fine, not a problem. It is in the case of the first runtime errorI reported (caused by array bound checking), that is when I see just hex addresses.

So in the case of the array bound check error, I tried your first suggestion but when I use Debug>Attach, the executing program with the error is listed but greyed out and I cannot attach to it.

I seewhat you have inmindwith the Diassembly, yes if I could identify the source code line corresponding to the hex address in the error message that would be a good workround. Unfortunately when I try it, the address from the error message just shows the object code, no source code. Does the source file have to be open or something - there are many source files in the project...I will continue experimenting. Or maybe I can use a map file from the linker.

Thanks

[chris-sn]

IanH,

The VS stack is showing me sensible routine names in the divide by zero case, which I forced to happen just before the block of code in which the array bound error occurred. So I am reasonably sure the stack was ok.

Furthermore, having talked more tothe the developers ( I am kind of doing a dev system support role here), all of the cases where after a runtime error the call stack has been hex only (no routine names) have been cases of array bound checking errors. I have no evidence of the problem occurring in other situations.

So I am wondering if the problem is specific to runtime errors caused by array bound checking (or at least a small class of errors of which array bound checking is one), which may be why you haven't seenit?

In answer to your questions - Optimisation is disabled for all of FOTRAN language, C++ language, and project settings. Here are the command line settingsfor the FORTRAN and C++ wrapper projects:

FORTRAN COMPILER (as used to construct the static library)
/nologo /debug:full /Od /extend_source:132 /warn:declarations /Qsave /iface:cvf /module:"Debug/" /object:"Debug/" /traceback /check:bounds /libs:dll /threads /dbglibs /c

C++ COMPILER ( as used to construct the wrapper and DLL)

/Od /I "..\Core\uic" /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /D "_USRDLL" /D "TM_CALC_EXPORTS" /D "_VC80_UPGRADE=0x0600" /D "_WINDLL" /D "_MBCS" /Gm /EHsc /RTC1 /MDd /Fp".\tm_calc___Win32_Debug/tm_calc.pch" /Fo".\tm_calc___Win32_Debug/" /Fd".\tm_calc___Win32_Debug/" /FR".\Debug\" /W3 /nologo /c /ZI /TP /errorReport:prompt

DLL LINKER (for the wrapper. Looking at it I suspect there is some legacy stuff that could be cleared out!)

/OUT:".\Debug/tm_calc.dll" /INCREMENTAL /NOLOGO /LIBPATH:"C:\Program Files\VNI\imsl\fnl600\IA32\lib" /LIBPATH:"C:\Program Files\Intel\Compiler\11.1\060\lib\ia32" /LIBPATH:"C:\Program Files\Microsoft Visual Studio 9.0\VC\lib" /DLL /MANIFEST /MANIFESTFILE:".\Debug\tm_calc.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:".\tm_calc___Win32_Debug/tm_calc.pdb" /MAP:".\tm_calc___Win32_Debug/tm_calc.map" /DYNAMICBASE:NO /IMPLIB:".\tm_calc___Win32_Debug/tm_calc.lib" /MACHINE:X86 /ERRORREPORT:PROMPT odbc32.lib odbccp32.lib ..\Libraries_Debug\network.lib ..\Libraries_Debug\GTLStatic.lib ..\Libraries_Debug\database.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib ".\debug\tm_calc_lib.lib"

Thanks

[jimdempseyatthecove]

Can you reliably get the error to occur?

Does it occur at a particular call statement the Nth time (e.g. 1st call, always call number 1234,...)?

If so, try linking with larger stack size declaration.

Then see if error behavior changes (different routine, or different call number)

You may have a case where a fault occurs, but there isn't sufficient stack space to handle the fault.

Jim Dempsey

[William_H_Intel3]

You have incremental linking enabled when you create the dll. Incremental linking will disable automatic PC correlation. Try turning off incremental linking.

Hope that helps...Bill

[William_H_Intel3]

You have incremental linking enabled when you create the dll. Incremental linking will disable automatic PC correlation. Try turning off incremental linking.

Hope that helps...Bill

[chris-sn]

Bill, nice thought but sadly no effect.

Jim, I can get the runtime error to reproduce 100% of the time the same way. My problem isn't how to fixthe specific piece of ourcode so it doesn't generate that particular array bounds check error, that was done and checked into the mainline repository a while back. My problem is that every so often someone hits a new occurrence of the array bound check error, and has a lot more trouble than they should finding where the error is coming from. This really doesn't feel like a stack corruption or overflow problem, it feels much more like something to do with interplay between the C++ and FORTRAN compilers and runtimes. Referring back to Wendy's point about tracebacks it feels as if the array bound checking error is triggering something in the C++ code whilst the divide by zero is entirely within the FORTRAN. Does that make any sense?

*If* it only occurs with array bounds check errors, the good news is that it will become less and less an issue as the offending lecgacy code is identified and fixed.

Regards, Chris

[chris-sn]

Oops - this seems to have been at least partly my fault. Doing some tidying up I realised that not all of the files created by the builds were going to the same place, there was some legacy settings causing some of the files to go to a different folder. Strangely this didn't seem to stop anything running, but having fixed it to be consistent my array bounds checking case now generates an error report from the FORTRAN runtime that has the FORTRAN routines named, as attached.

I still have the problem that when I break, the VS2008 call stack won't take me back into the offending FORTRAN, but now users will know which routine to put the breakpoints in.

[jimdempseyatthecove]

From this last .jpg it appears that your C/C++ callers are passing in 0-based array indexes whereas your FORTRAN code is using (requiring)1-based array indexes. IOW bad callingarguments.

I suggest you add to your DLL routines that take in array subscripts, sanity checking code that asserts 1-based indexing (and/or bounds checking). Using the FORTRAN array bounds checking will introduce overhead within your (bug-free)FORTRAN subroutine at all array indexing whereas a simple light weight test at subroutine entry point may be all that is reqired to catch the caller'sprogramming error. (not to mention that your code will run much faster). This test can be added as a conditional compile (or macro with FPP enabled) and thus you can make two versions of your DLL. One without the asserts, and one with the asserts (and both in Release build without debug symbols).

Also, the C++ code can call a C++ stub that calls the FORTRAN DLL. This stub can use a "back window" to detect if error occured inside the FORTRAN DLL, and if so raise a C++ exception. "back window" being a global status variable known to your stubs and DLL. Such asusing GetLastError/SetLastError or your own variable which may be in Thread Local Storage should you have a multi-threaded DLL.

Jim Dempsey

[Martyn_C_Intel]

If you are using the Intel C/C++ compiler, you should set the option /traceback for the C/C++ builds also.
(If necessary, just add it to the command line).
This causes additional information to be preserved to help you get the stack trace from the Fortran part of your programin a mixed language C & Fortran application.I'm not aware ofan equivalent option for the Microsoft C/C++ compiler.
See the Intel C/C++ compiler user & reference guide.

[chris-sn]

Jim,as you say there aresome arrays passed from C++ to FORTRAN - however IIRC these are just i/o filenames. 99% of the array access within thefortran calculator are to arrays created by the fortran itself, including the out-of-bounds access cases that have occurred so far.