Debugging the crash

[Vladimir_P_2]

I was wondering if someone could suggest an approach to debugging / finding a problem.

Setup (note two versions of Fortran): 

• Ubuntu 14.04.3
• Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.5.223 Build 20150805
• Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.0.0.084 Build 20101006
  
• Piece of software, which is somewhat large

Usage:

1. Compile software using ifort 12.0.0.084 with optimization on (-O3 and/or O2)
2. Compile software using ifort 15.0.5.223 with optimization on (-O3 and/or O2)
3. Compile software using ifort 15.0.5.223 with debug options (-check all -warn all -warn notruncated_source -g -traceback -debug inline_debug_info)

Results:

1. and 3. work and produce the same results, 3. (debug) doesn't show anything significant (a few unused variables),  while 2. crashes with segfault for no apparent reason.

Note: I've updated to update 5 for 2015 (15.0.5.223) just today because I had this same problem on update 4. However, I know for a fact that I was able to compile / run the code using 15 update 4 just a couple of weeks ago. Between now and before there were some linux updates and nothing else significant that I can think of.

Was wondering if someone can suggest an approach to debugging this.

[Kevin_D_Intel]

For build 2, under the debugger, I'm guessing you are unable to get any sense of where the fault occurs?  Is there no call stack provided if you run build 2 under the debugger and use "backtrace"?

Have you tried build 2 with -g -traceback -O2 (explicitly)?   -g implies at least -O0 so you can override that by throwing -O2 so having the debug symbols and traceback might help get information on the fault with build 2.

[Vladimir_P_2]

Well, the output provided by traceback tells me where it occurs, but if, for exampe I add a write statement in front of that line - traceback points to a different location (often a comment line, etc). Similar things happen when there are memory issues when assigning array elements outside of bounds, etc. But these are usually found with -check bounds.

• -static -g -traceback -O0  - works
• -static -g -traceback -O1  - works
• -static -g -traceback -O2  - does not
• -static -g -traceback -O2 -check bounds - works (does -check bounds disable some optimizations?)

[Vladimir_P_2]

And another note, the -O2 that doesn't work on ubuntu, runs just fine on another machine with RHEL.

[Kevin_D_Intel]

Check bounds introduces additional code, https://software.intel.com/en-us/node/579521 but does not disable optimizations that I'm aware of.

Do the systems have an identical CPU?

Have you checked the default shell stack limit (ulimit -s) between the Ubuntu vs. RHEL systems?

I didn't offer this initially since it felt unrelated but maybe there's a chance it is. You can refer to Determining Root Cause of Segmentation Faults SIGSEGV or SIGBUS errors for some additional things to look into (if you haven't already).