Debugging an application that works with -O0 but not -O1

[jfg]

Hi all,

I have a baffling issue in a large application I have been working on. I have a previous version of this application that worked fine at all levels of optimization, but some recent changes I made seem to have broken the program. The program crashes out with a segmentation fault if I try to compile it -O1 or above, but does notcrash at -O0 (which is automatically turned on with -g). What's worse is that print statements inserted into the code seem to affect whether the code crashes or not...I can successfully run at -O3 if I load the program up with diagnostic print statements. My hunch is that this is some sort of out-of-bounds error which is not being caught, and that it doesn't manifest when the for loops are not unrolled (hence it works at -O0).

I have tried -check all, -ftrapuv, -nozero, -warn all at -O0 through -O3, but nothing is being caught when the program crashes. Are there any other compiler flags for ifort that could help me troubleshoot this issue?

Thanks for reading.

[Steven_L_Intel1]

First of all, forget -ftrapuv. It does nothing useful. Which compiler version are you using?

[jfg]

ifort (IFORT) 11.1 20091130

I have actually just tried compiling the same program in gfortran, and it has the same behavior...crashes at anything above -O0.

Some additional details: I am using openmpi, so ifort is being run through the mpif90 wrapper. Also, the crash is occuring within a functor. The segmentation fault error at -O1+ looks like the following:

*** Process received signal ***

Signal: Segmentation fault (11)

Signal code: Address not mapped (1)

Failing at address: 0x18

[ 0] /lib64/libpthread.so.0 [0x3979c0e4c0]

[ 1] application_name(eos_helm_+0xe) [0x542a0e]

[ 2] application_name(eos_newt_functions_mp_newt_eos_+0xfb) [0x5461ab]

[ 3] application_name(fminln_mp_fmin_+0x10a) [0x54a8ca]

[ 4] application_name(annewt_+0x13a) [0x51cafa]

[ 5] application_name(newt_wrappers_mp_run_annewt_+0xd6) [0x6c0036]

[ 6] application_name(eos_+0x7ee) [0x445f5e]

[ 7] application_name(wd_intout_+0x2228) [0x6d7828]

[ 8] application_name(simulation_init_+0x50e) [0x48369e]

[ 9] application_name(driver_initflash_+0x5c5) [0x444205]

[10] application_name(MAIN__+0x3b) [0x44d71b]

[11] application_name(main+0x3c) [0x43ee4c]

[12] /lib64/libc.so.6(__libc_start_main+0xf4) [0x397941d974]

[13] application_name [0x43ed59]

[Steven_L_Intel1]

Well, that it also fails with gfortran tells you something. I believe you can use -G -O1 to get debug symbols but some optimization. Looks as if it's failing in a pthread call, suggesting to me that you've corrupted the stack or some other data area.

One thing you can do is selectively compile sources with -O1 and see which one (hopefully) is required to be -O1 to get the error.

[Ron_Green]

You could throw all the diagnostic options at it as well:

-g -O1 -traceback -check all -gen-interfaces -warn interfaces -fp-stack-check

and run the code.



ron