Optimization-related crash

[gib]

I'm using ifort 10.1 on Windows XP

The following two functions are in a module, as is divide_dist(:), a
derived type allocatable array. The function zig_rnor() returns a
Gaussian random variate - this is well tested.

When the program executes dividetime() is called many times
successfully, then at a repeatable point it fails with:
forrtl: error (65): floating invalid
The line number indicated is the "end function" line of function
dividetime. The argument gen is as expected (note ictype is unused)
and the values in divide_dist() are also correct. The check on member
%class is to ensure that the "select case" is getting the right value.

After pulling out quite a lot of hair over a day, I found that if I
comment out the three lines marked with !!!!! the code executes
correctly. (Note that the code originally used select because there
are other possible values for %class. I culled out the other cases in
the process of tracking this bug down. The code shown is exactly what
I am running with.)

At this point I finally (belatedly?) turned my attention to the
optimization level. I have been using /O3. If I drop back to /O1 the
program executes without error.

Apparently the optimization is messing up the logic of my code. The
reason for my post is that I'm wondering what it is about this simple
function that allows the compiler's optimization to break it.
Conversely, how should I modify the code to make it robust to
optimization?

This is the first time I've had an error (that I knew about) caused by
the optimizer.

On comp.lang.fortran Steve Lionel suggested that this could be a result of a bug in the floating point stack. He suggested compiling with /Qfp-stack-check but this didn't show anything. He also suggested trying /QxW but the program still crashes.

The debug version seems to run OK. Putting write statements in can stop the program from crashing.

type dist_type
integer :: class
real :: p1, p2, p3
end type

type(dist_type), allocatable :: divide_dist(:)

!----------------------------------------------------------------
real function dividetime(gen,ictype)
integer :: gen,ictype

write(*,*) 'divide_dist: ',gen,divide_dist(gen)%class
if (divide_dist(gen)%class /= LOGNORMAL_DIST) then
write(*,*) 'Bad distn class: ',divide_dist(gen)%class
stop
endif

dividetime = 1000
select case (divide_dist(gen)%class) !!!!!
case (LOGNORMAL_DIST) !!!!!
dividetime = rv_lognormal(divide_dist(gen)%p1,divide_dist(gen)%p2)
end select !!!!!
end function

!----------------------------------------------------------------
real function rv_lognormal(p1,p2)
real :: p1,p2
real :: r

r = zig_rnor() ! returns N(0,1) r.v.
rv_lognormal = exp(p1+r*p2)
end function

[gib]

I should add that I am well aware that Heisenbugs like this are usually the result of a programming error, and this one probably is too. I have turned on all run-time checking, so I am reasonably confident that it is not an array bounds violation error. It looks as if either the stack is getting messed up, or somehow the code space is being corrupted.

[abhimodak]

Is it possible for you to give the complete code so that the problem can be reproduced?

Abhi

[gib]

Quoting - abhimodak

Is it possible for you to give the complete code so that the problem can be reproduced?

Abhi

Unfortunately this is quite a big program, 8 source files each 20k-70k. The error occurs (for the current case) after about 4 minutes of execution. It wouldn't be impossible to provide the source code, but it probably isn't desirable.

[TimP]

If your purpose is to test for compiler bugs, your example may make sense. However, if you are trying to make a maintainable program, and not create pitfalls for a compiler when it tries to combine conditionals in order to optimize, it's better to write what is meant, and not place dead code indiscriminately. I see 2 more reasonable choices:

[cpp]....

if (divide_dist(gen)%class /= LOGNORMAL_DIST) then
   write(*,*) 'Bad distn class: ',divide_dist(gen)%class
   stop
   !   dividetime = 1000
else
   dividetime = rv_lognormal(divide_dist(gen)%p1,divide_dist(gen)%p2)
endif

end function


or (if you must use select case, when a single if..then..endif will do)

....

select case (divide_dist(gen)%class) 
  case (LOGNORMAL_DIST)
    dividetime = rv_lognormal(divide_dist(gen)%p1,divide_dist(gen)%p2)
  default
    write(*,*) 'Bad distn class: ',divide_dist(gen)%class
    stop
    !   dividetime = 1000
end select

end function


[/cpp]

I'm not totally surprised that a compiler comes to grief trying to combine and simplify code when you make it unnecessarily complicated. Commenting out the dividetime = 1000 is optional, but someone trying to maintain what you have written may question whether the dead code is intentional.

[Steven_L_Intel1]

I'm sorry to hear that my suggestion didn't help. Clearly you are running into a different problem.

The way I usually diagnose this sort of thing is to identify, in the debugger, which specific instruction is raising the "invalid" error and then try to find out where the data came from. It's a tedious effort to backtrack through the code. If it's a memory location, I look at that in a Memory window and see where/when it changes value. I may also try removing/disabling sections of the code to help narrow my focus.

[gib]

Your comment is not helpful. I've been making all sorts of temporary changes trying to track down this bug, and as a result there are many lines of commented code.

[Steven_L_Intel1]

I'm sorry you did not find it helpful - I was explaining how I would go about tracking it down and I have done this many times in the past with similar issues. Perhaps you have another approach which works better for you. I generally try to identify the smallest possible program that still shows the problem, which makes it much easier to work with.

If you'd like us to take a look at it, please submit an issue to Intel Premier Support and attach a ZIP of the project along with everything needed to build the application and instructions for reproducing the problem.

[gib]

The "debug" version does not fail.

I have disabled Inline Function Expansion, and now the error (65) occurs in function rv_lognormal().

I have modified the code slightly:

real function rv_lognormal(p1,p2)

real :: p1, p2

real :: r, z

r = my_rnor() ! an alternative r.v. generator (makes no difference)

z = p1 + r*p2

rv_lognormal = exp(z)

end function

The floating invalid error occurs at the line with exp(z).

I can identify the occasion when the error occurs through a global variable istep, the crash happens when istep = 6347.

If I place the statement

if (istep == 6347) z = 1

before the line

rv_lognormal = exp(z)

the error still occurs at the exp(z) line. However if I insert a write statement to see the values of p1, p2, r, z the error goes away.

Compiling with /O1 does not give an error.

[gib]

Quoting - Steve Lionel (Intel)

I'm sorry you did not find it helpful - I was explaining how I would go about tracking it down and I have done this many times in the past with similar issues. Perhaps you have another approach which works better for you. I generally try to identify the smallest possible program that still shows the problem, which makes it much easier to work with.

If you'd like us to take a look at it, please submit an issue to Intel Premier Support and attach a ZIP of the project along with everything needed to build the application and instructions for reproducing the problem.

Steve, I wasn't replying to your post when I said that (look at the post numbers). Your comments are always helpful.

My approach to bug-tracking is similar to yours. Unfortunately this is a rather complex simulation program, and reducing it to a smaller program presents a challenge. I might take you up on the suggestion that I sent the code to Premier Support - that would be a last resort.

Thanks.

[gib]

The program seems OK with these optimization settings:

Maximize Speed

Global Optimizations (/Og)

Inline Function Expansion Any Suitable

Favor Fast Code

and the command line options /QxT /Qfp-stack-check

Changing to Maximize Speed plus Higher Level Optimizations (/O3) causes it to crash.

I have the option of foregoing the higher level optimizations, and accepting (presumably) a slight speed decrease.

[Steven_L_Intel1]

Does it fail in a Debug configuration when you set the optimization level up? You can still debug optimized code, but it's harder.

The source lines may not be relevant - that is, the actual error probably happened some time previous.

[abhimodak]

Hi Gib

I also have my share of problems with optimize, optimize+debug etc. However, I found that two cases that were critical to me ran fine with compiler version 11.0.039 Beta when using exactly same set of options that fail with 10.1. If possible, you may want to give it a shot. This surely does not answer why but as Steve Lionel has pointed out in one of my previous posts, the compiler developer may have made changes that have taken care of previous flaws in addition to issues known to them.

It is too bad that we won't know how and why and will be worried of what else may be be wrong but I just thought I would share my opinion with you.

Sincerely

Abhi

[gib]

Quoting - abhimodak

Hi Gib

I also have my share of problems with optimize, optimize+debug etc. However, I found that two cases that were critical to me ran fine with compiler version 11.0.039 Beta when using exactly same set of options that fail with 10.1. If possible, you may want to give it a shot. This surely does not answer why but as Steve Lionel has pointed out in one of my previous posts, the compiler developer may have made changes that have taken care of previous flaws in addition to issues known to them.

It is too bad that we won't know how and why and will be worried of what else may be be wrong but I just thought I would share my opinion with you.

Sincerely

Abhi

Hi Abhi,

It sounds as if it would be worth testing the new version. Is installation easy? Would it replace 10.1, or be installed alongside it?

Cheers

Gib

[gib]

Quoting - Steve Lionel (Intel)

Does it fail in a Debug configuration when you set the optimization level up? You can still debug optimized code, but it's harder.

The source lines may not be relevant - that is, the actual error probably happened some time previous.

As I recall the Debug config with full optimization ran past that failure point.

But I have now found that my previous post about Maximize Speed being OK was premature. It did run past the previous failure point, so I assumed all was OK. Now I've been testing with a case closer to my real problem (previously I was working on a reduced-size problem to speed it up) and find that again I'm getting error 65 at the same line (exp(z)). This compile was with no Global Optimizations, and with Inline Function Expansion disabled.

I will try to get a smaller problem to fail with these optimization settings, and also do more exhaustive testing to try to confirm that failure occurs only when optimizing. Of course it's impossible to prove that the code is OK by this method - one can only prove that it's not OK. My program is an agent-based simulation, tracking hundreds of thousands of agents, with heavy use of random variables. Almost any change to the input variables leads to a different execution path.

Thanks for your help.

Gib

[Steven_L_Intel1]

11.0 will install alongside 10.1. You can select the compiler you want to use in VS through Tools > Options > Intel Fortran > Compiler.

[gib]

Quoting - Steve Lionel (Intel)

11.0 will install alongside 10.1. You can select the compiler you want to use in VS through Tools > Options > Intel Fortran > Compiler.

It is too soon to say with confidence, but it appears that the compiler option that is causing my problem is /QxT. I have compiled with Maximum Speed plus Higher Level Optimizations, and Global Optimizations, and removed /QxT. The small test case runs OK, and fails if I put back /QxT. I have been running for a couple of hours with a much bigger problem, and so far all seems fine.

I will try version 11.0 beta.

[TimP]

Quoting - gib

It is too soon to say with confidence, but it appears that the compiler option that is causing my problem is /QxT. I have compiled with Maximum Speed plus Higher Level Optimizations, and Global Optimizations, and removed /QxT. The small test case runs OK, and fails if I put back /QxT. I have been running for a couple of hours with a much bigger problem, and so far all seems fine.

I will try version 11.0 beta.

I have seen people set /QxT, thinking that this would optimize for Core CPU, then run into intermittent failures on earlier CPUs. The CPUID check doesn't work in a useful way in this case. With ifort 10.1 and 11.0, there are so few differences between /QxP and /QxT that it is possible to run for quite a while before noticing a problem with a bad combination. Some customers did ask for /QxP to generate code similar to /QxT, except for the restriction on instruction set, and the compiler team appeared to have acted on that request.

When you say that you compared /QxT with some other choice, what was that other choice? Was it an option which evaluates expressions and math functions in double precision?

I know that you said suggestions weren't useful to you, yet you continue to post. If you depend unnecessarily on multiple floating point conditional expressions to always give consistent results, you are leaving open a possibility for trouble.

[gib]

Quoting - Steve Lionel (Intel)

11.0 will install alongside 10.1. You can select the compiler you want to use in VS through Tools > Options > Intel Fortran > Compiler.

BTW, I didn't mention that my machine is an Intel quad core, which is why I thought that /QxT might give some speedup.

[TimP]

Quoting - gib

BTW, I didn't mention that my machine is an Intel quad core, which is why I thought that /QxT might give some speedup.

In that case, /QxT is apprpropriate. If you are getting better results with promotion of expressions to double, you should consider whether making it explicit, or options /assume:protect_parens /Qprec-div /Qprec-sqrt may be useful.