Memory issue - arrays being overwritten

[cully]

We have discovered a problem with running a large Fortran program on a 32bit Linux machine. All the evidence seems to point to a memory issue. What is concerning is that the job appears to run successfully as no error messages are given nor does the job crash with a lack of sufficient memory message.

Investigations have shown that an array is being overwritten by a section of the Fortran code that should not alter the components of the array. The values of the array M%SoilMoisture(:,:,:) are being altered by the following line of code

! Rho=P/(GasConstant*T).
M%Rho(:,:,:,M%New)=M%P(:,:,:,M%New)/&
(GasConstant*M%T(:,:,:,M%New))

which it should never do. M%New is either 1 or 2 for the two time levels stored. Rho, P, T and SoilMoisture are all allocated pointers, with the allocation done just once in a single block of code,

Allocate(NWPMet%P(HGrid%nX,HGrid%nY,ZGrid%nZ,2))
Allocate(NWPMet%Rho(HGrid%nX,HGrid%nY,ZGrid%nZ,2))
Allocate(NWPMet%T(HGrid%nX,HGrid%nY,ZGrid%nZ,2))
Allocate(NWPMet%SoilMoisture(HGrid%nX,HGrid%nY,2))

whereon this occasion HGrid%nX=640, HGrid%nY=481, ZGrid%nZ=53

The size of the job is teetering around 2Gb so it is arguably on the limit of what could be run on a 32bit desktop machine.

We have tried both version 9.1 and 10.0 of the Intel compiler and the problem occurs using both. We have tried a number of compiler options including array bounds checking, namely ifort -Wp,-macro=no -C -w -extend_source -nbs -Vaxlib -auto and ifort -Wp,-macro=no -O3 -w -extend_source -nbs -Vaxlib -auto. We have also tried no optimisation (-O0 option) which does not work either.

We can been able to solve this problem in a number of ways
(a) reducing the size of the Fortran program so it requires less memory
(b) running on a 64 bit machine (allowing access to more memory)
(c) settling ulimit to an appropriate value (2000000 allows the job to run successfully whereas unlimited does not)
(d) Replacing -auto in the compiler line with -heap-arrays
(e) Replacing the above bit of code by

!Rho=P/(GasConstant*T).
Dok=1,ZGrid%nZ
M%Rho(:,:,k,M%New)=M%P(:,:,k,M%New)/ &
(GasConstant*M%T(:,:,k,M%New))
EndDo

solves the problem. However, bizarrely replacing it with the following code does not

!Rho=P/(GasConstant*T).
M%Rho(:,:,1:ZGrid%nZ,M%New)=M%P(:,:,1:ZGrid%nZ,M%New)/ &
(GasConstant*M%T(:,:,1:ZGrid%nZ,M%New))


(If we use -auto and -heap-arrays together with the following compiler options: ifort -Wp,-macro=no -C -traceback -w -extend_source -nbs -Vaxlib -auto -heap-arrays then the code crashes with the error "forrtl: severe (408): fort: (7): Attempt to use pointer ANCILLARYMETSOILMOISTURE when it is not associated with a target". However adding the option "-g" or using the compiler options: ifort -Wp,-macro=no -O3 -w -extend_source -nbs -Vaxlib -auto -heap-arrays is okay. We think this error is not really connected and we are not really sure whether using both -auto and -heap-arrays is sensible but mention it just in case it is of relevance. We think this failure is due to passing an unassociated pointer into a subroutine which receives it as an array. When it is not associated we don't use it in the subroutine. We think this is okay to do(?) but we think it is causing the error for these compiler options.)

We would like to be confident that this overwriting of arrays can never occur behind the scenes with us being unaware. We would be interested in any advice as to what one should do to avoid this situation. Are there any compiler options which would be of use. Should we always use -heap-arrays to be confident?

Any advice greatly appreciated.

[Steven_L_Intel1]

Have you tried a current, supported compiler? Try adding "-gen-interface -warn interface" and see if you get additiional errors.

[cully]

Quoting - Steve Lionel (Intel)

Have you tried a current, supported compiler? Try adding "-gen-interface -warn interface" and see if you get additiional errors.

Many thanks for your suggestions and for taking the time to help. We've tried the additional compiler options you suggest and this does not produce any additional errors (and indeed the problem still exists in that the array is overwritten without any warning). At the moment we do not have access to a more up-to-date version of the compiler but we are trying to see if it is possible for us to pursue this option. In the meantime, if there are any more suggestions we'd be very happy to hear.

[Steven_L_Intel1]

Unless you can provide a short (if possible) but complete test case, further speculation would not be fruitful.