Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.
28381 Discussions

"forrtl: severe (151): allocatable array is already allocated" Only Release

hentall_maccuish__ja
New Contributor II
7,082 Views

Hello,

I am getting an error "forrtl: severe (151): allocatable array is already allocated" only in release configuration. This is not called within a loop and  to try and get around this strange issue I am dellocating first but traceback is still throwing an error on the second of these two lines (but only in release not debug):

            if (allocated(xreg2)) deallocate(xreg2)
            allocate(xreg2(regPeriods, dimAboveMed, regPeriods-1+controls))

I'm a bit at a loss to try to trouble shoot this any more. If I deallocate without testing for allocatation first it gives an access violation on the first line and if I comment out the first line it gives an access violation later when I try and use xreg2 (both of which seem like correct errors because I am not allocating xreg anywhere else).

Thanks,   

0 Kudos
1 Solution
Arjen_Markus
Honored Contributor I
6,969 Views

The observation about commenting out these two lines strongly suggests there is a memory problem. Have you tried building the program with various run-time checking options on, such as array bounds checking?

View solution in original post

27 Replies
hentall_maccuish__ja
New Contributor II
5,504 Views

Could this be masking a memory issue? What triggers this issue is increasing controls from 3 to 4 so it seems this might be a misleading error in relation to a memory issue. But not sure why it would only happen in release  configuration. The other dimension variables have values regPeriods =24, dimAboveMed = 96940 and xreg is an arraoy of double precision reals. 

Thanks,

0 Kudos
Arjen_Markus
Honored Contributor I
5,495 Views

It is quite difficult to give adequate advice about this. Have you got a small reproducer for the problem or is the complete program available? 

You might want to try with the Intel Inspector tool from Intel oneAPI.

 

Alternatively, could it be that you first assign a value to xreg2 and then later do the explicit allocation? Though that error should have been solved by the deallocate statement.

 

jimdempseyatthecove
Honored Contributor III
5,477 Views

Is this a multi-threaded program .AND. xreg2 is shared .AND. this section of code is within a parallel region?

 

Jim Dempsey

 

hentall_maccuish__ja
New Contributor II
5,464 Views

Hi Jim,

It is multi-threaded (using omp) but this is not within a shared region which I thinks makes the question about whether shared void. I have also set the number of cores to run the paralllel code on to 1, so it wasn't really parallel and there could be no data race, and I got the same error.

Hard to create a minimal replicable example as I haven't figured out the conditions that it happens under but there is no prior assigment of xreg2. Here is the full subroutine it is contained in. 

My hypothesis about it masking a memory error seems incorrect as I set dimAboveMed to 1947 and the reduction in that dimension didn't compensate for the increase in control. The strange thing is that if I comment out lines 46 and 47 xreg2 sucessfully allocates.

    subroutine DIDreg(params,grids, modelObjects)
    implicit none

    !inputs
    type (structparamstype), intent(in) :: params

    !changing
    type(gridsType), intent(inout) :: grids
    type (modelObjectsType), intent(inout) :: modelObjects(numPointsType,Tperiods)

    !local
    real (kind=rk):: yreg(regPeriods, 3*numsims)
    real (kind=rk) :: xreg(regPeriods, 3*numsims, regPeriods-1+controls), assetstore( 3*numsims)
    logical :: mask(regPeriods,3*numsims) !, ols
    real (kind=rk) :: beta(regPeriods+2, 1)
    real (kind=rk), allocatable :: yreg2(:, :)
    real (kind=rk), allocatable :: xreg2(:, :, :)
    logical, allocatable :: mask2(:,:)
    logical:: recal = .true.
    real (kind=rk) :: ypath(Tperiods, numSims) !income
    real (kind=rk) :: cpath(Tperiods, numSims)  !consumption
    integer :: lpath(Tperiods, numSims) !labour supply
    integer :: apath(Tperiods + 1,numSims) !this is the path at the start of each period, so we include the 'start' of death
    integer :: spa, i,j, dimAboveMed, counter, total
    real (kind=rk) :: AIME(Tperiods + 1,numSims), medianA

    xreg = 0.0
    do SPA =1,3
        if (rank == 0) write (*,*) ' SPA ', spa
        call setSPA(SPA)
        if (modelChoice==1 .OR. SPA==1) then
            if (recal) call solveValueFunction( params, grids, modelObjects, .TRUE., .TRUE. )
        end if
        if (rank == 0) then
            call simWithUncer(params, grids, grids%Simy, cpath, apath, lpath, ypath, AIME, SPA, modelObjects, .false. )
            do i = firstReg, lastReg
                !y variable is labour supply
                yreg(i-firstReg+1,1+(SPA-1)*numsims:SPA*numsims) = lpath(i,:)
                !Indicator below spa
                xreg(i-firstReg+1,1+(SPA-1)*numsims:SPA*numsims,1) = abs(i<Tretire+SPA-1)

                do j = 1,numsims
                    !Wealth
                    !xreg(i-firstReg+1,1+(SPA-1)*numsims:SPA*numsims,2) = grids%Agrid(1,1, apath(i,:) )
                    xreg(i-firstReg+1,j+(SPA-1)*numsims,2) = grids%AgridFine(1,1, apath(i,j) )
                    xreg(i-firstReg+1,j+(SPA-1)*numsims,3) = grids%AgridFine(1,1, apath(i,j) )**2
                    xreg(i-firstReg+1,j+(SPA-1)*numsims,4) = grids%AgridFine(1,1, apath(i,j) )**3
                    !!assetstore(1+(SPA-1)*numsims:SPA*numsims) = grids%AgridFine(1,1, apath(i,:) )
                    if (i==Tretire-1+SPA) then
                        assetstore(j+(SPA-1)*numsims) = grids%AgridFine(1,1, apath(i,j) )
                    end if
                end do

                !Constant
                xreg(i-firstReg+1,1+(SPA-1)*numsims:SPA*numsims,controls) = 1.0
                !Age dummy with first period excluded category
                if (i>firstReg) xreg(i-firstReg+1,1+(SPA-1)*numsims:SPA*numsims,i-firstReg+controls) = 1.0
            end do
        end if
        if (modelChoice==1 ) then
            do i=1,Tperiods
                do j=1,4
                    if (allocated(modelObjects(j,i)%EV) ) deallocate(modelObjects(j,i)%EV )
                    if (allocated(modelObjects(j,i)%policy) ) deallocate(modelObjects(j,i)%policy)
                    if (allocated(modelObjects(j,i)%V) ) deallocate(modelObjects(j,i)%V)
                end do
            end do
        end if
    end do

    if (rank == 0) then
        mask = .true.
        call doReg(yreg, xreg, 3*numsims, regPeriods, regPeriods-1+controls, mask, .false., beta)
        write (*,*) 'Treatment effect', beta(1,1)

        medianA = 34869.00 !3.0*34869.00 !7.0*34869.00 !34869.00 ! 24846.00
        write (*,*) "median assets", medianA
        dimAboveMed = count(assetstore>medianA)
        Total = count(assetstore > 0.0) !count(xreg(3,:,2)>0.0)
        write (*,*) dimAboveMed, 'of', total, 'above med'
        if (dimAboveMed > 10) then
            counter = 0              
            allocate(yreg2(regPeriods, dimAboveMed))
            if (allocated(xreg2)) deallocate(xreg2)
            allocate(xreg2(regPeriods, dimAboveMed, regPeriods-1+controls))
             write (*,*) "aloocated xreg2!"
            if (allocated(mask2)) deallocate(mask2)
            allocate(mask2(regPeriods,dimAboveMed))
            do j = 1,3*numsims
                if (assetstore(j) > medianA ) then
                    counter = counter + 1
                    yreg2(:, counter) = yreg(:, j)
                    xreg2(:, counter,  =  xreg(:, j, :)
                    mask2(:,counter) = mask(:,j)
                end if

            end do
            write (*,*) "Mean wealth of those above median: ", sum(xreg2(3, :, 2))/dimAboveMed

            call doReg(yreg2, xreg2, dimAboveMed, regPeriods, regPeriods-1+controls, mask2, .false., beta)
            write (*,*) 'Treatment effect above median', beta(1,1)
            deallocate(yreg2, xreg2, mask2)
        end if
    end if



    end subroutine

 

 

 

 

 

0 Kudos
Arjen_Markus
Honored Contributor I
6,970 Views

The observation about commenting out these two lines strongly suggests there is a memory problem. Have you tried building the program with various run-time checking options on, such as array bounds checking?

jimdempseyatthecove
Honored Contributor III
5,444 Views

>> The strange thing is that if I comment out lines 46 and 47 xreg2 sucessfully allocates.

That is strange...

Might lines 46 and/or 47 result in an index out of range writing to xreg that then causes a trashing of the array descriptor of xreg2 in such a manner that it looks both deallocated and allocated?

BTW firstReg and lastReg are not defined.

 

Jim Dempsey

hentall_maccuish__ja
New Contributor II
5,453 Views

Yes the debug configuration has nearly all of these set (details below). As I mentioned it does not produce this error in the debug configuration but it also does not throw any other error like out of bounds.

hentall_maccuish__ja_0-1622733876341.png

 

 

0 Kudos
Arjen_Markus
Honored Contributor I
5,448 Views

Perhaps use these extra options (but not the debugging option) in the Release configuration? I have noticed in the past that a debug version may hide problems such as an uninitialised variable because it is more likely to have a value zero than a random value like 234562323523 or some such.

hentall_maccuish__ja
New Contributor II
5,433 Views

Using the extra option on the release configuration fixed the issue. That seems to be because turning on these checks turns of optimization. When I turned the checks off again and disabled optimization (I mean flag /od) it also worked. Does this seem like a compiler bug? I can't see why optimizatgion should produce this difference in behaviour.

@jimdempseyatthecove firstReg and lastReg are global parameters they have values 4, and 9.

0 Kudos
jimdempseyatthecove
Honored Contributor III
5,411 Views

At the top of the code section of the subroutine insert

    print *,"xreg2 is allocated?", allocated(xreg2)
    if(allocated(xreg2)) print *,size(xreg2), lbound(xreg2)

and sprinkle these statements down towards the problem statement. Maybe you will discover something.

(include on both sides any CALL statements)

Jim Dempsey

hentall_maccuish__ja
New Contributor II
5,294 Views

So I tried this suggestion of putting the statements

print *,"xreg2 is allocated?", allocated(xreg2)
if(allocated(xreg2)) print *,size(xreg2), lbound(xreg2)

And everyone returned false including the ones immediatly before the allocate statement that fails with error allocatable already allocated. That is at least consitent as the statement 

if (allocated(xreg2)) deallocate(xreg2)

Did not catch xreg2 as allocated either. However it makes this error even ore obscure to me.

0 Kudos
Steve_Lionel
Honored Contributor III
5,427 Views

Anything that changes the sequence of code, or layout of data in memory, can change behavior. The chance that it is a compiler bug is very low. Indeed, your experiment increased the odds of it being memory corruption due to an error in the program.

It might be interesting to run under Intel Inspector's memory analysis, though I suspect the error will hide again if you do.

hentall_maccuish__ja
New Contributor II
5,312 Views

So finally got round to running the inspector on this. It identifies lots of memory leaks but none of them make sense to me and I would have put them down to false postive which I know the inspector can throw. The detected memory leaks falls mostly into two groups. Open file statements and allocate statements. They generally also occur outside of any paralell region. Below is a code snipiet that show the relevant code for a couple of representative examples.

 

program Console1
use Header
real (kind=rk), allocatable :: moments(:,:),weights(:,:)
integer :: ios
call setModel
allocate( moments(3,lastMoment),weights(3,lastMoment))
open (unit = 1001,file=trim(pathMoments) // 'LSMoments_60.txt', action='read', IOSTAT = ios)

end program Console1
module Header
implicit none
integer, parameter :: rk = selected_real_kind(15)
integer, parameter :: lastMoment = 21
character(len=250), protected :: path
    contains
    subroutine setModel
    implicit none
    path = "C:\Users\jamie_m\Dropbox\SourceCode\upgradeProject\VSProj - Copy\outBaseline\"
    end subroutine
end module Header


 

 

Lines 6 and 7 of the main program are being flagged as memory leaks. Does anyone see something wrong in what I've done here?

 

Thanks

0 Kudos
Arjen_Markus
Honored Contributor I
5,305 Views

As it happens, I am experimenting with the Intel Inspector myself. My interpretation of these messages in combination with the source code is that the file is open until the program stops and the memory is still allocated as well. The Inspector program can be told not to report that sort of things, as you know the file will be closed when the program stops and the memory will also be deallocated without you as a programmer having to take explicit action.

 

hentall_maccuish__ja
New Contributor II
5,300 Views

Thanks. Do you know what combination of settings I should change for it not to report these as errors?

0 Kudos
jimdempseyatthecove
Honored Contributor III
4,984 Views

You should add "implicit none" to program Console1.

After doing this, you should see an error on line 7 relating to pathMoments.

Without the implicit none, the variable pathMoments will be implicitly be created, but the value will be undefined. Running the resultant program should result in an error, and that may complicate the matter of the memory leak reporting.

Jim Dempsey

0 Kudos
hentall_maccuish__ja
New Contributor II
4,979 Views

Hi Jim,

Sorry the code above was just a snippet as I was having trouble creating a minimal replicable example. The full program doesn't have these issue (i.e. has implicit none and path id defined). I have got to the bottom of this issue now anyway, although I would still like to know how I could do a better job of catching these errors in the future.

Thanks,

Jamie

0 Kudos
Arjen_Markus
Honored Contributor I
5,301 Views

I use the GUI and there you have the option, when setting up a new analysis to turn off "Report still-allocated memory at application exit". The command-line it produced for this setting was:

 

inspxe-cl -collect mi2 -knob detect-invalid-accesses=true -knob detect-leaks-on-exit=false -knob detect-resource-leaks=false -knob enable-memory-growth-detection=false -knob enable-on-demand-leak-detection=false -knob still-allocated-memory=false -knob stack-depth=16 -mrte-mode=auto -module-filter-mode=include -app-working-dir ...

I suspect "-knob still-allocated-memory=false" is meant for this.

hentall_maccuish__ja
New Contributor II
5,257 Views

Unticking the flag "Report still-allocated memory at application exit" did not cause the inspector to stop falgging these open and allocate statements as causing memory problems.

I have cleared up one or two little issue the inspector flagged and that did not fix the already allocated error I am getting. Apart from lots of allocate and open statements as innocous as the example above the only thing the inspector is finding now is an issue "Memory leak z_Windows_NT_util.cpp libiomp5md.dll".  

0 Kudos
hentall_maccuish__ja
New Contributor II
5,126 Views

So as predictied by a couple of people here it was caused by an overflow issue. Line 15 should read "real (kind=rk) :: beta(regPeriods-1+controls, 1)" rather than "real (kind=rk) :: beta(regPeriods+2, 1)" so that when parameter controls was increased from 3 to 5 the calls to doReg passed a smaller array than was anticiapted.

My remaining doubts is how could I have detected this more quickly as it took a couple of days to get to the bottom of this? I don't understand why neither the inspector nor the combintation of diagnoistic flags I showed above detected the misaligned between what I was passing to the subroutine and what it expected. Have I missed some compiler options that would have detected this for me? 

Thanks,

0 Kudos
Reply