Memory leak with Fortran subroutine

[stankoz]

Hello,

I have converted a publicly available fortran simulation into a subroutine within a C project. I am encountering steadily increasing memory usage with each call to the routine. Running Inspector with the 'memory leaks' option produced no problems. I think there may be an issue with the use of pointers and the SAVE declaration attribute within a module of the subroutine. Do I need to manually deallocate every allocatable array and targets of pointers before returning from the subroutine? What is the most thorough way of clearing up all memory usage at the end of each simulation run? All local allocatable arrays will be deallocated once the routine exits, correct?

Here is an example of the module:

      MODULE VDFMODULE
        INTEGER, SAVE, POINTER              ::MT3DRHOFLG
        INTEGER, SAVE, POINTER              ::MFNADVFD
        INTEGER, SAVE, POINTER              ::NSWTCPL
        INTEGER, SAVE, POINTER              ::IWTABLE
        INTEGER, SAVE, POINTER              ::NSRHOEOS
        REAL,    SAVE, POINTER              ::DENSEMIN        
        REAL,    SAVE, POINTER              ::DENSEMAX
        REAL,    SAVE, POINTER              ::DENSEREF
        REAL,    SAVE, POINTER              ::FIRSTDT
        REAL,    SAVE, POINTER              ::DNSCRIT
        REAL,    SAVE, POINTER              ::DRHODPRHD
        REAL,    SAVE, POINTER              ::PRHDREF
        REAL,    SAVE, POINTER              ::HDRY
        REAL,    SAVE, POINTER              ::HNOFLO
        INTEGER, SAVE, POINTER,  DIMENSION(:)                ::MTRHOSPEC
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::PS
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::RHOCR
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::RHOCC
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::RHOCV
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::HSALT
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::ELEV
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::DCDT
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:,:)          ::COLDFLW
        REAL,    SAVE, POINTER,  DIMENSION(:,:,:)            ::PSOLDITER
        REAL,    SAVE, POINTER,  DIMENSION(:)                ::DRHODC
        REAL,    SAVE, POINTER,  DIMENSION(:)                ::CRHOREF
      TYPE VDFTYPE
        INTEGER, POINTER              ::MT3DRHOFLG
        INTEGER, POINTER              ::MFNADVFD
        INTEGER, POINTER              ::NSWTCPL
        INTEGER, POINTER              ::IWTABLE
        INTEGER, POINTER              ::NSRHOEOS
        REAL,    POINTER              ::DENSEMIN        
        REAL,    POINTER              ::DENSEMAX
        REAL,    POINTER              ::DENSEREF
        REAL,    POINTER              ::FIRSTDT
        REAL,    POINTER              ::DNSCRIT
        REAL,    POINTER              ::DRHODPRHD
        REAL,    POINTER              ::PRHDREF
        REAL,    POINTER              ::HDRY
        REAL,    POINTER              ::HNOFLO
        INTEGER, POINTER,  DIMENSION(:)                ::MTRHOSPEC
        REAL,    POINTER,  DIMENSION(:,:,:)            ::PS
        REAL,    POINTER,  DIMENSION(:,:,:)            ::RHOCR
        REAL,    POINTER,  DIMENSION(:,:,:)            ::RHOCC
        REAL,    POINTER,  DIMENSION(:,:,:)            ::RHOCV
        REAL,    POINTER,  DIMENSION(:,:,:)            ::HSALT
        REAL,    POINTER,  DIMENSION(:,:,:)            ::ELEV
        REAL,    POINTER,  DIMENSION(:,:,:)            ::DCDT
        REAL,    POINTER,  DIMENSION(:,:,:,:)          ::COLDFLW
        REAL,    POINTER,  DIMENSION(:,:,:)            ::PSOLDITER
        REAL,    POINTER,  DIMENSION(:)                ::DRHODC
        REAL,    POINTER,  DIMENSION(:)                ::CRHOREF
      END TYPE
      TYPE(VDFTYPE),SAVE:: VDFDAT(10)
      END MODULE VDFMODULE

Thank you

[Steven_L_Intel1]

Not enough information here. First, adding SAVE on a module variable is redundant - the standard says they implicitly have the SAVE attribute. But it's fine to include it just so the reader understands. Second, I see no allocatable variables here, only pointers. What you have not shown is the subroutine code - that is the most relevant part of the problem. If your subroutine has local variables that are POINTERs, then yes, you must deallocate them before returning. If the local variables are ALLOCATABLE and do not have the SAVE attribute, then they automatically get deallocated on exit.

[stankoz]

Thanks Steve, your answer is helpful nonetheless since I thought I might have to deal with the SAVE pointers differently than the others. The subroutine is actually an aggregation of about 100 subroutines so I just figured this was a place to start. There are even more modules with more pointers. There are allocatable arrays without SAVE in the main subroutine, so I will ignore those. Should I also not worry about all the pointers declared in modules? There are many routines that use various elements of these modules but I didn't see and pointers declared locally to any routine. What would be cause for concern as far as memory leaks with pointers used in the subroutine?

[Steven_L_Intel1]

With pointers, if you do pointer assignment to a pointer without deallocating its previous contents first, you can get a leak. I would have expected Inspector XE's memory leak detection to see a problem if it existed. Let me strongly suggest you use ALLOCATABLE rather than POINTER unless you need pointer assignment or something else allocatables cannot do.

[stankoz]

I have no idea why pointers are used so extensively in the code; I did not write it nor do I have experience with fortran pointer variables. There seems to be allocate statements for all the pointer variables so I suppose I could try switching them to allocatables but then I'm not sure what to do with routines like the following: SUBROUTINE SVDF1PSV(IGRID) USE VDFMODULE VDFDAT(IGRID)%MT3DRHOFLG=>MT3DRHOFLG VDFDAT(IGRID)%MFNADVFD=>MFNADVFD VDFDAT(IGRID)%NSWTCPL=>NSWTCPL VDFDAT(IGRID)%IWTABLE=>IWTABLE VDFDAT(IGRID)%NSRHOEOS=>NSRHOEOS VDFDAT(IGRID)%DENSEMIN=>DENSEMIN VDFDAT(IGRID)%DENSEMAX=>DENSEMAX VDFDAT(IGRID)%DENSEREF=>DENSEREF VDFDAT(IGRID)%FIRSTDT=>FIRSTDT VDFDAT(IGRID)%DNSCRIT=>DNSCRIT VDFDAT(IGRID)%DRHODPRHD=>DRHODPRHD VDFDAT(IGRID)%PRHDREF=>PRHDREF VDFDAT(IGRID)%HDRY=>HDRY VDFDAT(IGRID)%HNOFLO=>HNOFLO VDFDAT(IGRID)%MTRHOSPEC=>MTRHOSPEC VDFDAT(IGRID)%PS=>PS VDFDAT(IGRID)%RHOCR=>RHOCR VDFDAT(IGRID)%RHOCC=>RHOCC VDFDAT(IGRID)%RHOCV=>RHOCV VDFDAT(IGRID)%HSALT=>HSALT VDFDAT(IGRID)%ELEV=>ELEV VDFDAT(IGRID)%DCDT=>DCDT VDFDAT(IGRID)%COLDFLW=>COLDFLW VDFDAT(IGRID)%PSOLDITER=>PSOLDITER VDFDAT(IGRID)%DRHODC=>DRHODC VDFDAT(IGRID)%CRHOREF=>CRHOREF C RETURN END I was surprised 'detect memory leaks' found nothing too since I am consistently using an additional 40 MB of RAM with every call to the simulation. I have started an Inspector run with the option to 'detect memory problems,' which will take a while. I suppose the leak could be on the C side of the program as well; I'll ask a question about that in the other forum. The simulation code is kind of a mess. It combines lots of Fortran 77 with many common blocks with modern Fortran and modules. I'm hesitant to mess with the code too much. Is there no way to manually make sure that all memory is cleared upon exiting the simulation? thank you for your assistance

[Steven_L_Intel1]

Since you're using pointer assignments, then you have to stay with pointers. But without an ALLOCATE, there can't be a memory leak, so your first task should be to find where the allocations are done.

[Steven_L_Intel1]

There originally was a much longer list of file types for attachments - but when the forum went live, it got dropped. This should get fixed soon. No .f95, though - I consider that an abomination. .f90 is the free-form file type. Also, we had a syntax highlighter, but it gave us some trouble so it has been turned off until it can get fixed.

[John_Campbell]

I'd support what Steve has said. ALLOCATABLE arrays are automatically released when you exit their scope. Similarly POINTER arrays that are currently associated with an allocated array are also released. However, POINTER arrays that are allocated are not automatically released when you exit their scope. I have written a simple example which shows these 3 different cases. If you run this program, while using Task Manager, you will clearly see the memory leakage that occurs when you allocate a POINTER array and do not DEALLOCATE. Note also that POINTER arrays can have multiple ALLOCATE. In this case the memory previously allocated is not automatically released. The following loop would leak previous allocates; do i = 1,30 allocate (p_array(im), stat=is) end do If you are allocating POINTER arrays, you must explicitly DEALLOCATE them before exiting their scope or re-allocating. I'm not sure of a fool-proof way of determining if a POINTER array is already allocated, before you try to allocate it. This is easily done with an allocatable array. While I have embraced the use of ALLOCATE since F90+, I find POINTER is taking us into an area where programming bugs become too frequent and should be avoided. John PS : excluding a file leak.f95 from being attached does look a bit odd for a fortran forum !

[John_Campbell]

Steve, As for using .f95 as a suffix, we are a diverse group of programmers. I suspect your experience withe F90 and F95 standard committees might colour your view. For me, it was not until F95 was released that I gave up on F77. I am also including a simple example of re-allocating a pointer, which shows how easy it is to leak memory in this way. . ! program to test memory leak2 ! integer*4, parameter :: array_mb = 500 ! integer*4 :: i, im, is integer*4, pointer, dimension(:) :: p_array integer*4 :: mb = 1024*1024/4 ! im = array_mb * mb ! do i = 1,30 !zz deallocate (P_array, stat=is) ! explicit deallocation would fix this problem allocate (p_array(im), stat=is) call use_array (p_array, im) write (*,*) i,' p_array allocated' end do ! end subroutine use_array (array, im) ! integer*4 im, array(im), i ! do i = 1,im array(i) = im-i end do end (I'll wait for the improvements to Attachments) John

[Steven_L_Intel1]

John, my dislike of .f95 predates my involvement in the standard. I will be writing a "Doctor Fortran" post on this subject soon. For now, let me just say that the .f90 file type was never intended to indicate which revision of the standard the source conformed to. You can write fixed-form source that is Fortran 2008 compliant. As for pointers, yes, it is trivial to leak memory with pointers. I just had not seen any code previously in this thread that did any allocations.

[stankoz]

Thanks for the guidance guys. The Inspector run for 'Detect Memory Problems' just finished with 72 problems: missing allocations, uninitialized memory access, and memory leaks. I'm still not sure why the 'Detect Memory Leaks' turned up nothing. Looks like I have a fair amount of work to do on this mess of spaghetti. Let me know if it would be helpful to upload the Inspector results file or the summary XML file. In John's example, most of the pointer usage as far as I can tell is the third variety (i.e. declared as pointers not allocatable, then allocated but not deallocated). The following statements (among others) show up as memory leaks as well as uninitialized memory access: ALLOCATE(MT3DRHOFLG,MFNADVFD,IWTABLE,NSRHOEOS,DENSEMIN, 1 DENSEMAX,DENSEREF,DNSCRIT,DRHODPRHD,PRHDREF,HDRY,HNOFLO) ALLOCATE(PS(NCOL,NROW,NLAY),RHOCR(NCOL,NROW,NLAY), 1 RHOCC(NCOL,NROW,NLAY),RHOCV(NCOL,NROW,NLAY), 2 HSALT(NCOL,NROW,NLAY),ELEV(NCOL,NROW,NLAY), 3 DCDT(NCOL,NROW,NLAY),COLDFLW(NCOL,NROW,NLAY,NCOMP), 4 PSOLDITER(NCOL,NROW,NLAY)) I will go back through and manually deallocate everything that is allocated for the first round of fixes. Should I also just add the ALLOCATABLE attribute to all the pointer declarations? Would this fix the uninitialized memory access? There are also some issues with the C code and usage of realloc and malloc. I will look through the Inspector documentation on how to deal with some of these issues, then maybe start another thread in the C forum if needed.

[Steven_L_Intel1]

You can't "add" ALLOCATABLE - ALLOCATABLE and POINTER are mutually exclusive. Since you are using pointer assignment to remember pointers, you can't use allocatable. Make sure you don't deallocate things you are saving pointers to.

[stankoz]

Oh yeah...I was thinking about the declaration ALLOCATABLE, TARGET, thinking it would work for pointers too but that doesn't make sense...sorry. I'm still a little confused though. Considering the above example, VDFDAT is of type VDFTYPE and contains pointers that are saved in the module. Since these pointers are assigned in the above subroutine, does that mean I cannot deallocate them? Can I not just deallocate everything that has been allocated as the last command block before the main routine returns? How else would I fix the memory leak that Inspector identifies in the allocate statements? I do not need these pointers for anything once the calculations are complete, I extract all the values I'm interested in and pass them to the wrapper algorithm.

[Steven_L_Intel1]

Well, yes, you could use ALLOCATABLE, TARGET. But that wouldn't help you here because as module variables they are implicitly SAVE. You may as well explicitly deallocate all of them when you are done.

[stankoz]

Basically, since this routine gets called repeatedly within an optimization, I just want to clear/free all memory usage after every simulation run. It would be nice if there was a way to free all with one command, but that because I'm lazy. I've located all the variables that get allocated so I will just deallocate them individually. Is it good practice to use IF(ALLOCATED(p_array))DEALLOCATE(p_array) for every pointer, or can I just use DEALLOCATE(p_array_1,p_array_2,...) ? As far as the uninitialized memory access, I'm not sure I understand the error. Should I just initialize all these pointers to zero after they are allocated?

[Steven_L_Intel1]

If instead of making the variables module-level you declared them in the simulation's entry point (I am assuming here that there is just one call per simulation - if there are multiple calls, then ignore this paragraph) and made them ALLOCATABLE, they'd be automatically deallocated on exit. Otherwise, you have to deallocate them. I do suggest the IF test. Regarding initialization - it is your responsibility to initialize the variables after allocation.

[stankoz]

Got it. Thank you for your patience. First time I've ever seen pointers used in Fortran. And Thank you John for your helpful examples. Much appreciated. Also, sorry for not giving more problem specific detail regarding the program structure. It is highly modularized and the VDFMODULE is used by many of the routines that are called within the simulation, but the main simulation routine itself only uses a few of the variables and has mostly a bunch of subroutine calls. The routine that allocates the pointers is separate from the routines that assign/use them. Sorry if this confused you. RE: initialization - I'm not sure why they are not initialized. I was trying to alter the code as little as possible (under guidance of one of the agency's developers), so what is the danger if I left the 'uninitialized memory access' problems unaddressed? There are only two of them. RE: deallocation - I have added a DEALLOCATE statement for each variable in the module at the end of the main simulation routine and started another Inspector run to detect memory problems (I still haven't addressed the issues in the C code). So far though, it looks as if the major memory leak culprits have been patched up. No more 40MB increase with each simulation call!!!

[Steven_L_Intel1]

If you read from an uninitialized memory location, you'll get unpredictable results.

[John_Campbell]

Mixing posts a bit, I noticed in the code examples above in this post: ... MODULE VDFMODULE INTEGER, SAVE, POINTER ::MT3DRHOFLG INTEGER, SAVE, POINTER ::MFNADVFD ... ALLOCATE(MT3DRHOFLG,MFNADVFD,IWTABLE,NSRHOEOS,DENSEMIN, 1 DENSEMAX,DENSEREF,DNSCRIT,DRHODPRHD,PRHDREF,HDRY,HNOFLO) ... This implies that a integer pointer variable (not an array) can be in an ALLOCATE statement. . When was this legal fortran ? or is this an extension ? or has always been illegal ? . I know I might have my head in the sand, but I have rarely needed to use POINTER, probably because I don't think that way when constructing a data structure approach. I've always used pseudo pointers via subroutine arguments, although there must be more functionality available for POINTERS which I don't include in my coding approach. . John . PS, shame about loosing the blank lines and leading spaces in this new forum

[IanH]

Legal as of F90 for scalar pointers, F2003 for scalar allocatables.

[John_Campbell]

Ian, In my previous post, I should have also asked, why would you want to use ALLOCATE for a scalar pointer ? ALLOCATE is used to allocate a variable amount of memory, while a scalar is a fixed memory size. I could think of other changes to the standard that would be more helpful than this. I don't see the benefit of this coding approach. John