Memory leaks with variable format expressions

[netphilou31]

Hi,

Just a simple question and concern, is it known that variable format expressions lead to memory leaks?

To illustrate my concern, we have a dll which is loaded/unloaded hundreds of times. In this dll we use variable format expressions in form of ...,<N>(1X, PG13.6),...to read/write data.

We have been facing an "insufficient memory resources" issue when the dll is loaded more than 1000 times. After investigating all other possible memory leaks (mis-deallocation, etc..), and removing complete sections of code, we have found that the problem was due to the use of this type of formats. Each time the dll is unloaded around 1MB of memory is lost in our case. Replacing these formats by constants formats solved the issue.

Is it something which is known/documented somewhere?

Subsidiary question: does replacing <N>(1X, PG13.6) by *(1X, PG13.6) can also solve the issue?

Best regards,

P.S.: I have found an old post dating from 2015, which already mention the issue (https://community.intel.com/t5/Intel-Fortran-Compiler/Memory-leak-in-FORTRAN-format-with-variable-string-length/td-p/1058894) but it seems not being solved (I am using OneAPI compiler).

Phil.

[Steve_Lionel]

It's not an issue I ever encountered, but it doesn't astonish me. I don't know the exact mechanism the compiler uses for "thunks" (see Doctor Fortran in "Think, Thank, Thunk" - Doctor Fortran (stevelionel.com)), but it is highly likely that the presence of VFEs (which involve thunks) requires allocation of some space when the DLL is loaded, and there may not be code to deallocate it on an unload, as it is expected that the process is running down. Not using a VFE would be a workaround, and yes, the F2008 "Infinite repeat count" feature was intended to replace such usages, so if that works for you, I recommend it as a way of avoiding the thunk allocation.

I would suggest that repeatedly loading and unloading a DLL is wasteful and could be problematic, though I could imagine some usages where that may be useful. 

[netphilou31]

Hi Steve,

Thanks for your reply and always advised comments (and, from time to time, exiting your retirement to provide your great experience to unfortunate coders). I have changed all the involved formats, but for some of them I also had to slightly change the corresponding source code because I noticed that using the F2008 "Infinite repeat count" works well only when located at the very end of the format string! furthermore, anything (constant text or line feed "/" for example) appearing after such a repeat count expression is completely ignored! but anyway, I noted to not use such VFEs in the future.

To come back to your comment about the loading/unloading of a DLL during the workflow, I agree but in this particular case, I think we cannot do differently, since it's by design that the DLL must work this way (and unfortunately, we cannot change the specifications).

Best regards,

Phil.

[Ron_Green]

if you send a typical VFE expression in your code, I can wrap it with a main program and run it through Intel Inspector to check for leaks.  I'd like to use an actual vfe from you just so I get something close to what you are using.

If there is no memory leak in a reproducer like this then we can point the finger at the unloading of the DLL and it's allocations.

[Steve_Lionel]

Infinite repeat count works ONLY for a parenthesized format group. It does work, I use it often.

[netphilou31]

Hi Steve,

I know that, but if you add something after the parenthesis it is simply ignored, let's consider:

100 format(1X,I3,*(1X,1PG13.6),/,'END OF DATA')

Neither the line feed nor the 'END OF DATA' string are processed!

you need to change the format to:

100 format(1X,I3,*(1X,1PG13.6))

and add an additional write satement:

write(IUL, 100) I,(X(J),J=1,N)
write(IUL, '(A)')) 'END OF DATA'

and changing the format to:

100 format(1X,I3,*(1X,1PG13.6,:),/,'END OF DATA')

does not solve the issue.

Best regards,

[FortranFan]

The use of asterisk (*) as a repeat specifier corresponds to "unlimited-format-item" per the Fortran standard and the standard stipulates this "unlimited-format-item" be the last on the comma-separated list.

[Steve_Lionel]

Very easy to reproduce.

program VFEtest
    implicit none
    integer :: n
    integer :: a(3) = [1,2,3]
    
    n = 3
    print '(<n>(I2))',a
    end

 

[Steve_Lionel]

That's the way the standard is written. * acts like a "very large number" and it just repeats the parenthesized group until a data transmitting descriptor (the G in your example) is reached but there are no more I/O list items. No further format items are processed.

[netphilou31]

Good to know that.

I was wondering if it was a normal behavior or a side effect.

Thanks.

[Steve_Lionel]

This behavior is not specific to the unlimited-format-item. Quoting the standard (emphasis mine):

"Whenever format control encounters a data edit descriptor in a format specification, it determines whether there is a corresponding effective item specified by the input/output list. If there is such an item, it transmits appropriately edited information between the item and the record, and then format control proceeds. If there is no such item, format control terminates. "

The colon edit descriptor performs a similar check, except that if there is another item then processing continues past the colon.

If you absolutely need some formatting to appear after the I/O list is exhausted, then you can't have a data edit descriptor without a corresponding list item. In an example such as yours, this means specifying the group repeat count precisely, which is why a VFE was used. The standard alternative would be to construct a format in a character variable, such as with internal WRITE, that has the appropriate value, then use that variable as the format.

[Steve_Lionel]

Oh - and regarding the memory leak... That Intel internal procedure that allocated the memory, _intel_alloc_bpv, is creating the storage for the "thunk" I described. Here, "bpv" means "bound procedure value". VFEs are essentially internal (contained) procedures - when they are called, they need the variable context of the program unit containing the format. As I explain in the Doctor Fortran post I linked to, this used to be a bit of executable code written to the stack, but that became impossible when stacks became non-executable due to malware concerns. Evidently, the compiler run-time allocates a big chunk of dynamic memory to store these BPVs (each one is small but there could be many of them) but fails to deallocate that on DLL unload. Probably an easy fix. BPVs are also used when you pass an internal procedure as an actual argument.

[FortranFan]

@netphilou31 ,

With your use of G edit descriptor - a good choice really since it's generic and support any intrinsic type including CHARACTER type - note the Fortran standard is reasonably flexible that you and any fellow developers can consider some "clever" programming: e.g.,

   character(len=*), parameter :: NL = new_line("") // "\n" !<-- suffix only for illustration
   real, parameter :: x(*) = [( i*atan(1.0), integer :: i = 1, 4 )]
   character(len=*), parameter :: fmtg = "(*(1pg13.6,:))"
   print fmtg, x, NL
end 

The above program behavior using Intel Fortran (IFORT) processor:

C:\temp>ifort /standard-semantics p.f90
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.

Microsoft (R) Incremental Linker Version 14.30.30706.0
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:p.exe
-subsystem:console
p.obj

C:\temp>p.exe
 0.785398       1.57080       2.35619       3.14159
\n