"insufficient virtual memory" at runtime - command line environment for compiling/linking - Page 2

Olaf_L_ · ‎01-11-2013

Hello, and good afternoon!

Concerning: Compiler mogration to your product / program worked well with other compiler (Lahey) for many years - here "insufficient virtual memory" at runtime -

To introduce ourselves: We are a team of 3 developers, in the process of testing Intel Fortran for migration of existing code to it (using evaluation version of compiler). As you probably hear often with Fortran, it has a lot of legacy code (F77) included.

We are planning for purchase of 2 or 3 licenses of the intel fortran compiler as soon as the main issues, which I'll outline below, are resolved, possibly with your help.

Platform: Windows, currently testing on both x32 and x64 -- and it ist NOT from within visual studio - but essentially commandline -- so far we work with Eclipse so commandline invoking from Eclipse appears as a quick way do it. Setting up with the prerequisites (free Microsoft studio and SDKs installed as needed for linking especially) and usage of scripts as provided (ifortvars.bat calling cascade of others) is in place --
as is evidenced in the fact that it compiles (small changes to the code itself were necessary such as how to process commandline arguments etc.

If there is any great way for setup in Eclipse, setting the environment there, we'd certainly appreciate that also.

-----

The current plan is to see whether we will be able to manage the conversion -- as we have a run time error that looks like a menory leak of sorts (I dunno whether it actually is but it looks like it). That wasn't ever observed with an exe compiled with the old compiler.
This appears everytime the program runs *long enough* - and not right after start but after a few minutes of running (numerical calculations).
If you just start the gui (see below details what we use) and use it or let it sit there open, nothing bad happens with the memory (there are window redraw issues, but I save this for another time or get it worked out - unless you tell me this may give some clues for the crash problem at hand).
So, no program crash just from the gui. Also, if you start a "calculations" job (number crunching job) and if it is short enough to finish, there will also be no program crash.
I can observe that the "RAM" used by the process grows rapidly, up and until the breakdown. For comparison: A "Healthy" version of the application uses about 18MB RAM just when the gui is started; 170MB RAM wenn doing a "calculations" job, as per taskmgr. Used RAM with the intel compiled version is now gradually growing up to 1.4GB (on my machine) before the crash. GUI alone is just as small as it should be. This happens with both x32 and x64 versions. If a "calculations" job is short enough then the program does not crash. If you do not close the application (gui) and start another job it will add memory and then crash, i.e.; the same job as before (that went through) may now crash real soon.

Some code details that I want to mention: From the "old age" there are still quite a number of common blocks, which generate warnings at compile time. I may send it to you, see below. Also, I studied the forums and it is believed that ALLOCATE statements can, at times, cause problems. We do have them in the code but I believe they are not executed the way I test right now.

I have prepared additional information;

Please find below the error message. Also find the output of the "set" command in cmd window and one compiler log.

We use Intel Composer XE 2013; also Microsoft Visual Studio 10.0 and Windows SDK 7.1, as visible in the environment variables output if you need it.
The environment as for the information I am forwarding to you is x32. In the end, we plan to to a x64 application though.
We also use a toolkit called Winteracter for graphical user interface programming of our application. The linked libraries are recongnizable in the compile messages.
No other libraries are linked into the executable. Other than what the compiler asks us to. We had "unresolved externals" initially and I researched the libraries that need to be linked. I added them manually into the makefile and they are visilble in the compile protocol.
No DLLs are created or called at this stage (That is planned and was done before but we first want to get the basics right).

If you could get back to us and possibly direct us in the right direction, suggest tests we should perform, compiler flags and such, please let us know.

I did read through the forums but so far found no answer to my issues. If this is Steve who will answer I have read up on how you've helped others as well.
It seems to me that our setup here is slightly different from what many people use (Eclipse rather than Visual Studio).

Also, I would like to say thank you in advance for your help.

If we get our program converted successfully, we intend to become loyal customers; we were with Lahey for 17 years; they seem to have come to some sort of end with compiler development, especially on Windows.
By the way we also consider Linux. We are regular users of Linux (for other tasks at the moment) but it may also be an option for the development work in question. It seems to be far less compicated to set up the environment there (not using something as Visual Studio). I think I played with a trial version of the compiler once. We would consider switching if, for example, you told us that this avoids issues that can cause the present problem on Windows that I described.

Best Regards,

Olaf Lieser

----

Here is how the error is announced. 2 version of the message. I am curious why there are 2 ways by the system to report it. I have not yet nailed which of the 2 messages appears when. Are there even 2 general problems present?

forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
adcos_0162.exe 004F4934 Unknown Unknown Unknown
adcos_0162.exe 004C2E06 Unknown Unknown Unknown
adcos_0162.exe 004AB1B2 Unknown Unknown Unknown
adcos_0162.exe 004772A8 Unknown Unknown Unknown
adcos_0162.exe 001786D1 Unknown Unknown Unknown
adcos_0162.exe 0017374F Unknown Unknown Unknown
adcos_0162.exe 00113715 Unknown Unknown Unknown
adcos_0162.exe 002E2C1C Unknown Unknown Unknown
adcos_0162.exe 0027CCBF Unknown Unknown Unknown
adcos_0162.exe 004A86F3 Unknown Unknown Unknown
adcos_0162.exe 0046A832 Unknown Unknown Unknown
kernel32.dll 752F33AA Unknown Unknown Unknown
ntdll.dll 77609EF2 Unknown Unknown Unknown
ntdll.dll 77609EC5 Unknown Unknown Unknown

----

forrtl: severe (41): insufficient virtual memory

Stack trace terminated abnormally.

UPDATE: I noticed that the makelog.txt file gives only standard output; warnings appear elsewhere (standard error?), therefore - here is a snippet of it. This repeats as the common blocks are called from several routines. This snipped may all be unimportant - but here it is, just in case.

G:\daten\ae\ADCoS_j162_intelWinter>make adcos
ifort -Qvec-report0 -include:"c:\winteval\lib.if8" /heap-arrays /assume:byterecl /c fed_0264_1.f90
Intel(R) Visual Fortran Compiler XE for applications running on IA-32, Version 13.0.0.089 Build 20120731
Copyright (C) 1985-2012 Intel Corporation. All rights reserved.

fed.inc(16): remark #6375: Because of COMMON, the alignment of object is inconsistent with its type - potential performance impact [VOLALL]
real*8 volAll ! Volumen (Verdraengung) jedes Elementes (basierend nur auf Aussenquerschnitt)
----------------------^
fed.inc(93): remark #6375: Because of COMMON, the alignment of object is inconsistent with its type - potential performance impact [FXAERONABE]
real*8 fxAeroNabe,fyAeroNabe
----------------------^
fed.inc(93): remark #6375: Because of COMMON, the alignment of object is inconsistent with its type - potential performance impact [FYAERONABE]
real*8 fxAeroNabe,fyAeroNabe
---------------------------------^
fed.inc(94): remark #6375: Because of COMMON, the alignment of object is inconsistent with its type - potential performance impact [FXAERONABE_OLD]
real*8 fxAeroNabe_old,fyAeroNabe_old
----------------------^
fed.inc(94): remark #6375: Because of COMMON, the alignment of object is inconsistent with its type - potential performance impact [FYAERONABE_OLD]
real*8 fxAeroNabe_old,fyAeroNabe_old
-------------------------------------^
fed.inc(91): remark #6375: Because of COMMON, the alignment of object is inconsistent with its type - potential performance impact [T_H_EIN]
real*8 cw_turm,cw_turm_lokal,t_h_ein
--------------------------------------------^

Johannes_Rieke · ‎01-16-2013

Hi Olaf,

sorry for the wrong tip on stack instead of heap.

If you use /check:bounds be aware that if you have something like this

[fortran]CHARACTER somechar*(*)[/fortran]

in your code, you get every time there an error, although it works quite good.

[fortran]CHARACTER(len=*),DIMENSION(*) :: somechar[/fortran]

will work instead with array bound checking. I have old code with a lot of 'old' character definition in paramter list and I don't want to change the 2000 subroutines for it. Currently, I can't use the array bound checking in runtime as a drawback. But I am a lucky one, my program runs without memory issues.

You must have a lot of degrees of freedoms, if your Cuthill-McKee runs into stack problems...

However good luck in finding and fixing the problem.

Kind regards,

Johannes

Olaf_L_ · ‎01-17-2013

Regarding the character statement: yes we do have some of the old kind; I will see how many and if manual conversion would be quick. And yes, range checking is part of our routine tests when the code has changed; I will need this. The old compiler accepted it as it was and did a proper arrays check and subroutine header variable mismatch check and stuff.

Regarding the Cuthill-McKee: I do actually not trust that where "death knell" occurs there is necessarily the location of the problem. The program may have been "mortally hurt" before and stumble on for a while before crashing. I have experienced such things before.

Our degrees of freedom are not that many actually. We have to carry quite many variables, mostly set up as arrays. Certainly Old Style. The core of the program was created in the late 90s (sooo 90s so to speak) with a conservative attitude on top of that -- meaning any F95 or F2003 features or something like that are largely absent.

Essentially it is a FEA process that reoccurrs many times (order 1e6 times) with a real small number of DOFs compared to, say, today's commercial 3D FEA models.

------

Next thing will be the tests that Steve suggested. I did not get to it yet -- but I will report back and post here in the forum what comes out of it.

Olaf

John_Campbell · ‎01-20-2013

Steve, You stated earlier in this post : "if you are compiling from the command lines, some diagnostics go to standard error. If you are redirecting compiler output, use 2>&1 to merge the streams" . I have been looking for how to do this for a long time. (DOS help is hard to find). Thanks for the advice, as it has solved a problem I have had with a batch file for a simple compile and run. The following simple batch file now works well for my needs. Others might find it helpful. . John. . set options=/Tf %1.f95 /free /O3 /QxHost del %1.obj del %1.exe now >> %1.tce echo ifort %options% >> %1.tce ifort %options% 1>>%1.tce 2>&1 type %1.tce dir %1.* >> %1.tce %1 >> %1.tce notepad %1.tce

Olaf_L_ · ‎02-25-2013

.... actually to give an update on this after some weeks:

(1) I also tried another vendors compiler in the meantime, NO SUCH mem problems occur; so it is only with the intel compiler. Still, I kept investigating with the intel compiler also.

(2) The usage of Intel Inspector pointed to the problem that the Intel compilation had: Locally defined subroutines with no access to global variables whatsoever; arrays plus the array dimensions passed into the subroutine (the latter defined in the main program or a higher level subroutine). In our view completely legal, even in the old days (f77 -- whereas f9x should be downwards compatible should it not?), and used for a long time with no issues, as mentioned. There were some 7 such locations causing problems. The program as a whole has many more than that, most of which had no such issues.

Intel Inspector then just pointed to the subroutine header line as the bad one. I proceeded to take out one variable after the other from the header and handed them over via extra module statements.

Turns out that -- and I consider this as a workaround -- I need to pass the array *dimensioning* variable outside the subroutine header, i.e.; via a module. So I proceeded to define a "parameter module" containing these dimensions and shared this module in any soubroutine that uses the array(s) in question. The arrays themselves are still passed via the header.

Once again, I consider this a workaround; especially since I (A) think that the code before was completely legal and never had issues (B) I do not completely understand WHY this helps and (C) the arrays and the dimensions are currently passed via different routes and I do not generally think this is a good idea.

Remarks to (C): Passing the array in the module as well would require major changes to the code: Remember some of this is old code and many variables are handled via common blocks. I would therefore need to restructure all of this, or created duplicate variables. At this stage I am hesitant to do any of this. The alternative to revert to many more global variables rather than local ones can't seem to be a good idea I believe. I think local programming is in many cases just easier concerning the safe handling of variables, namespaces and such.

With these changes, no more memory leaks are reported by Intel Inspector, and no increase of mem usage is indicated by the OS; the program (just "solver", no gui library linked so far) runs just fine.

IanH · ‎02-26-2013

What did Inspector actually say the problem was? From your description I think you are talking about explicit shape dummy arguments. They are still very much in the language.

SergeyKostrov · ‎02-26-2013

>>... >>Win32bit crashes as it exceeds some 2GB (growing to there gradually, starting from 150MB) It is a well known limitation for 32-bit applications that they can't allocate more than 2GB of memory and in a real life for heavy applications the crash could even happen at significantly lower amounts of allocated memory. Take into account that many dependent DLLs are mapped to the address space of the application. >>Win64bit gradually takes all mem it can get, starting from 150MB until paralysation (my machine has 10GB plus 3GB swap ) 3GB is Not enough and try to use 16GB or 24GB values. Also, there are two values that control size of the Virtual Memory file, Min and Max sizes. Here is an example, if your application tries to allocate 8GB of memory than a Min value for the Virtual Memory file has to be set to 8GB or higher, and a Max value for the Virtual Memory file has to be set to 16GB or higher ( use Min x 2 as a rule ). I recently did a test with some Fortran codes ( Matrix multiplication / 16Kx16K / REAL(8) data type ) and without option heap-arrays:1024 ( that was already mentioned in a post earlier ) it didn't work even if I have on my computer 32GB of physical memory and 64GB of Virtual Memory ( Min value ).

Olaf_L_ · ‎02-26-2013

@ IanH

Intel Inspector simply says "Memory Leak"

@Sergey

I appreciate your efforts to try to find a solution.

You see, the problem is not the fact that the program demands more mem than a 32bit environment would be able to provide. Kindly read the problem description and the subsequent correspondence: In a healthy state the program comsumes a mere 150MB, constantly. It also does in its latest state (with workaround as described in my last post 1 day ago).

The cause I am after is the described memory leaks of a compiled "unhealthy executable", which lead to ever increasing mem usage (and therefore ultimately crash on Win32 or paralysis on Win64). Now what creates a "healthy executable" in this context: (1) Compilation of the code with other vendors' compilers or (2) compilation with Intel on Linux or (3) doing the workaround described in my last post, after being pointed there by Intel Inspector.

So, only with intel on Windows (compiling the original Fortran code without the workaround -- the same that has run for years with other compilers) do I have these issues, i.e.; an "unhealthy executable".

So I do not have a problem due to memory limitation, but one due to memory leaks

In any case: Again, thank you for your comment.

John_Campbell · ‎02-26-2013

Olaf,

You could create a subroutine entry/exit report (to file) that reports the value of key dimension arguments and uses "sizeof" to report the size of key arrays. The report could include the variable name, nominal dimensions, SIZEOF and LOC. The use of LOC might be useful to indicate how the memory leak is shifting. ( try converting LOC to a string as 130,000,000 to make the address more readable)
I'm not sure how extensive the call structure is, but developing a report for entering key routines might give you a better picture.
You indicated you used few F2003+ extensions. Do you use ALLOCATE or POINTER ? Allocatable variables should be automatically released on exit, but pointer allocations can become lost and a source of memory leakage, especially where an allocated pointer is linked to a new memory allocation.
As you are not getting a stack overflow, the problem looks more like allocatable arrays, rather than automatic arrays.

I hope some of these ideas might help.

John

IanH · ‎02-26-2013

A code example (as a starting point - declarations of the actual arguments, the call statement itself, the salient bits of the specification part of the called subroutine) would be nice to see.

So there's a memory leak associated with explicit shape dummy arguments? I could imaging memory allocation happening if the actual argument was an expression, or perhaps a non-contiguous (or unknown contiguous) array. Is this the case?

Missing an explicit interface when one was needed could also create some fun. How did compiling and running with the diagnostic and runtime check flags go?