Fatal compilation error: Out of memory Asking for..

[arney]

I have a program that has a large data module for which I am getting a fatal compilation error.  What makes my case especially curious is that, after investigation, I am finding that it compiles successfully simply by removing the comment lines.  This makes no sense to me, as comment lines aren't supposed to be compiled.  I have attached the example modules and compilation output.  

1. Successfully Compiled Routine: 30,570 lines

2. UNsuccessfully Compiled Routine: 66,198 lines

Again, the only difference is the comment lines in the code (i.e, there are approximately 36,000 more comment lines in the unsuccessful version).  This doesn't make sense to me given the files are only different because of the commented out lines (i.e, log of changes). Nevertheless, this is causing the problem as far as I can tell.

I have attached the routines, along with compilation logs in the zip file.  Again, the data themselves are identical in these fortran files, but just not the comment lines.  You are welcome to try these on your end to see if you successfully compile when my computer does not.  

Attachments: BER_Compile_Fail_to_Intel.zip, which includes

1. BER_Data_HK.HK311_EPD.20170607.f90: HK3.1.1:  (unsuccessful compile)
2. BER_Data_HK.HK311_ERG.20170519.f90: HK3.1.1:  (successful compile)
3. BuildLog.Prj02ModuleData.BER_Data_HK.EPD.20170607.htm (IVF Release unsuccessful compile log)
4. BuildLog.Prj02ModuleData.BER_Data_HK.ERG.20170509.htm (IVF Release successful compile log)
5. Params.f90 (supporting module).

Thanks! 

[andrew_4619]

compiles OK for me. There is no program there to make only a module with  lits of data statements and a module with parameter statements. Both compile OK.

C:\Users\...\Desktop\ifort_test>ifort /c /O2 /Qsave Params.for BER_Data_HK.HK311_EPD.20170607.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on IA-32, Version 17.0.4.210 Build 20170411
Copyright (C) 1985-2017 Intel Corporation.  All rights reserved.


C:\Users\...\Desktop\ifort_test>dir *.obj
 Volume in drive C is OS
 Volume Serial Number is 4273-53C3

 Directory of C:\Users\...\Desktop\ifort_test

22/06/2017  19:31         1,435,231 BER_Data_HK.HK311_EPD.20170607.obj
22/06/2017  19:31             2,417 params.obj

[arney]

Yes, I just provided modules for compilation testing, not the entire program.  It was only failing for this module, hence the request for assistance.

Glad to hear you were successful with both routines!  However, it still has me puzzled and doesn't particularly help me as I don't have access to another system configuration where this compile would be successful.  My system doesn't have a significant amount of memory, and it's an AMD.

The compiler settings you used differ slightly from mine:  ifort /nologo /O2 /Qsave /libs:static /threads /winapp /c /Qm32 .  Any idea if these could have a role?  

Also, I'm compiling from within VS2015, not command line.  Wouldn't seem to matter, but just pointing that out.  

Thx!

[andrew_4619]

Well it sound that maybe it is a lack of system resources. I can't test your system..... What I would suggest is try the command line build that I did. If that works try it with your options, if that works it is some issue with the VS IDE or the extra resources it uses. As a quick fix you could compile that source on the command line and include the object file(s) in your project rather than the source file.

[mecej4]

The files differ in a more substantial way than just the comment parts. Here, for example, are the first file-diffs:

   Data EH0(HC$  ,Bag2$ ,001), DH(HC$  ,Bag2$ ,001) / 3.551539, 0  /
   Data EH0(HC$  ,Bag2$ ,001), DH(HC$  ,Bag2$ ,001) / 3.551539413, 0 /

It should be expected that the compilation of the second file would take more time since thousands of longer decimal numbers have to be converted to internal (binary) format. I find that compiling the longer file takes about five times longer than the shorter one; the increase appears out of proportion, since the EPD file is only twice as long as the ARS file.

Whether compilation fails or not depends on the compiler version and target architecture (32-bit or 64-bit). Here are my results (compilation time in seconds, from the command line), on a Sandybridge laptop (i7-2720QM) with 8G of RAM, using /c /O2 /Qsave:

   Compiler       ARS     EPD
   
17.0.4, 64 bit   5.540  30.313
17.0.4, 32 bit   6.403  32.520

16.0.4, 32 bit   5.363   FAILED, out of memory
16.0.4, 64 bit   5.380  27.525

[arney]

Thank you.  I had not realized the data had different numbers of decimals.  Your comments make sense.  The ARS version had the data populated via a format that applied precision-limitation of 7-digits or less (limit for REAL data).  The EPD version had no limitations on decimal precision for populating the data. I never realized that could make a difference, as I was just thinking of the REAL variable type limits, not the process to read/compile/store the REAL value.

[Steve_Lionel]

I find it hard to believe that a few extra digits in a single-precision value would have any detectable effect on compile time or memory. Something else is happening. /Qm32 simply says to compile for 32-bit - the other options have no significant effect on compiling (just select which set of libraries are named in OBJCOMMENT directives.)

I could not get the sources to fail with the 18.0 beta compiler (only one I have on this system right now.) I'd recommend submitting a problem report through the Intel Service Portal.

[arney]

Steve.. Thank you for your comments, and CONGRATS on your retirement!  That was actually my initial thought in regards to the extra digits: even if a > 7-significant figure value is being used to store into a single-precision variable, it still is eventually just stored with the same amount of memory (4-bytes).  

[arney]

follow up on Steve's recommendation: how do I submit a problem to the Intel Service Portal? 

[jimdempseyatthecove]

Although this is a compiler bug, and should be corrected, let me offer a simple solution:

Write a simple data import subroutine that OPENs BER_Data_HK.HK311_EPD.20170607.f90 as an input file, scans for Data, parses the variable name and indicies (converting parameter text), then loads the appropriate array entry.

In this manner, the input file remains the same. IOW can be used as data file or module file.

Jim Dempsey

[Steve_Lionel]

To submit a problem report, go to https://software.intel.com/en-us/support/ and click the link under Individual Support. When it asks you how to find which product you want, select "Search by name" and enter Fortran.

[arney]

ok.. I submitted a support request.  

FWIW.. both routines compile in Compaq Visual Fortran 6.6c on a Windows XP IA-32 in about 4 seconds.   no issues.

[andrew_4619]

For what it is worth your 32 bit build failed in 17.0.1.143  and mecej4 noted it failed in 16.0.4 however it worked for both me and mecej4 in 17.0.4 so why not test on that version? If intel say it is a bug it may be "fixed" already there won't be a patch issued for your current version.

[arney]

ok... I'll download update 4 and see if it matters.

[mecej4]

I ran the compilations again, on an older (2006) HP Pavilion PC with an AMD 64-X2-4200+ CPU running Windows 10, using CVF 6.6C and Ifort 17.0.4 (32-bit). I chose /opt:0 for CVF and /Od for Ifort. The compilation times (given in seconds, see below) reveal that something is amiss in the Ifort compiler -- has it been optimized for source files that are very different from the present examples and, as a result, the performance for the present example is really poor? Note that I refer to the optimization of the compiler code itself, not the optimization of the object code for the example.

                ARS    EPD
CVF 6.6C        2.6    4.7
IFort 17.0.4   24.8  116.7

Arney, perhaps you can add this extra information to your support request.

[arney]

I provided a link to this thread in my support request.  hope they'll review what's presented here.

[arney]

Just updating everyone... I downloaded iFort 17.0.4 (Update 4) and, although it took a terribly long time, it did successfully compile the EPD version.  Still puzzling why it only takes 4 seconds in CVF and probably 12 minutes in IFort 17.0.4 32-bit (note: I have an AMD chip, so wouldn't be surprised if this is a factor in some way), but I doubt this will be something Intel Support will investigate further.  

Compilation Command and Switches: ifort /nologo /O2 /Qsave /libs:static /threads /winapp /c /Qm32 %1.f90

[andrew_4619]

      REAL, DIMENSION(POL$,MODE$,TECH$) :: EN0 
      REAL, DIMENSION(POL$,MODE$,TECH$) :: EM0 
      REAL, DIMENSION(POL$,MODE$,TECH$) :: EH0 
      REAL, DIMENSION(POL$,MODE$,TECH$) :: EV0 
      REAL, DIMENSION(POL$,MODE$,TECH$) :: ES0 
      REAL, DIMENSION(POL$,MODE$,TECH$) :: DN  
      REAL, DIMENSION(POL$,MODE$,TECH$) :: DM  
      REAL, DIMENSION(POL$,MODE$,TECH$) :: DH  
      REAL, DIMENSION(POL$,MODE$,TECH$) :: DV  
      REAL, DIMENSION(POL$,MODE$,TECH$) :: DS 

Consider the above which are 11x10x330 arrays. You have a DATA statement that initialised 1 element of EV0 , then data statements to set single elements of a load of other arrays and then a data statement that sets another element of EV0 etc etc. The DATA assignments for EV0 are spread over 66,000 lines of code.  Also if you consider the storage order  of the elements of EV0 you are jumping around all over the place. I suspect that the methods used for sorting this out are in no way optimised (and indeed are very ineffecient) for dealing with data  arranged in this way. If you had one set of assignments that filled EV0 in storage order, and then one that filed EM0 etc maybe it would be a lot faster. 

[arney]

Your point may be valid about the population of the arrays in the data statements are not exactly in the most optimal order.  They are filled in array dimension order, but the issue was really that the two routines only differed by amount of embedded comments.  The order is the same in both, but one compiled and one didn't (although now it does VERY SLOWLY, in Update 4).

[Steve_Lionel · Accepted solution]

That you have a AMD processor has absolutely no bearing on compile-time. Given the difference between CVF and ifort, it is almost certainly due to the very different designs of the compilers' optimizers and code generators in how they deal with certain constructs.