Community
cancel
Showing results for 
Search instead for 
Did you mean: 
avinashs
New Contributor I
283 Views

Long compilation time for a subroutine containing data assignments

I have a subroutine that contains about 15,000 lines of code. Except for the declaration statements, the rest of the code assigns data values to the various components of a user-defined type. During compilation, this subroutine takes several minutes (about 6-7 minutes) to compile each time it is newly generated. While it has to be compiled infrequently, I am curious as to why it takes so much time and whether there is a better way to store/assign the data. With each version of Intel Fortran, including the current Intel OneAPI, the time required for compilation keeps increasing.

The compiler settings in Intel Fortran are

/nologo /O3 /assume:norealloc_lhs /real-size:64 /fpconstant /fp:consistent /gen-interfaces /iface:cref /libs:dll /threads /Qm64 /c

With the same settings that are common to both compilers, gfortran takes about 4-5 seconds to compile the same file.

0 Kudos
17 Replies
andrew_4619
Valued Contributor III
270 Views

A couple of comments. I found some years back (I have not tested recently) that above maybe around 15000 lines in a source file and the compile time increased quite a lot in a very non-linear way. I split my code so that no files were bigger than around 10000 lines and compile times for the full project dropped significantly. One could think of so many reasons why algorithms might cause such an effect.   

I also recall a thread some years back where a guy had a huge number of data statements and that was killing the compiler also. You could look for that one.  Split the source file if that is easy to do or read the data at run time from a binary file and take the time hit at run time.

 

Ronald_G_Intel
Moderator
251 Views

I opened a bug report on this for IFX Beta.  But even IFORT has been slow on this initialization.  I saw this in VASP - what code are you working with? 

One could argue that when you have this much data to initialize it's best to use an input file rather than explicit initializations.  One could counter argue that having the initialization and values in initialization statements saves problems with keeping a separate data file and avoid a possible error on file opening and reading.

IFX is really bad at this initialization.  Ifort only so-so.  I had one file that took IFX about 20minutes to compile.  It has somewhere around 50,000 initialization statements, 1 per line.  I thought the compiler hung.  That's the one that I opened the bug report against.  They made SOME improvements but it's still a work in progress.  gfortran is pretty fast at it, by comparison.  I'll go check on that bug.

POSTSCRIPT: I've added this report to the bug report, which is in progress.

Steve_Lionel
Black Belt Retired Employee
233 Views

ifort has traditionally had a hard time with code that has a lot of DATA initializations. This has to do with the way the code generator analyzes the array element initializations to make sure they don't overlap. This has been an issue in ifort since it first appeared in 2004 - CVF didn't have this issue.

 
andrew_4619
Valued Contributor III
218 Views

Steve do the use of array constructors get around that problem of slow overlap checking?

Ulrich_M_
New Contributor I
204 Views

I find it useful to turn off optimization in modules that are not run-time sensitive to reduce compile times. Just try and put

!!DIR$ NOOPTIMIZE

at the beginning of the file that contains all the data assignments.

Steve_Lionel
Black Belt Retired Employee
191 Views

@andrew_4619 they might well - I don't think I tried it. It would be a lot of work to convert, though. The problematic sources have code that assigns a single value to individual elements - hundreds or thousands of times. 

avinashs
New Contributor I
165 Views

Thanks for the useful and informative responses. It seems that the long compilation time is an artifact of the IVF compiler. I tried all the suggestions but they made minimal difference. Decreasing the number of lines to 500 reduced the compilation time from 8 to 2 minutes. The compiler directive NOOPTIMIZE did not seem to make any difference at all. Reading from a data file is not an option since I want to be able to distribute the data as a DLL without have to resort to knowing the users' directory structure. Further, I am storing only the nonzero elements of the data structure so reading them will require a complex procedure to determine the nonzero elements or have the subroutine doing the reading at run time be autogenerated while writing the data file for reading. Compared to these complications, it seems that longer compilation time is acceptable. 

FortranFan
Honored Contributor I
138 Views

@avinashs ,

Can you please post a small illustration of your subroutine with some assignments with the implication the compilation time will worsen as the number of lines increase?  You can use dummy values as you see fit.  Thanks,

avinashs
New Contributor I
130 Views

@FortranFan - Thank-you for your interest and help. Attached is a file with 16424 lines. The data portion is a good though not exact representation of my original file, which also has some additional executable statements to process the data.

gfortran took 5.56 seconds of stopwatch time to compile this file.

Intel® Fortran Compiler Classic 2021.1.1 [IA-32] took 6.52 minutes, comparable to my original file, and 74 times slower than gfortran. 

If of interest, I can generate files such as the above of varying lengths quite easily. My original file has since grown to 23000 lines, which takes 8-9 minutes with Intel Fortran.

jimdempseyatthecove
Black Belt
116 Views

 

C:\test>echo !time!
 7:13:32.51

C:\test>ifort /c long_comp_time_16424.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 19.1.0.166 Build 20191121
Copyright (C) 1985-2019 Intel Corporation.  All rights reserved.


C:\test>echo !time!
 7:13:52.67

 

20.16 seconds

And 32-bit compiler time about the same:

C:\test>echo !time!
 7:17:03.47

C:\test>ifort /c long_comp_time_16424.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on IA-32, Version 19.1.0.166 Build 20191121
Copyright (C) 1985-2019 Intel Corporation.  All rights reserved.


C:\test>echo !time!
 7:17:23.57

Jim Dempsey

 

Ulrich_M_
New Contributor I
101 Views

The !DIR$ nooptimize directive makes a massive difference on my machine. It has to be put just after the subroutine declaration,

subroutine long_compilation_time(DB, info)
!DIR$ nooptimize
  implicit none

  type :: my_var
.
.
.

Also see here

https://software.intel.com/content/www/us/en/develop/documentation/fortran-compiler-oneapi-dev-guide...

 

FortranFan
Honored Contributor I
92 Views

@avinashs ,

As shown by @jimdempseyatthecove , for me too the compilation time with default options is under 30 seconds for your code attachment with ~16,000 lines.  But it increases to about 2:30 minutes with "/O3" optimization option.   This is on a  i7-6820HCQ 2.7 GHz, 16 GB RAM laptop device. 

FortranFan
Honored Contributor I
88 Views

@avinashs ,

By the way, from where do you receive the values that go into the assignments?  Are they output from some other (sub)program, or some instrumented device like in a lab or a plant or some such?  Is this subroutine with the long list of assignments auto-generated?

Instead of using assignments in Fortran code, have you considered getting those list of values into  CHARACTER variable(s) or named constant(s) via whatever batch or (semi)automated processes you may have?  That is, have them ultimately on the Fortran side as an 'internal file` in Fortran parlance?  And then get the values assigned to your derived type components via 'internal file' IO in a Fortran READ instruction?

With suitable attention to this via such Fortran language options toward such serialization and deserialization (https://en.wikipedia.org/wiki/Serialization) , you may get your whole process of such data exchange streamlined.  The long compilation time will become an irrelevant issue for you, an added bonus!

avinashs
New Contributor I
73 Views

From the responses, it is clear that there is a problem with the Intel compiler on my computer since it is taking an inordinately long time regardless of the options. However, not necessarily a problem with the computer itself since gfortran compiles it very quickly as expected.

@FortranFan I have not investigated other options since this is merely a compilation time issue - there are no issues at run time (which would have been more serious). The original subroutine is auto-generated by another Fortran subroutine that reads the raw data files. The components of the structure that are zero are not stored in the file to reduce the size.

GVautier
New Contributor II
63 Views

Could it be related to memory shortage and disk swap?

Steve_Lionel
Black Belt Retired Employee
52 Views

No, it's an inefficient algorithm in the compiler "back end". It's been known for at least a decade, but the team that maintains that part of the compiler (not the Fortran team) hasn't shown an interest in resolving it.

Ulrich_M_
New Contributor I
34 Views

Are any files on network drives involved in your ifort compilation? In my experience, this can slow things down considerably. If so, put the entire project and output files on the local hard disk and check whether this helps...