Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.
28381 Discussions

Long compilation time for a subroutine containing data assignments

avinashs
New Contributor I
1,790 Views

I have a subroutine that contains about 15,000 lines of code. Except for the declaration statements, the rest of the code assigns data values to the various components of a user-defined type. During compilation, this subroutine takes several minutes (about 6-7 minutes) to compile each time it is newly generated. While it has to be compiled infrequently, I am curious as to why it takes so much time and whether there is a better way to store/assign the data. With each version of Intel Fortran, including the current Intel OneAPI, the time required for compilation keeps increasing.

The compiler settings in Intel Fortran are

/nologo /O3 /assume:norealloc_lhs /real-size:64 /fpconstant /fp:consistent /gen-interfaces /iface:cref /libs:dll /threads /Qm64 /c

With the same settings that are common to both compilers, gfortran takes about 4-5 seconds to compile the same file.

0 Kudos
17 Replies
andrew_4619
Honored Contributor II
1,777 Views

A couple of comments. I found some years back (I have not tested recently) that above maybe around 15000 lines in a source file and the compile time increased quite a lot in a very non-linear way. I split my code so that no files were bigger than around 10000 lines and compile times for the full project dropped significantly. One could think of so many reasons why algorithms might cause such an effect.   

I also recall a thread some years back where a guy had a huge number of data statements and that was killing the compiler also. You could look for that one.  Split the source file if that is easy to do or read the data at run time from a binary file and take the time hit at run time.

 

0 Kudos
Ron_Green
Moderator
1,758 Views

I opened a bug report on this for IFX Beta.  But even IFORT has been slow on this initialization.  I saw this in VASP - what code are you working with? 

One could argue that when you have this much data to initialize it's best to use an input file rather than explicit initializations.  One could counter argue that having the initialization and values in initialization statements saves problems with keeping a separate data file and avoid a possible error on file opening and reading.

IFX is really bad at this initialization.  Ifort only so-so.  I had one file that took IFX about 20minutes to compile.  It has somewhere around 50,000 initialization statements, 1 per line.  I thought the compiler hung.  That's the one that I opened the bug report against.  They made SOME improvements but it's still a work in progress.  gfortran is pretty fast at it, by comparison.  I'll go check on that bug.

POSTSCRIPT: I've added this report to the bug report, which is in progress.

0 Kudos
Steve_Lionel
Honored Contributor III
1,740 Views

ifort has traditionally had a hard time with code that has a lot of DATA initializations. This has to do with the way the code generator analyzes the array element initializations to make sure they don't overlap. This has been an issue in ifort since it first appeared in 2004 - CVF didn't have this issue.

 
0 Kudos
andrew_4619
Honored Contributor II
1,725 Views

Steve do the use of array constructors get around that problem of slow overlap checking?

0 Kudos
Steve_Lionel
Honored Contributor III
1,698 Views

@andrew_4619 they might well - I don't think I tried it. It would be a lot of work to convert, though. The problematic sources have code that assigns a single value to individual elements - hundreds or thousands of times. 

0 Kudos
Ulrich_M_
New Contributor I
1,711 Views

I find it useful to turn off optimization in modules that are not run-time sensitive to reduce compile times. Just try and put

!!DIR$ NOOPTIMIZE

at the beginning of the file that contains all the data assignments.

0 Kudos
avinashs
New Contributor I
1,672 Views

Thanks for the useful and informative responses. It seems that the long compilation time is an artifact of the IVF compiler. I tried all the suggestions but they made minimal difference. Decreasing the number of lines to 500 reduced the compilation time from 8 to 2 minutes. The compiler directive NOOPTIMIZE did not seem to make any difference at all. Reading from a data file is not an option since I want to be able to distribute the data as a DLL without have to resort to knowing the users' directory structure. Further, I am storing only the nonzero elements of the data structure so reading them will require a complex procedure to determine the nonzero elements or have the subroutine doing the reading at run time be autogenerated while writing the data file for reading. Compared to these complications, it seems that longer compilation time is acceptable. 

0 Kudos
FortranFan
Honored Contributor II
1,645 Views

@avinashs ,

Can you please post a small illustration of your subroutine with some assignments with the implication the compilation time will worsen as the number of lines increase?  You can use dummy values as you see fit.  Thanks,

0 Kudos
avinashs
New Contributor I
1,637 Views

@FortranFan - Thank-you for your interest and help. Attached is a file with 16424 lines. The data portion is a good though not exact representation of my original file, which also has some additional executable statements to process the data.

gfortran took 5.56 seconds of stopwatch time to compile this file.

Intel® Fortran Compiler Classic 2021.1.1 [IA-32] took 6.52 minutes, comparable to my original file, and 74 times slower than gfortran. 

If of interest, I can generate files such as the above of varying lengths quite easily. My original file has since grown to 23000 lines, which takes 8-9 minutes with Intel Fortran.

0 Kudos
jimdempseyatthecove
Honored Contributor III
1,623 Views

 

C:\test>echo !time!
 7:13:32.51

C:\test>ifort /c long_comp_time_16424.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 19.1.0.166 Build 20191121
Copyright (C) 1985-2019 Intel Corporation.  All rights reserved.


C:\test>echo !time!
 7:13:52.67

 

20.16 seconds

And 32-bit compiler time about the same:

C:\test>echo !time!
 7:17:03.47

C:\test>ifort /c long_comp_time_16424.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on IA-32, Version 19.1.0.166 Build 20191121
Copyright (C) 1985-2019 Intel Corporation.  All rights reserved.


C:\test>echo !time!
 7:17:23.57

Jim Dempsey

 

0 Kudos
Ulrich_M_
New Contributor I
1,608 Views

The !DIR$ nooptimize directive makes a massive difference on my machine. It has to be put just after the subroutine declaration,

subroutine long_compilation_time(DB, info)
!DIR$ nooptimize
  implicit none

  type :: my_var
.
.
.

Also see here

https://software.intel.com/content/www/us/en/develop/documentation/fortran-compiler-oneapi-dev-guide-and-reference/top/language-reference/a-to-z-reference/o-to-p/optimize-and-nooptimize.html

 

0 Kudos
FortranFan
Honored Contributor II
1,599 Views

@avinashs ,

As shown by @jimdempseyatthecove , for me too the compilation time with default options is under 30 seconds for your code attachment with ~16,000 lines.  But it increases to about 2:30 minutes with "/O3" optimization option.   This is on a  i7-6820HCQ 2.7 GHz, 16 GB RAM laptop device. 

0 Kudos
FortranFan
Honored Contributor II
1,595 Views

@avinashs ,

By the way, from where do you receive the values that go into the assignments?  Are they output from some other (sub)program, or some instrumented device like in a lab or a plant or some such?  Is this subroutine with the long list of assignments auto-generated?

Instead of using assignments in Fortran code, have you considered getting those list of values into  CHARACTER variable(s) or named constant(s) via whatever batch or (semi)automated processes you may have?  That is, have them ultimately on the Fortran side as an 'internal file` in Fortran parlance?  And then get the values assigned to your derived type components via 'internal file' IO in a Fortran READ instruction?

With suitable attention to this via such Fortran language options toward such serialization and deserialization (https://en.wikipedia.org/wiki/Serialization) , you may get your whole process of such data exchange streamlined.  The long compilation time will become an irrelevant issue for you, an added bonus!

0 Kudos
avinashs
New Contributor I
1,580 Views

From the responses, it is clear that there is a problem with the Intel compiler on my computer since it is taking an inordinately long time regardless of the options. However, not necessarily a problem with the computer itself since gfortran compiles it very quickly as expected.

@FortranFan I have not investigated other options since this is merely a compilation time issue - there are no issues at run time (which would have been more serious). The original subroutine is auto-generated by another Fortran subroutine that reads the raw data files. The components of the structure that are zero are not stored in the file to reduce the size.

0 Kudos
GVautier
New Contributor II
1,570 Views

Could it be related to memory shortage and disk swap?

0 Kudos
Steve_Lionel
Honored Contributor III
1,559 Views

No, it's an inefficient algorithm in the compiler "back end". It's been known for at least a decade, but the team that maintains that part of the compiler (not the Fortran team) hasn't shown an interest in resolving it.

0 Kudos
Ulrich_M_
New Contributor I
1,541 Views

Are any files on network drives involved in your ifort compilation? In my experience, this can slow things down considerably. If so, put the entire project and output files on the local hard disk and check whether this helps...

0 Kudos
Reply