I have a subroutine that contains about 15,000 lines of code. Except for the declaration statements, the rest of the code assigns data values to the various components of a user-defined type. During compilation, this subroutine takes several minutes (about 6-7 minutes) to compile each time it is newly generated. While it has to be compiled infrequently, I am curious as to why it takes so much time and whether there is a better way to store/assign the data. With each version of Intel Fortran, including the current Intel OneAPI, the time required for compilation keeps increasing.
The compiler settings in Intel Fortran are
/nologo /O3 /assume:norealloc_lhs /real-size:64 /fpconstant /fp:consistent /gen-interfaces /iface:cref /libs:dll /threads /Qm64 /c
With the same settings that are common to both compilers, gfortran takes about 4-5 seconds to compile the same file.
A couple of comments. I found some years back (I have not tested recently) that above maybe around 15000 lines in a source file and the compile time increased quite a lot in a very non-linear way. I split my code so that no files were bigger than around 10000 lines and compile times for the full project dropped significantly. One could think of so many reasons why algorithms might cause such an effect.
I also recall a thread some years back where a guy had a huge number of data statements and that was killing the compiler also. You could look for that one. Split the source file if that is easy to do or read the data at run time from a binary file and take the time hit at run time.
I opened a bug report on this for IFX Beta. But even IFORT has been slow on this initialization. I saw this in VASP - what code are you working with?
One could argue that when you have this much data to initialize it's best to use an input file rather than explicit initializations. One could counter argue that having the initialization and values in initialization statements saves problems with keeping a separate data file and avoid a possible error on file opening and reading.
IFX is really bad at this initialization. Ifort only so-so. I had one file that took IFX about 20minutes to compile. It has somewhere around 50,000 initialization statements, 1 per line. I thought the compiler hung. That's the one that I opened the bug report against. They made SOME improvements but it's still a work in progress. gfortran is pretty fast at it, by comparison. I'll go check on that bug.
POSTSCRIPT: I've added this report to the bug report, which is in progress.
ifort has traditionally had a hard time with code that has a lot of DATA initializations. This has to do with the way the code generator analyzes the array element initializations to make sure they don't overlap. This has been an issue in ifort since it first appeared in 2004 - CVF didn't have this issue.
I find it useful to turn off optimization in modules that are not run-time sensitive to reduce compile times. Just try and put
at the beginning of the file that contains all the data assignments.
@andrew_4619 they might well - I don't think I tried it. It would be a lot of work to convert, though. The problematic sources have code that assigns a single value to individual elements - hundreds or thousands of times.
Thanks for the useful and informative responses. It seems that the long compilation time is an artifact of the IVF compiler. I tried all the suggestions but they made minimal difference. Decreasing the number of lines to 500 reduced the compilation time from 8 to 2 minutes. The compiler directive NOOPTIMIZE did not seem to make any difference at all. Reading from a data file is not an option since I want to be able to distribute the data as a DLL without have to resort to knowing the users' directory structure. Further, I am storing only the nonzero elements of the data structure so reading them will require a complex procedure to determine the nonzero elements or have the subroutine doing the reading at run time be autogenerated while writing the data file for reading. Compared to these complications, it seems that longer compilation time is acceptable.
Can you please post a small illustration of your subroutine with some assignments with the implication the compilation time will worsen as the number of lines increase? You can use dummy values as you see fit. Thanks,
@FortranFan - Thank-you for your interest and help. Attached is a file with 16424 lines. The data portion is a good though not exact representation of my original file, which also has some additional executable statements to process the data.
gfortran took 5.56 seconds of stopwatch time to compile this file.
Intel® Fortran Compiler Classic 2021.1.1 [IA-32] took 6.52 minutes, comparable to my original file, and 74 times slower than gfortran.
If of interest, I can generate files such as the above of varying lengths quite easily. My original file has since grown to 23000 lines, which takes 8-9 minutes with Intel Fortran.
C:\test>echo !time! 7:13:32.51 C:\test>ifort /c long_comp_time_16424.f90 Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 220.127.116.11 Build 20191121 Copyright (C) 1985-2019 Intel Corporation. All rights reserved. C:\test>echo !time! 7:13:52.67
And 32-bit compiler time about the same:
C:\test>echo !time! 7:17:03.47 C:\test>ifort /c long_comp_time_16424.f90 Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on IA-32, Version 18.104.22.168 Build 20191121 Copyright (C) 1985-2019 Intel Corporation. All rights reserved. C:\test>echo !time! 7:17:23.57
The !DIR$ nooptimize directive makes a massive difference on my machine. It has to be put just after the subroutine declaration,
subroutine long_compilation_time(DB, info) !DIR$ nooptimize implicit none type :: my_var . . .
Also see here
By the way, from where do you receive the values that go into the assignments? Are they output from some other (sub)program, or some instrumented device like in a lab or a plant or some such? Is this subroutine with the long list of assignments auto-generated?
Instead of using assignments in Fortran code, have you considered getting those list of values into CHARACTER variable(s) or named constant(s) via whatever batch or (semi)automated processes you may have? That is, have them ultimately on the Fortran side as an 'internal file` in Fortran parlance? And then get the values assigned to your derived type components via 'internal file' IO in a Fortran READ instruction?
With suitable attention to this via such Fortran language options toward such serialization and deserialization (https://en.wikipedia.org/wiki/Serialization) , you may get your whole process of such data exchange streamlined. The long compilation time will become an irrelevant issue for you, an added bonus!
From the responses, it is clear that there is a problem with the Intel compiler on my computer since it is taking an inordinately long time regardless of the options. However, not necessarily a problem with the computer itself since gfortran compiles it very quickly as expected.
@FortranFan I have not investigated other options since this is merely a compilation time issue - there are no issues at run time (which would have been more serious). The original subroutine is auto-generated by another Fortran subroutine that reads the raw data files. The components of the structure that are zero are not stored in the file to reduce the size.
No, it's an inefficient algorithm in the compiler "back end". It's been known for at least a decade, but the team that maintains that part of the compiler (not the Fortran team) hasn't shown an interest in resolving it.
Are any files on network drives involved in your ifort compilation? In my experience, this can slow things down considerably. If so, put the entire project and output files on the local hard disk and check whether this helps...