Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

Equivalence statements and compile times

David_P_1
Beginner
1,100 Views

Hello,

I'm somewhat new to Fortran and I had several questions. I'm currently trying to integrate a generated Fortran codebase to operate off a segment of memory passed to it from a C++ environment. One of my goals is to make as little change to the Fortran codebase as possible. I've been able to get several small scale examples of this method working fine with the C_PTR type with a derived type and also passing a byte array, but the problems arise with the compile times once I use the full size of the memory to be shared. The amount of memory I'm trying to share is about 15MB in size with about 800,000 variables of different types mapped against it. I'm pretty sure my approach of mapping these variables to be compatible with the generated codebase is the cause of the long compile times which last for days. I also imagine I'm not using the best practices for this problem.

For my first method, I created a large derived type containing all the variables that is passed through an exposed library function into the Fortran library from an identically defined struct in the C++ environment. To make that derived type compatible with the generated Fortran codebase I created a file that is #include'd so that the preprocessor replaces variables names that are #define'd to refer to a member of the large shared derived type. This compilation lasts for days and I never saw it to completion.

For my second method I defined a large byte array that's defined in a module along with an equivalence statement for every variable contained in the library to be mapped against. I pass the byte array from C++ through an exposed library method, and then the passed array is copied to the global array, and once all operations are finished in that subroutine the global module array is copied back to the subroutine's passed array so that changes are reflected in the passed C++ byte array. This compilation also lasts days. I tried to reduce the compile times by using the multiprocess compiler option and breaking the array into 5 separate arrays but that doesn't seem to have much of an impact. I'm also concerned about the runtime efficiency of this method since every time interact with the Fortran library a large array copy takes place instead of operating off a reference to the memory in the previous method.

To reduce compilation times, mostly in the second method, I've tried using equivalence statements in the Fortran source files only for the variables contained but this isn't allowed due to restrictions on using equivalence outside of the global array module definition. To get around that I've tried defining the passed array as COMMON but it seems like using common on subroutine arguments isn't allowed either. I could define the define a common array which is used in every source file, with equivalence statements only as needed, but I would still have to copy that array back and forth like described earlier, which worries me.

There might be an obvious solution to all this, but I am pretty new and some of the documentation/examples relating to this use case are a little sparse. I can provide some example code to show what I'm trying to do at a small scale if that might help with any suggestions.

Thanks,

David

 

0 Kudos
37 Replies
FortranFan
Honored Contributor II
801 Views

@David P.

Code written per any of your descriptions will compile rather quickly, in a matter of minutes if not seconds.

Long compile times you are seeing or never finishing the compilation are indicative of some systems-related issue such as license file problems or anti-virus software, etc. and nothing to do with your Fortran and C++ code.

Hopefully you will find quick help, more details you can provide the faster it will be.

0 Kudos
jimdempseyatthecove
Honored Contributor III
801 Views

>>I'm currently trying to integrate a generated Fortran codebase... with the generated codebase is the cause of the long compile times which last for days

The problem may not be related to the mapping of the C++ data to the Fortran code, but rather a problem with auto-generated Fortran code that looks something like:

result = { humongous expression here
      & here  
      & here
      ...
      & here }

There are other threads on this forum that discuss this subject.

Jim Dempsey

0 Kudos
David_P_1
Beginner
801 Views

Thanks for the responses,

FortranFan,

I don't have any anti-virus running on this system, so I don't think that's the problem. Is there an easy way to diagnose if it might be a licensing issue? The system I'm compiling on is completely offline.

Jim,

There's no complex expressions taking place specifically in the code that's being compiled. The generated code just contains calls to subroutines defined in another library where the actual computations take place using the variables that are contained in the shared memory, and some basic assignment statements.  

I guess I should also mention that I'm using the Intel Visual Fortran Compiler XE 15.0.7.287

I'm going to try a couple more configurations to try to debug the issue, and then I might be able to post a generic example of my code so that it's clear exactly what I'm doing.

0 Kudos
jimdempseyatthecove
Honored Contributor III
801 Views

Because you are using the preprocessor, you should be aware that fpp defaults to honoring C and C++ style comments. C/C++ comment sequence // is also the Fortran character concatenation operator. To disable // as comment in fpp

/B                  Do not recognize C++ style comments.
/C                  Do not recognize C style comments.

Other potential pitfalls

Difference between "include" and "#include", and an editing mistake that leads to infinite recursion of include/#include.

Jim Dempsey

0 Kudos
andrew_4619
Honored Contributor II
801 Views

I have found in the past the really long code blocks e.g. very subroutines slows things down as a guess the compiler has to make multiple passes over the whole subroutine. I have also found in the past that 3 files of say 10,000 lines of code compiles much faster than 1 file of 30,000 lines. I guess in some compile opperations where you have a block of code with N things you make searches that are maybe related to N*N in size or worse higher order so it is very non-linear. I guess you need to generate some smaller cases to  experiment if possible.

.

0 Kudos
David_P_1
Beginner
801 Views

Jim,

Thanks for the suggestion, however at the moment I've been mainly trying to build the second method I posted about which doesn't use the pre-processor, as I was previously thinking that it was possible that the pre-processor was causing the long build times, but I no longer think that's the case.

Andrew,

I was thinking that this might be the case, that I simply have too many variable declarations and equivalence statements in one file, which I'm hoping isn't the case. I have already tried cutting my one global shared array down into 5 separate arrays and I'm not sure how successful I can be getting them to be much smaller.

I was going to upload an example of one of the array files I'm using from my second detailed method that uses a large number of equivalence statements but I keep getting errors on upload, maybe the file is too large (13MB)? Below is essentially what I'm working with at the moment, and just this one source file has been compiling over an hour now. I realize the variable declarations aren't aligned in the correct order but I didn't think that would negatively impact compile times (I'm ignoring that error for the moment).

MODULE example_arr_module
USE, INTRINSIC :: ISO_C_BINDING
	IMPLICIT NONE
	byte, dimension(3758471) :: example_arr
	logical*1 var99245
	equivalence(var99245, example_arr(1))
	logical*1 var99246
	equivalence(var99246, example_arr(2))
	real*8 var99247
	equivalence(var99247, example_arr(3))
	real*8 var99248
	equivalence(var99248, example_arr(11))
	real*8 var99249
	equivalence(var99249, example_arr(19))
!.
!.
! a bunch more of these sort of declarations
!.
!.
	logical*1 var830326
	equivalence(var830326, example_arr(3758415))
	logical*1 var830327
	equivalence(var830327, example_arr(3758416))
	integer*4 var830328
	equivalence(var830328, example_arr(3758417))
	integer*4 var830329
	equivalence(var830329, example_arr(3758421))
	real*8 var830330
	equivalence(var830330, example_arr(3758425))
	logical*1 var830331
	equivalence(var830331, example_arr(3758433))
	logical*1 var830332
	equivalence(var830332, example_arr(3758434))
	logical*1 var830333
	equivalence(var830333, example_arr(3758435))
	integer*4 var830334
	equivalence(var830334, example_arr(3758436))
	integer*4 var830335
	equivalence(var830335, example_arr(3758440))
	real*8 var830336
	equivalence(var830336, example_arr(3758444))
	logical*1 var830337
	equivalence(var830337, example_arr(3758452))
	logical*1 var830338
	equivalence(var830338, example_arr(3758453))
	logical*1 var830339
	equivalence(var830339, example_arr(3758454))
	integer*4 var830340
	equivalence(var830340, example_arr(3758455))
	integer*4 var830341
	equivalence(var830341, example_arr(3758459))
	real*8 var830342
	equivalence(var830342, example_arr(3758463))
end module example_arr_module

 

0 Kudos
andrew_4619
Honored Contributor II
801 Views

What about trying

MODULE mod1
        IMPLICIT NONE
        byte, dimension(3758471) :: example_arr
        logical*1 var99245
        logical*1 var99246
        real*8 var99247
        .....
        .....
END MODULE mod1

MODULE mod2
        USE mod1
        IMPLICIT NONE
        equivalence(var99245, example_arr(1))
        equivalence(var99246, example_arr(2))
        equivalence(var99247, example_arr(3))
        .......
END MODULE mod2

With 2 files. Extending that concept you could break that into a larger number of nested modules.

Also: logical(1) :: var99245 form I think is 'easier' for the compiler to parse having written a parser once .... It probably makes no odds to you aas the source is auto generated.

 

0 Kudos
IanH
Honored Contributor II
801 Views

andrew_4619 wrote:

What about trying

MODULE mod1
        IMPLICIT NONE
        byte, dimension(3758471) :: example_arr
        logical*1 var99245
        logical*1 var99246
        real*8 var99247
        .....
        .....
END MODULE mod1

MODULE mod2
        USE mod1
        IMPLICIT NONE
        equivalence(var99245, example_arr(1))
        equivalence(var99246, example_arr(2))
        equivalence(var99247, example_arr(3))
        .......
END MODULE mod2

That shouldn't fly - "The name of an equivalence-object shall not be a name made accessible by use association."  At a basic level, the compiler has to know how to layout the module variables when it compiles the module.

0 Kudos
andrew_4619
Honored Contributor II
801 Views

That's a fair cop guv. Not used an equivalence since 1988....

This should work....

MODULE mod1
    IMPLICIT NONE
    integer(1)          :: example_arr(100)
    logical(1), pointer :: var99245
    logical(1), pointer :: var99246
    real(8), pointer    :: var99247
END MODULE mod1

MODULE mod2
    USE mod1
    use ISO_C_BINDING
    IMPLICIT NONE
    contains
    subroutine var_init()
        CALL C_F_POINTER(c_loc(example_arr(1)), var99245)
        CALL C_F_POINTER(c_loc(example_arr(2)), var99246)
        CALL C_F_POINTER(c_loc(example_arr(3)), var99247)
    end subroutine var_init
END MODULE mod2   
program main
      use mod2
      implicit none
      example_arr = 0_1
      call var_init()
      write(*,'(Z0)') example_arr(1:10)      
      var99245 = .true._1
      var99246 = .false._1
      var99247 = 99.123_8
      write(*,'(Z0)') example_arr(1:10) 
end program main

 

0 Kudos
andrew_4619
Honored Contributor II
801 Views

That would allow mod1 to be made up of mod1_001, mod1_002,....

And indeed we could have var_init_001, var_init_002, var_init_003,.... to allow the code to be in more digestable portions...

Always best to eat an elephant one byte at a time.

0 Kudos
mecej4
Honored Contributor III
801 Views

From the Ifort 17 language reference manual for C_F_Pointer():

If the value of cptr is the C address of a Fortran variable, it must have the TARGET attribute.

To comply with this, Line-3 of the code in #10 must specify the TARGET attribute for example_arr. The compiler appears unable to check this requirement, even with /check:all /stand:f03

Once you specify TARGET you have to accept its negative effects on optimization.

The OP's code probably contains equivalence object lists that do not satisfy the restrictions of modern Fortran (esp. kind and type conformity). Converting those using Fortran pointers will probably be difficult, if not impossible. Your example program of #10, for instance, is not portable because LOGICAL and INTEGER variables are, in effect, equivalenced.

0 Kudos
jimdempseyatthecove
Honored Contributor III
801 Views

Why not use (try):

subroutine takeStructFromC(Cstruct)
use, intrinsic :: ISO_C_BINDING
implicit none
type, BIND(C) :: Cstruct_t
   ! optional SEQUENCE here
   ... ! define the data types here
end type Cstruct_t
type(Cstruct_t) :: Cstruct
... code
ASSOCIATE(var99247=>Cstruct%someVar)
     .or.
#define var99247 Cstruct%someVar
     .or.
call someProcedure(Cstruct%someVar)
   ...
subroutine someProcedure(var99247))
   implicit none
   real :: var99247
   ...

Jim Dempsey
  

0 Kudos
Lorri_M_Intel
Employee
801 Views

Is there a compile-time initializer anywhere in this?   That is, you have this huge array; do you have an initial value declared for any element of it?

                 --Lorri

0 Kudos
andrew_4619
Honored Contributor II
801 Views

mecej4 wrote:

From the Ifort 17 language reference manual for C_F_Pointer():

If the value of cptr is the C address of a Fortran variable, it must have the TARGET attribute.

Yes I had standard checking and that does not give any warnings, maybe that is compiler bug?. I think the issue of portability is probably not that relevant as the whole concept relies on the auto-code generator understanding the storage sizes on the target system etc. I think the author is not so worried about optimistation as he is not at the first base getting a code that compiles!!!  The code I attached does work  I make no claims as to the legality or efficiency.....

 

0 Kudos
andrew_4619
Honored Contributor II
801 Views

jimdempseyatthecove wrote:

Why not use (try):

subroutine takeStructFromC(Cstruct)
use, intrinsic :: ISO_C_BINDING
implicit none
type, BIND(C) :: Cstruct_t
   ! optional SEQUENCE here
   ... ! define the data types here
end type Cstruct_t
type(Cstruct_t) :: Cstruct
... code
ASSOCIATE(var99247=>Cstruct%someVar)
     .or.
#define var99247 Cstruct%someVar
     .or.
call someProcedure(Cstruct%someVar)
   ...
subroutine someProcedure(var99247))
   implicit none
   real :: var99247
   ...

Jim Dempsey
  

 

I think the OP tried something like that, the compiler chokes compiling the structure due to its size I think....

0 Kudos
David_P_1
Beginner
801 Views

Thanks again for all the suggestions everyone,

I may not be able to access the system where I'm doing testing today so I may not be able to provide the best info.

Andrew,

I think cutting what I have down in size even further would probably help with the compile times, I'll have to do some testing and edit the scripts that I'm using to generate this code from the other generated Fortran code, lol. You are correct in that my main issue here is getting something to compile in a reasonable amount of time.

Jim,

I think this is similar to my first method described in the initial post, where I created a huge 800,000+ member derived type inside of a module. From what I remember this file itself took hours to compile. An example of that is:

MODULE exampleType
USE, INTRINSIC :: ISO_C_BINDING
    IMPLICIT NONE
    TYPE :: example
		real(C_DOUBLE) var1
		real(C_DOUBLE) var2
		real(C_DOUBLE) var3
		real(C_DOUBLE) var4
!
! lots more member definitions
!
		real(C_BOOL) var800000
		real(C_BOOL) var800001
		real(C_BOOL) var800002
    END TYPE example
END MODULE exampleType

I also created another module that declared this type so that I could use global reference this type in any file that used that module. This way also included a file that #define'd references to var1 within the generated source as defined_example%var1, where defined_example is the derived type declaration inside the previously mentioned module. Sorry if that's not what you were suggesting. Both of these methods work well on a small scale.

Lorri,

I don't have any initial values for any of the array members declared to any value yet. That is something that I plan on doing but I first wanted to do some basic testing. Do you think that could affect compile times?

Thanks again everybody.

0 Kudos
jimdempseyatthecove
Honored Contributor III
801 Views

Why not:

real(C_DOUBLE) :: var(800002)

Then use AWK, or other favorite editing tool to search the auto-generated Fortran file for varnnn and replace with var(nnn). IOW parenthesize the sequence number?

Note, if you are using nmake, you can easily make a rule, If you are using MS VS, you can define a pre-build procedure.

Jim Dempsey

0 Kudos
FortranFan
Honored Contributor II
801 Views

David P. wrote:

..  I'm currently trying to integrate a generated Fortran codebase to operate off a segment of memory passed to it from a C++ environment. One of my goals is to make as little change to the Fortran codebase as possible ..

@David P.,

You state in the original post, "One of my goals is to make as little change to the Fortran  codebase  as  possible .." - what does "as little" mean?  Are you indicating absolutely no change, or if named COMMON blocks - implying global data - are present, then can you add BIND(C) bindings to such blocks?

See this:

subroutine foo( s, lens .. )
! Say this represents some "generated" FORTRAN (legacy-style) code
! that makes use of global data and processes dummy arguments like so

   ! Argument list
   character(len=*) s
   integer          lens
   ..

   ! Local variables
   ..

   ! Global variables
   integer i_foo
   real r_foo
   common / foodata / r_foo, i_foo
   bind(C, name="foodata") / foodata / !<-- can such bindings be added?

   ..
   s(1:lens) = "Hello World!"
   ..

   return

end subroutine foo

 

See the question on line 17 above.  Can you even make such a change?  But before you answer this, please confirm if you have such global data to begin with and if so, what does your "generated" Fortran code, the ones that do the actual work, look like i.e., what type of data are involved?

If so, you can interoperate all your global data in the Fortran code with "extern C" structs in C++ code as well with the use of a stream of bytes to provide the memory required for the data in Fortran subprogram dummy arguments (like s above).  You will just invoke a data broker layer that sets things up for the Fortran calculations and this should compile quickly.

0 Kudos
Steve_Lionel
Honored Contributor III
801 Views

The Intel compiler has compile time issues with very large arrays that are compile-time (DATA)  initialized. I don't recall seeing any compile-time issues related to EQUIVALENCE.

0 Kudos
andrew_4619
Honored Contributor II
697 Views

for amusement I made the small program below that creates a module with a number (LOTS) of random variables of various types. I then compiled the resultant file. With LOTS=10000 it takes a few seconds to compile. With LOTS=100000 I don't know how long it takes because it has been working away the last 20 minutes..... It seems we are paging windows maybe as CPU is on 30% but there is quite a bit of disk access going on and memory use is creeping up and up

program make_lots_of_vars
    implicit none
    integer, parameter :: lots = 100000
    integer, parameter :: ntypes = 6
    character(len=*), parameter :: gfmt_txt='(A)' 
    character(len=*), parameter :: gfmt_var='(A,i0)' 
    character(len=*), parameter :: gpd='   ' 
    integer :: iun, istat,l1, itype
    real    :: rty
    
    CALL RANDOM_SEED()    ! seeds using time/date
    open(newunit=iun, file='lots.f90', status='unknown', iostat = istat)
    if (istat /= 0 ) stop 'open fail'
    
    write(iun,gfmt_txt) 'module stuff'
    write(iun,gfmt_txt) gpd//'implicit none'
    
    do l1 = 1 , lots
       CALL RANDOM_NUMBER(rty)
       rty = rty * real(ntypes) 
       itype = int( rty ) + 1
       selectcase (itype)
           case(1)
              write(iun,gfmt_var) gpd//'Logical(1) :: var',l1  
           case(2)
              write(iun,gfmt_var) gpd//'Logical(4) :: var',l1
           case(3)
              write(iun,gfmt_var) gpd//'real(4) :: var',l1 
           case(4)
              write(iun,gfmt_var) gpd//'real(8) :: var',l1 
           case(5)
              write(iun,gfmt_var) gpd//'integer(4) :: var',l1 
           case(6)
              write(iun,gfmt_var) gpd//'character(len=20) :: var',l1  
           case default
               write(iun,gfmt_var) gpd//'real(4) :: var',l1 
           end select
    enddo
    write(iun,gfmt_txt) 'end module stuff'   
    !write(iun,gfmt_txt) 'program use_stuff' 
    !write(iun,gfmt_txt) gpd//'use stuff'    
    !write(iun,gfmt_txt) gpd//'implicit none'
    !write(iun,gfmt_txt) 'end program use_stuff'
    close(iun)   
end program make_lots_of_vars

 

0 Kudos
Reply