- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I'm somewhat new to Fortran and I had several questions. I'm currently trying to integrate a generated Fortran codebase to operate off a segment of memory passed to it from a C++ environment. One of my goals is to make as little change to the Fortran codebase as possible. I've been able to get several small scale examples of this method working fine with the C_PTR type with a derived type and also passing a byte array, but the problems arise with the compile times once I use the full size of the memory to be shared. The amount of memory I'm trying to share is about 15MB in size with about 800,000 variables of different types mapped against it. I'm pretty sure my approach of mapping these variables to be compatible with the generated codebase is the cause of the long compile times which last for days. I also imagine I'm not using the best practices for this problem.
For my first method, I created a large derived type containing all the variables that is passed through an exposed library function into the Fortran library from an identically defined struct in the C++ environment. To make that derived type compatible with the generated Fortran codebase I created a file that is #include'd so that the preprocessor replaces variables names that are #define'd to refer to a member of the large shared derived type. This compilation lasts for days and I never saw it to completion.
For my second method I defined a large byte array that's defined in a module along with an equivalence statement for every variable contained in the library to be mapped against. I pass the byte array from C++ through an exposed library method, and then the passed array is copied to the global array, and once all operations are finished in that subroutine the global module array is copied back to the subroutine's passed array so that changes are reflected in the passed C++ byte array. This compilation also lasts days. I tried to reduce the compile times by using the multiprocess compiler option and breaking the array into 5 separate arrays but that doesn't seem to have much of an impact. I'm also concerned about the runtime efficiency of this method since every time interact with the Fortran library a large array copy takes place instead of operating off a reference to the memory in the previous method.
To reduce compilation times, mostly in the second method, I've tried using equivalence statements in the Fortran source files only for the variables contained but this isn't allowed due to restrictions on using equivalence outside of the global array module definition. To get around that I've tried defining the passed array as COMMON but it seems like using common on subroutine arguments isn't allowed either. I could define the define a common array which is used in every source file, with equivalence statements only as needed, but I would still have to copy that array back and forth like described earlier, which worries me.
There might be an obvious solution to all this, but I am pretty new and some of the documentation/examples relating to this use case are a little sparse. I can provide some example code to show what I'm trying to do at a small scale if that might help with any suggestions.
Thanks,
David
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@David P.
Code written per any of your descriptions will compile rather quickly, in a matter of minutes if not seconds.
Long compile times you are seeing or never finishing the compilation are indicative of some systems-related issue such as license file problems or anti-virus software, etc. and nothing to do with your Fortran and C++ code.
Hopefully you will find quick help, more details you can provide the faster it will be.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
>>I'm currently trying to integrate a generated Fortran codebase... with the generated codebase is the cause of the long compile times which last for days
The problem may not be related to the mapping of the C++ data to the Fortran code, but rather a problem with auto-generated Fortran code that looks something like:
result = { humongous expression here
& here
& here
...
& here }
There are other threads on this forum that discuss this subject.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the responses,
FortranFan,
I don't have any anti-virus running on this system, so I don't think that's the problem. Is there an easy way to diagnose if it might be a licensing issue? The system I'm compiling on is completely offline.
Jim,
There's no complex expressions taking place specifically in the code that's being compiled. The generated code just contains calls to subroutines defined in another library where the actual computations take place using the variables that are contained in the shared memory, and some basic assignment statements.
I guess I should also mention that I'm using the Intel Visual Fortran Compiler XE 15.0.7.287
I'm going to try a couple more configurations to try to debug the issue, and then I might be able to post a generic example of my code so that it's clear exactly what I'm doing.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Because you are using the preprocessor, you should be aware that fpp defaults to honoring C and C++ style comments. C/C++ comment sequence // is also the Fortran character concatenation operator. To disable // as comment in fpp
/B Do not recognize C++ style comments.
/C Do not recognize C style comments.
Other potential pitfalls
Difference between "include" and "#include", and an editing mistake that leads to infinite recursion of include/#include.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have found in the past the really long code blocks e.g. very subroutines slows things down as a guess the compiler has to make multiple passes over the whole subroutine. I have also found in the past that 3 files of say 10,000 lines of code compiles much faster than 1 file of 30,000 lines. I guess in some compile opperations where you have a block of code with N things you make searches that are maybe related to N*N in size or worse higher order so it is very non-linear. I guess you need to generate some smaller cases to experiment if possible.
.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Jim,
Thanks for the suggestion, however at the moment I've been mainly trying to build the second method I posted about which doesn't use the pre-processor, as I was previously thinking that it was possible that the pre-processor was causing the long build times, but I no longer think that's the case.
Andrew,
I was thinking that this might be the case, that I simply have too many variable declarations and equivalence statements in one file, which I'm hoping isn't the case. I have already tried cutting my one global shared array down into 5 separate arrays and I'm not sure how successful I can be getting them to be much smaller.
I was going to upload an example of one of the array files I'm using from my second detailed method that uses a large number of equivalence statements but I keep getting errors on upload, maybe the file is too large (13MB)? Below is essentially what I'm working with at the moment, and just this one source file has been compiling over an hour now. I realize the variable declarations aren't aligned in the correct order but I didn't think that would negatively impact compile times (I'm ignoring that error for the moment).
MODULE example_arr_module USE, INTRINSIC :: ISO_C_BINDING IMPLICIT NONE byte, dimension(3758471) :: example_arr logical*1 var99245 equivalence(var99245, example_arr(1)) logical*1 var99246 equivalence(var99246, example_arr(2)) real*8 var99247 equivalence(var99247, example_arr(3)) real*8 var99248 equivalence(var99248, example_arr(11)) real*8 var99249 equivalence(var99249, example_arr(19)) !. !. ! a bunch more of these sort of declarations !. !. logical*1 var830326 equivalence(var830326, example_arr(3758415)) logical*1 var830327 equivalence(var830327, example_arr(3758416)) integer*4 var830328 equivalence(var830328, example_arr(3758417)) integer*4 var830329 equivalence(var830329, example_arr(3758421)) real*8 var830330 equivalence(var830330, example_arr(3758425)) logical*1 var830331 equivalence(var830331, example_arr(3758433)) logical*1 var830332 equivalence(var830332, example_arr(3758434)) logical*1 var830333 equivalence(var830333, example_arr(3758435)) integer*4 var830334 equivalence(var830334, example_arr(3758436)) integer*4 var830335 equivalence(var830335, example_arr(3758440)) real*8 var830336 equivalence(var830336, example_arr(3758444)) logical*1 var830337 equivalence(var830337, example_arr(3758452)) logical*1 var830338 equivalence(var830338, example_arr(3758453)) logical*1 var830339 equivalence(var830339, example_arr(3758454)) integer*4 var830340 equivalence(var830340, example_arr(3758455)) integer*4 var830341 equivalence(var830341, example_arr(3758459)) real*8 var830342 equivalence(var830342, example_arr(3758463)) end module example_arr_module
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What about trying
MODULE mod1 IMPLICIT NONE byte, dimension(3758471) :: example_arr logical*1 var99245 logical*1 var99246 real*8 var99247 ..... ..... END MODULE mod1 MODULE mod2 USE mod1 IMPLICIT NONE equivalence(var99245, example_arr(1)) equivalence(var99246, example_arr(2)) equivalence(var99247, example_arr(3)) ....... END MODULE mod2
With 2 files. Extending that concept you could break that into a larger number of nested modules.
Also: logical(1) :: var99245 form I think is 'easier' for the compiler to parse having written a parser once .... It probably makes no odds to you aas the source is auto generated.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
andrew_4619 wrote:
What about trying
MODULE mod1 IMPLICIT NONE byte, dimension(3758471) :: example_arr logical*1 var99245 logical*1 var99246 real*8 var99247 ..... ..... END MODULE mod1 MODULE mod2 USE mod1 IMPLICIT NONE equivalence(var99245, example_arr(1)) equivalence(var99246, example_arr(2)) equivalence(var99247, example_arr(3)) ....... END MODULE mod2
That shouldn't fly - "The name of an equivalence-object shall not be a name made accessible by use association." At a basic level, the compiler has to know how to layout the module variables when it compiles the module.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That's a fair cop guv. Not used an equivalence since 1988....
This should work....
MODULE mod1 IMPLICIT NONE integer(1) :: example_arr(100) logical(1), pointer :: var99245 logical(1), pointer :: var99246 real(8), pointer :: var99247 END MODULE mod1 MODULE mod2 USE mod1 use ISO_C_BINDING IMPLICIT NONE contains subroutine var_init() CALL C_F_POINTER(c_loc(example_arr(1)), var99245) CALL C_F_POINTER(c_loc(example_arr(2)), var99246) CALL C_F_POINTER(c_loc(example_arr(3)), var99247) end subroutine var_init END MODULE mod2 program main use mod2 implicit none example_arr = 0_1 call var_init() write(*,'(Z0)') example_arr(1:10) var99245 = .true._1 var99246 = .false._1 var99247 = 99.123_8 write(*,'(Z0)') example_arr(1:10) end program main
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That would allow mod1 to be made up of mod1_001, mod1_002,....
And indeed we could have var_init_001, var_init_002, var_init_003,.... to allow the code to be in more digestable portions...
Always best to eat an elephant one byte at a time.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
From the Ifort 17 language reference manual for C_F_Pointer():
If the value of cptr is the C address of a Fortran variable, it must have the TARGET attribute.
To comply with this, Line-3 of the code in #10 must specify the TARGET attribute for example_arr. The compiler appears unable to check this requirement, even with /check:all /stand:f03 .
Once you specify TARGET you have to accept its negative effects on optimization.
The OP's code probably contains equivalence object lists that do not satisfy the restrictions of modern Fortran (esp. kind and type conformity). Converting those using Fortran pointers will probably be difficult, if not impossible. Your example program of #10, for instance, is not portable because LOGICAL and INTEGER variables are, in effect, equivalenced.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why not use (try):
subroutine takeStructFromC(Cstruct) use, intrinsic :: ISO_C_BINDING implicit none type, BIND(C) :: Cstruct_t ! optional SEQUENCE here ... ! define the data types here end type Cstruct_t type(Cstruct_t) :: Cstruct ... code ASSOCIATE(var99247=>Cstruct%someVar) .or. #define var99247 Cstruct%someVar .or. call someProcedure(Cstruct%someVar) ... subroutine someProcedure(var99247)) implicit none real :: var99247 ...
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is there a compile-time initializer anywhere in this? That is, you have this huge array; do you have an initial value declared for any element of it?
--Lorri
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
mecej4 wrote:
From the Ifort 17 language reference manual for C_F_Pointer():
If the value of cptr is the C address of a Fortran variable, it must have the TARGET attribute.
Yes I had standard checking and that does not give any warnings, maybe that is compiler bug?. I think the issue of portability is probably not that relevant as the whole concept relies on the auto-code generator understanding the storage sizes on the target system etc. I think the author is not so worried about optimistation as he is not at the first base getting a code that compiles!!! The code I attached does work I make no claims as to the legality or efficiency.....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
jimdempseyatthecove wrote:
Why not use (try):
subroutine takeStructFromC(Cstruct) use, intrinsic :: ISO_C_BINDING implicit none type, BIND(C) :: Cstruct_t ! optional SEQUENCE here ... ! define the data types here end type Cstruct_t type(Cstruct_t) :: Cstruct ... code ASSOCIATE(var99247=>Cstruct%someVar) .or. #define var99247 Cstruct%someVar .or. call someProcedure(Cstruct%someVar) ... subroutine someProcedure(var99247)) implicit none real :: var99247 ...Jim Dempsey
I think the OP tried something like that, the compiler chokes compiling the structure due to its size I think....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks again for all the suggestions everyone,
I may not be able to access the system where I'm doing testing today so I may not be able to provide the best info.
Andrew,
I think cutting what I have down in size even further would probably help with the compile times, I'll have to do some testing and edit the scripts that I'm using to generate this code from the other generated Fortran code, lol. You are correct in that my main issue here is getting something to compile in a reasonable amount of time.
Jim,
I think this is similar to my first method described in the initial post, where I created a huge 800,000+ member derived type inside of a module. From what I remember this file itself took hours to compile. An example of that is:
MODULE exampleType USE, INTRINSIC :: ISO_C_BINDING IMPLICIT NONE TYPE :: example real(C_DOUBLE) var1 real(C_DOUBLE) var2 real(C_DOUBLE) var3 real(C_DOUBLE) var4 ! ! lots more member definitions ! real(C_BOOL) var800000 real(C_BOOL) var800001 real(C_BOOL) var800002 END TYPE example END MODULE exampleType
I also created another module that declared this type so that I could use global reference this type in any file that used that module. This way also included a file that #define'd references to var1 within the generated source as defined_example%var1, where defined_example is the derived type declaration inside the previously mentioned module. Sorry if that's not what you were suggesting. Both of these methods work well on a small scale.
Lorri,
I don't have any initial values for any of the array members declared to any value yet. That is something that I plan on doing but I first wanted to do some basic testing. Do you think that could affect compile times?
Thanks again everybody.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why not:
real(C_DOUBLE) :: var(800002)
Then use AWK, or other favorite editing tool to search the auto-generated Fortran file for varnnn and replace with var(nnn). IOW parenthesize the sequence number?
Note, if you are using nmake, you can easily make a rule, If you are using MS VS, you can define a pre-build procedure.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
David P. wrote:
.. I'm currently trying to integrate a generated Fortran codebase to operate off a segment of memory passed to it from a C++ environment. One of my goals is to make as little change to the Fortran codebase as possible ..
@David P.,
You state in the original post, "One of my goals is to make as little change to the Fortran codebase as possible .." - what does "as little" mean? Are you indicating absolutely no change, or if named COMMON blocks - implying global data - are present, then can you add BIND(C) bindings to such blocks?
See this:
subroutine foo( s, lens .. ) ! Say this represents some "generated" FORTRAN (legacy-style) code ! that makes use of global data and processes dummy arguments like so ! Argument list character(len=*) s integer lens .. ! Local variables .. ! Global variables integer i_foo real r_foo common / foodata / r_foo, i_foo bind(C, name="foodata") / foodata / !<-- can such bindings be added? .. s(1:lens) = "Hello World!" .. return end subroutine foo
See the question on line 17 above. Can you even make such a change? But before you answer this, please confirm if you have such global data to begin with and if so, what does your "generated" Fortran code, the ones that do the actual work, look like i.e., what type of data are involved?
If so, you can interoperate all your global data in the Fortran code with "extern C" structs in C++ code as well with the use of a stream of bytes to provide the memory required for the data in Fortran subprogram dummy arguments (like s above). You will just invoke a data broker layer that sets things up for the Fortran calculations and this should compile quickly.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The Intel compiler has compile time issues with very large arrays that are compile-time (DATA) initialized. I don't recall seeing any compile-time issues related to EQUIVALENCE.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
for amusement I made the small program below that creates a module with a number (LOTS) of random variables of various types. I then compiled the resultant file. With LOTS=10000 it takes a few seconds to compile. With LOTS=100000 I don't know how long it takes because it has been working away the last 20 minutes..... It seems we are paging windows maybe as CPU is on 30% but there is quite a bit of disk access going on and memory use is creeping up and up
program make_lots_of_vars implicit none integer, parameter :: lots = 100000 integer, parameter :: ntypes = 6 character(len=*), parameter :: gfmt_txt='(A)' character(len=*), parameter :: gfmt_var='(A,i0)' character(len=*), parameter :: gpd=' ' integer :: iun, istat,l1, itype real :: rty CALL RANDOM_SEED() ! seeds using time/date open(newunit=iun, file='lots.f90', status='unknown', iostat = istat) if (istat /= 0 ) stop 'open fail' write(iun,gfmt_txt) 'module stuff' write(iun,gfmt_txt) gpd//'implicit none' do l1 = 1 , lots CALL RANDOM_NUMBER(rty) rty = rty * real(ntypes) itype = int( rty ) + 1 selectcase (itype) case(1) write(iun,gfmt_var) gpd//'Logical(1) :: var',l1 case(2) write(iun,gfmt_var) gpd//'Logical(4) :: var',l1 case(3) write(iun,gfmt_var) gpd//'real(4) :: var',l1 case(4) write(iun,gfmt_var) gpd//'real(8) :: var',l1 case(5) write(iun,gfmt_var) gpd//'integer(4) :: var',l1 case(6) write(iun,gfmt_var) gpd//'character(len=20) :: var',l1 case default write(iun,gfmt_var) gpd//'real(4) :: var',l1 end select enddo write(iun,gfmt_txt) 'end module stuff' !write(iun,gfmt_txt) 'program use_stuff' !write(iun,gfmt_txt) gpd//'use stuff' !write(iun,gfmt_txt) gpd//'implicit none' !write(iun,gfmt_txt) 'end program use_stuff' close(iun) end program make_lots_of_vars
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page