Re-initialize Fortran DLL called from VBA

[cmc]

Hi,

I have written a fortran dll which I am calling from Excel VBA. The following declaration is given at the top of the fortran code:

!dec$ attributes dllexport, stdcall, reference, alias: "fortran_dll" :: fortran_dll

And the following declaration is given in VBA:

Public Declare Sub fortran_dll _
Lib "path\fortran_dll.dll" _
(ByVal strings As String, ByRef len_strings As Integer, _
 ByRef array1 As Integer, ByRef array2 As Double, _
 ByRef array3 As Integer, ByRef array4 As Double)

The DLL works as intended the first time I call it, but throws NAN and INF-numbers the second time I call it. It seems that the DLL is not released from the previous calculation, and this causes memory problems or something. If I activate "Design Mode" in Excel between every time I call the DLL, it works.

I need to loop this DLL, because it is going to be called x number of times during an iteration process. Any suggestions?

\Carl

[mecej4]

Many details are missing and I know next to nothing about Visual Basic/Excel, but here is a potential problem: the order in which you pass the hidden string-length argument. Please read the Mixed Language chapter of the Intel Fortran reference manual. In particular, refer to the description of the /iface:mixed_str_len_arg compiler option and the corresponding Len:End / Len:Mixed clause in the DEC$ directives in the Fortran source of the DLL.

[Steve_Lionel]

You can't "reset" a DLL called from VBA. I am skeptical that doing so would solve your problem. mecej4 has some good suggestions.

[jimdempseyatthecove]

RE: first call seems good, subsequent calls bad

Remember that you cannot expect (require) Fortran to auto init (e.g. zero) variables. They (saved) are either undefined, initialized once at load time, or residual values left over from last call. 

[JAlexiou]

There is a possible bug in the VBA code since `Integer` means 16-bit `INTEGER(2)` and `Long` means 32-bit `INTEGER(4)`. 

Try the following, as I am suspecting the first call corrupts the memory stack. Also, the string length is passed by value I think.

Public Declare Sub fortran_dll Lib "path\fortran_dll.dll" _
(ByVal strings As String, ByVal len_strings As Long, _
ByRef array1 As Long, ByRef array2 As Double, _
ByRef array3 As Long, ByRef array4 As Double)

See https://bettersolutions.com/vba/data-types/index.htm for a list of data types in VBA, indicating that `Integer` is a 2-byte integer value. Unfortunately, the MS documentation for this very poor because of this changed after VB6 (in VB.NET) and the old (obsolete) documentation is hard to find (even though VBA is still current).

[DavidWhite]

In addition, I have found it necessary to ensure that strings passed from VBA to Fortran are initialized to their full declared length, e.g.

Dim Units As String * 20

It can be useful to fill the variable with spaces, 

String = SPACE(20)

Otherwise there can be occasions where the stack is again corrupted.

[cmc]

John Alexiou wrote:

There is a possible bug in the VBA code since `Integer` means 16-bit `INTEGER(2)` and `Long` means 32-bit `INTEGER(4)`. 

Try the following, as I am suspecting the first call corrupts the memory stack. Also, the string length is passed by value I think.

Public Declare Sub fortran_dll Lib "path\fortran_dll.dll" _
(ByVal strings As String, ByVal len_strings As Long, _
ByRef array1 As Long, ByRef array2 As Double, _
ByRef array3 As Long, ByRef array4 As Double)

See https://bettersolutions.com/vba/data-types/index.htm for a list of data types in VBA, indicating that `Integer` is a 2-byte integer value. Unfortunately, the MS documentation for this very poor because of this changed after VB6 (in VB.NET) and the old (obsolete) documentation is hard to find (even though VBA is still current).

I have declared the integer values in fortran as Integer(2), so that should be taken care of. See code below:

Fortran:
!dec$ attributes dllexport, stdcall, reference, alias: "fortran_dll" :: fortran_dll
!dec$ attributes reference :: input_characters, input_len_characters
!dec$ attributes reference :: input_integers, input_reals
!dec$ attributes reference :: output_integers, output_reals

subroutine fortran_dll (input_characters, input_len_characters, &
                        input_integers, input_reals,            &
                        output_integers, output_reals)

    implicit none
    
!==================================================================================================
!   declarations
!==================================================================================================
!   main routine:
    integer(2), intent(in)                          :: input_len_characters
    character(len=input_len_characters), intent(in) :: input_characters

    integer(2), intent(in)                          :: input_integers(3)
    real(8), intent(in)                             :: input_reals(36)
    integer(2), intent(out)                         :: output_integers(4)
    real(8), intent(out)                            :: output_reals(38)

[cmc]

mecej4 wrote:

Many details are missing and I know next to nothing about Visual Basic/Excel, but here is a potential problem: the order in which you pass the hidden string-length argument. Please read the Mixed Language chapter of the Intel Fortran reference manual. In particular, refer to the description of the /iface:mixed_str_len_arg compiler option and the corresponding Len:End / Len:Mixed clause in the DEC$ directives in the Fortran source of the DLL.

Hi. I think you're on to something here. I read through the mixed language documentation, but I'm a novice when it comes to these things, so I didn't understand much. But I read somewhere that the Fortran directive !DEC$ attribute STDCALL and REFERENCE together inform Fortran not to expect the hidden length arguments to be passed from VBA. So I'm a bit confused. Any suggestions of what I can do? I plotted some more from my code in the reply above...

[mecej4]

I noticed the "len_strings" argument in your VBA declaration, and my comment was associated with that. One possible value of "somewhere" is https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-attributes-reference-and-value#5E51D4F4-5023-42EA-B4F9-F0B3CA1F64D1 . On that page, you can note that directives can be used to specify the attributes of not only procedures but also of individual arguments of procedures, if needed.

Can a string be passed without also passing a length argument? That depends on whether a Fortran procedure is being called from some other language, or Fortran code is calling a routine in some other language, and what is going to be done with/to the string argument. There may be flexibility in the calling convention, but the choices have to be consistent.

For Fortran-C interfacing, Fortran 200X provides a standard, portable way of achieving interoperability.

[cmc]

mecej4 wrote:

I noticed the "len_strings" argument in your VBA declaration, and my comment was associated with that. One possible value of "somewhere" is https://software.intel.com/en-us/fortran-compiler-developer-guide-and-re... . On that page, you can note that directives can be used to specify the attributes of not only procedures but also of individual arguments of procedures, if needed.

Can a string be passed without also passing a length argument? That depends on whether a Fortran procedure is being called from some other language, or Fortran code is calling a routine in some other language, and what is going to be done with/to the string argument. There may be flexibility in the calling convention, but the choices have to be consistent.

For Fortran-C interfacing, Fortran 200X provides a standard, portable way of achieving interoperability.

One string can be passed without passing the length argument. However, I want to send many strings from VBA to Fortran. This was not straight forward, so the easiest solution I found was to combine all the strings to one long string with a delimiter between each string. And then I send this string and the length of it to Fortran. Then I divide the long string into individual ones in Fortran.

[JAlexiou]

The compiler adds the string length argument automatically, so you only need to declare it in VBA and not in Fortran.

subroutine fortran_dll (input_characters, input_integers, input_reals, output_integers, output_reals) 

    character(len=400), intent(in) :: input_characters
    integer(2), intent(in)                          :: input_integers(3)
    real(8), intent(in)                             :: input_reals(36)
    integer(2), intent(out)                         :: output_integers(4)
    real(8), intent(out)                            :: output_reals(38)

The string needs to be fixed length, so in VBA declare it with `Dim text as String*400` for example. 

Public Declare Sub fortran_dll Lib "path\fortran_dll.dll" _
(ByVal strings As String, ByVal len_strings As Long, _
ByRef array1 As Integer, ByRef array2 As Double, _
ByRef array3 As Integer, ByRef array4 As Double)

to be called as

Call fortran_dll(text,400,n,x(1),m,y(1))

So remember, when you export a function in Fortran with a string argument, the compiler will add an argument to the function for the string length (this might be either after the string or in the end depending on the compiler settings). In VBA in addition to the arguments declared in Fortran, you need to include the corresponding string lengths with type `Long`.

cmc wrote:

Quote:

I have declared the integer values in fortran as Integer(2), so that should be taken care of. See code below:

Fortran:
!dec$ attributes dllexport, stdcall, reference, alias: "fortran_dll" :: fortran_dll
!dec$ attributes reference :: input_characters, input_len_characters
!dec$ attributes reference :: input_integers, input_reals
!dec$ attributes reference :: output_integers, output_reals

subroutine fortran_dll (input_characters, input_len_characters, &
                        input_integers, input_reals,            &
                        output_integers, output_reals)

    implicit none
    
!==================================================================================================
!   declarations
!==================================================================================================
!   main routine:
    integer(2), intent(in)                          :: input_len_characters
    character(len=input_len_characters), intent(in) :: input_characters

    integer(2), intent(in)                          :: input_integers(3)
    real(8), intent(in)                             :: input_reals(36)
    integer(2), intent(out)                         :: output_integers(4)
    real(8), intent(out)                            :: output_reals(38)

[cmc]

Hi again. I have tried many different things, with no luck. As mentioned before, the DLL produce the correct outputs the first time I call it. The second time I call it, it produce outputs, but they are incorrect. I see now that all the outputs are actual values, but most of them are "unphysical", e.g. 2.00693690139177E+244 and 5.50092032905669E-240.

Isn't this a strong indication that VBA is providing wrong inputs for the second call? The DLL produce the correct outputs if I clear the memory on the Excel side by activating Design Mode. I'm thinking that that the Fortran code is correct and something fishy is happening on the VBA side.

PS, I cannot find any hidden string length argument in the call stack when I run the DLL in debug.

Fortran:

!dec$ attributes dllexport, stdcall, reference, alias: "fortran_dll" :: fortran_dll
!dec$ attributes reference :: input_characters, input_integers, input_reals
!dec$ attributes reference :: output_integers, output_reals

subroutine fortran_dll (input_characters, input_integers, input_reals,                 &
                         output_integers, output_reals)

    implicit none

    character(len=500), intent(in)  :: input_characters
    integer(2), intent(in)          :: input_integers(3)
    real(8), intent(in)             :: input_reals(36)
    integer(2), intent(out)         :: output_integers(4)
    real(8), intent(out)            :: output_reals(38)

VBA:

Public Declare Sub fortran_dll _
Lib "path/fortran_dll.dll" _
(ByVal strings As String, _
 ByRef array1 As Integer, ByRef array2 As Double, _
 ByRef array3 As Integer, ByRef array4 As Double)

Dim input_strings As String * 500
Dim input_integers(1 To 3) As Integer
Dim input_doubles(1 To 36) As Double
Dim output_integers(1 To 4) As Integer
Dim output_doubles(1 To 38) As Double

Call fortran_dll(input_strings, input_integers(1), input_doubles(1), _
                 output_integers(1), output_doubles(1))

[Steve_Lionel]

How about you construct a small, self-contained example with both Fortran and Excel and attach a ZIP here so we can try it.

[cmc]

Steve Lionel (Ret.) (Blackbelt) wrote:

How about you construct a small, self-contained example with both Fortran and Excel and attach a ZIP here so we can try it.

Hi Steve. I've made a simplified example, but I were not able to reproduce my error. So I guess it's not much of help. When I press calculate using my original setup, the calculation is correct the first time, but the second time I press calculate, the outputs are incorrect. For the simplified one, I can press calculate as many times I want without corrupting the outputs. Hmm...

If you want to try out the example, remember to update the path to the DLL in the VBA developer, and to locate the .dat-files in appropriate folder. On my computer they need to be located in c:\users\"name"\documents for the DLL to be able to read them. I have not managed to figure out how to modify this path. Note that the Fortran code is only reading one of the files. I included this part just to show how I access the files.

\Carl

[Steve_Lionel]

If the error can't be reproduced, the example won't help us help you. I suggest that you try to determine what the critical difference is between this test case and your original program.

[mecej4]

Faced with similar problems, I have used a procedure that CMC may consider using. 

1. Locate the routine and line number of a place in the program where a crash occurs or a variable attains an incorrect value.
2. Insert counters in the source code and run as in 1. to find how many times the routine located in 1. is called before the halt.
3. Place a STOP statement conditioned by the counter value at the location determined in 1. and 2.
4. Rerun the program with profiling enabled (with a different compiler that is known to be error-free for this particular program, if appropriate).
5. Replace all the subprograms that were not touched in 4. by dummy subroutines (to satisfy the linker), and check that the same behavior as before has been preserved.
6. At this point, if the number of lines of code of the whole program has been reduced significantly, use the reduced source as a new test program for further investigation with the compiler (and options) whose correctness is suspected.

All this work of bug-preserving-compactification is tedious, perhaps, but may be palatable when the alternative is to give up. 

[cmc]

Hi guys. I solved it! I managed to debug the DLL for the second call and quickly found the error. But I'm quite puzzled by it. Maybe you guys can explain? In one of my subroutines I have the following declaration at the top:

logical :: iteration_solved = .false.

What I discovered for the second call was that iteration_solved remembered the value from the end of the first call, i.e. .true. And then it did not enter the iteration process I had set up which was a do while loop that would go on until iteration_solved became .true. And then one of my arrays were out of bounds because of this and the DLL failed.

So what I did to fix it was to declare it the following way:

logical :: iteration_solved
iteration_solved = .false.

Why is this different?

[mecej4]

When you provide an initialization in a variable declaration, the variable acquires the SAVE attribute. The initialization itself happens only once, at program start.

When you set the variable value in an executable statement, the value is assigned whenever the statement is executed. If that statement occurs before any branching statements or subprogram calls, that assignment will be performed whenever the containing subprogram is called from somewhere else.

[cmc]

Hmm. I didn't realize the DLL was able to remember anything from call to call. I thought it was kind of "brain dead" and required initialization of all variables every time it was being called. This might come in handy...

[FortranFan]

cmc wrote:

Hmm. I didn't realize the DLL was able to remember anything from call to call. I thought it was kind of "brain dead" and required initialization of all variables every time it was being called. This might come in handy...

Such a feature, let's call it implicit SAVE, can be a double-edged sword.  Anyways, here's a link to Intel's documentation on SAVE:

https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-save#0091E04E-CAA0-4081-9415-E0ACD2323057

[Eugene_E_Intel]

cmc wrote:

Hmm. I didn't realize the DLL was able to remember anything from call to call. I thought it was kind of "brain dead" and required initialization of all variables every time it was being called. This might come in handy...

Loaded DLLs preserve internal state (e.g., global variables) until they are unloaded.  DLL is not unloaded until its load reference count goes to zero.  It's reference count is explicitly affected by LoadLibrary/FreeLibrary calls, but could also be implicitly affected by whatever other subsystem (e.g., COM or .NET) that decides to load DLL.  In .NET context, it could be especially tricky to unload certain kinds of DLLs, see https://stackoverflow.com/questions/4887847/hot-unload-reload-of-a-dll-used-by-an-application