Linking IVF Compiled DLL to Qt Application

[J_S_1]

Hi everybody,
This is my first attempt at library assembly, so please have mercy! Let me also start by saying to Steve Lionel: Wow, you answer a lot, I hope they are paying you well there at Intel!

I am trying to link an external library (written in .f90 and compiled with IVF on VS2012) to my Qt application (32bit based on Qt 5.5.0- MinGW 4.9.2).

I have a few questions:

1) Is this futile? Some of the research I have found suggests that IVF and MinGW are ABI incompatible. I really want to stay with the MinGW compiler in Qt because basically everything else we are doing with the software uses this.

2) It would be of advantage to be able to load the library only when called upon (which would only represent a fraction of cases). For this reason I have been attempting to use QLibrary but keep getting Segmentation faults when I try to call SUBROUTINES defined in my DLL (with resolve("my_function")):

I have defined my external call routines with:

 @ !DEC$ ATTRIBUTES C, REFERENCE, MIXED_STR_LEN_ARG, DLLEXPORT,   ALIAS:"sub_name" :: sub_name @
and import them using:

   @typedef int (*MyPrototype)(int i);
    MyPrototype Square = (MyPrototype) DLLname.resolve("sub_name"); @

3) Is there any way to check if the library has, in fact, loaded? Of course calling a subroutine would accomplish this, but that isn't working, and when I import the library DLLname.load() returns a positive result. and resolve.("sub_name") has a memoryaddress allocated to it. Does this suggest a type fault? Ie. passing the wrong identifier in to the fortran code.

Once again, thanks for reading and please feel free to take apart any flaws in logic I have, I'm not (yet) a programmer!

[J_S_1]

Ok, so I have just done a scan through and tried a couple of the methods above, but working with the above statements I have a few general remarks, which it would be great for you guys to comment on:

1) The most important module for my work contains literally hundreds of ALLOCATABLE types in the defined types. Should this suggest that perhaps we avoid using the BIND(C) capability of the newer version of Fortran?

2) Am I naively mistaken in thinking that you can simply reference a pointer to one of these data types, declared with a simple calling function as I have given for example above and then use this as input into the SUBROUTINES, which I will call from C with the help of a $DEC$ ATTRIBUTES exported function. Or does my main Program need to understand every data type which is being worked upon in the library?

3) I still have the option to create a static library instead of a dynamic library, would this be a much simpler implementation than trying to implement the library dynamically? The advantages of dynamic linking aren't really that profound in our case...

I really appreciate you all being so helpful

[IanH]

Perhaps some of these points have already been dealt with above, but...

J S. wrote:

Ok, so I have just done a scan through and tried a couple of the methods above, but working with the above statements I have a few general remarks, which it would be great for you guys to comment on:

1) The most important module for my work contains literally hundreds of ALLOCATABLE types in the defined types. Should this suggest that perhaps we avoid using the BIND(C) capability of the newer version of Fortran?

I don't think so.  Whether to use BIND(C) or not is reasonably separate from the types being passed across the procedure interface.

As has been stated above, the C (and hence C++/Qt) side of your program isn't going to know, without a considerable amount of effort, how to work with objects of derived type with allocatable components.  Practically you are going to be restricted to using the C address of those objects as an opaque handle (void*), that you then pass back to Fortran in order to actually manipulate the allocatable components.

My personal preference is to only use the facilities provided by the standard language (which in this case is BIND(C), versus the compiler directives) in my source that guarantee how things will work when C and Fortran are mixed, and to move the platform specific aspects out of my source and into the platform specific build environment (in this case, if necessary (see RO's posts), I would provide def files that mapped DLL entry points names across to my source files).  Others prefer everything to be in the source file - fair enough.

2) Am I naively mistaken in thinking that you can simply reference a pointer to one of these data types, declared with a simple calling function as I have given for example above and then use this as input into the SUBROUTINES, which I will call from C with the help of a $DEC$ ATTRIBUTES exported function. Or does my main Program need to understand every data type which is being worked upon in the library?

Your main program needs to understand the data types to the level that it will interact with them.  If it is just passing objects created by one procedure across to a different procedure, then it can just refer to the objects by their C address.  An example below.

3) I still have the option to create a static library instead of a dynamic library, would this be a much simpler implementation than trying to implement the library dynamically? The advantages of dynamic linking aren't really that profound in our case...

I prefer static libraries for tightly bound code from the point of view of ease of build and ease of distribution, but that is incidental here.  I don't think it makes a great deal of difference in your case.

The following C++ and Fortran files (and a linker def file) are an example of a C++ main program calling through to Fortran to manipulate a derived type with an allocatable component using an void * opaque handle on the C++ side.  I don't use Qt, so the dynamic loading is accomplished using the native Win32 api - with this platform specific nonsense all confined to the thing_wrapper::thing_functions::init function. 

(I think the Windows linker is smarter than it might otherwise be given credit for in terms of its ability to identify/demangle symbol names that are marked for export - hence some of the BIND(C) versus ATTRIBUTES C stuff above becomes moot.)

[J_S_1]

==> IanH:

Thankyou very very much for that last post. Perhaps too early to tell, but I think this is exactly what I needed, a way to call the functions from within Qt and pass values back and forth. I can save myself the hassle of trying to redefine every derived type, if I simply declare them in Fortran and then simply use the their C pointers in the subroutines (which I have exported).

I took the code and attempted to implement it within Qt. The main steps work. (Library loads, functions are recognised, handles are passed to C). A problem however is arising when trying to "execute" the program.

As a side note I have compiled the source code with the given def file in Intel Visual Fortran with Runtime Library (multithreaded) and use common windows apps (Yes (/Winapp)) and calling convention as default.

During compiling, while trying to set the value of the 'the_thing' the following error occurs:

warning: invalid conversion from 'const double*' to 'double*' [-fpermissive]
       function_table.set(opaque_handle, &the_value[0], the_value.size());

I tried a number of methods to avoid this issue, redifing the variables, removing the const. args etc. The crash can be avoided by setting the

-fpermissive

flag in my compiler (Qmake- gcc). The 'execute' function appears to not be working correctly, it returns simply a value of 0. I rewrote a new function 'Return' to return simply the values in my 'thing' variable, but it also returned 0. As such I cannot verify whether 'set' is working at all or whether it is simply an error with 'execute'.

[JVanB]

I suggest fixing the problems with the code rather than using -fpermissive.

//extern "C" typedef void (set_fortran_t)(void* thing_c_ptr, double value[], size_t size);
extern "C" typedef void (set_fortran_t)(void* thing_c_ptr, const double value[], size_t size);

Now if I compile the Fortran with

gfortran -shared Interop-Fortran.f90 -oInterop-Fortran.dll Interop-Fortran.def-rename-me.def

and the C++ with

g++ -Wall -std=c++11 Interop-Cpp.cpp -oInterop-Cpp

The Interop-Cpp.exe runs with output 24.

However, if I compile the Fortran with

ifort /nologo Interop-Fortran.f90 /dll Interop-Fortran.def-rename-me.def

And the C++ as above it runs but with output 0. So that seems to be progress, but I don't know why the interaction between ifort and g++ is causing difficulties.

[J_S_1]

==> RO. Thanks for the quick reply. I did already attempt that, but alas no luck. I'm guessing I need to specify a specific type of build option in ifort...

Unfortunately I don't have much wriggle room when it comes to the choice of compilers...

[JVanB]

Oh man, I was so close in Quote #25! The only thing I missed was

ifort /nologo /assume:realloc_lhs Interop-Fortran.f90 /dll Interop-Fortran.def-rename-me.def

And then it works with output of 24. Steve, could you once again push the button that administers the negative reinforcement to the one responsible for the default being /assume:norealloc_lhs? Thanks in advance.

[J_S_1]

==> RO. Thanks for the fix! Modifying the compiler flags allowed mine to work also!

Ok gentlemen, I am getting closer and closer to the goal. I have now a function everywhere to return a handle to all of these data types. I am however having an issue with trying to define one of my SUBROUTINES with the BIND(C) syntax.

The SUBROUTINES looks as follows:

SUBROUTINE FAST_InitializeAll( t_initial, p_FAST, ...ErrStat,InFile)
   ....

   TYPE(FAST_ModuleMapType), INTENT(INOUT) :: MeshMapData        
      
   INTEGER(IntKi),           INTENT(  OUT) :: ErrStat             ! Error status of the operation
   CHARACTER(*),             INTENT(  OUT) :: ErrMsg              ! Error message if ErrStat /= ErrID_None
   CHARACTER(*), OPTIONAL,   INTENT(IN   ) :: InFile              ! A CHARACTER string...
   

I get the compiler issue:

Error	1	 error #8532: A character dummy argument with length other than 1 is not interoperable.   [ERRMSG]	

After searching some more for solutions I found Steve's post from an earlier issue (forums/topic/284279), and tried the following fix:

CHARACTER, DIMENSION(*),             INTENT(  OUT) :: ErrMsg            
CHARACTER(*),DIMENSION(*), OPTIONAL,   INTENT(IN   ) :: InFile 

This led to the following error messages:

error #6364: The upper bound shall not be omitted in the last dimension of a reference to an assumed size array.   [ERRMSG]	
error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [INFILE]	

Are these types in inputs completely interoperable?

[Steven_L_Intel1]

CHARACTER(*) is not interoperable in version 15. In version 16 it is (due to support of TS29113), but not the way you want as such things are passed by "C descriptor". But I need to see more of the code to understand the diagnostics.

For a BIND(C) routine, you would generally use CHARACTER, DIMENSION(*) - this does allow you to pass an arbitrary-length character value to it through a special rule in the standard.

[J_S_1]

Sure, I'll try to fill you in with all of the important points where this variable is called. Obviously by the way the CHARACTER(*),DIMENSION(*) statement was a typo. So the Errmsg variable is first declared like this:

CHARACTER(1024)                       :: ErrMsg                                  ! Error message

Before it is used as an into to the subroutine:

SUBROUTINE FAST_InitializeAll( t_initial, p_FAST, ...ErrStat,InFile)

The variable is called soon thereafter with:

CHARACTER(LEN(ErrMSg))                  :: ErrMsg2

Within the subroutine the point where the code actually causes the error is:

ErrMsg  = ""

Before both are passed down through many further, embedded subroutines which use them as input.

[J_S_1]

I think this is not something I need to worry about, because I am going to be creating my variables all inside my fortran program anyway, so I will need to modify the four subroutines to take as input C_PTR types and then operate upon the variables at these addresses.

So if we take the examplatory subroutine above, the p_FAST variable for example, I will modify my code to take a C pointer as input so:

SUBROUTINE FAST_InitializeAll(....C_HANDLE...)

USE, INTRINSIC :: ICO_C_BINDING

TYPE(C_PTR), INTENT(IN), VALUE :: C_HANDLE
TYPE(FAST_ParameterType), POINTER :: p_FAST
CALL C_F_POINTER(C_HANDLE, p_FAST)

This now references everything back to p_FAST, as I wanted. In the original subroutine however, the variable is declared as:

TYPE(FAST_ParameterType), INTENT(INOUT) :: p_FAST        

Will there be an issue here because the variable gets operated upon in the original function, but is only declared through "INTENT(IN)" in my re-written subroutine, or does the call C_F_POINTER automatically mean that p_FAST is INTENT(OUT) and will be redefined?

[Steven_L_Intel1]

p_FAST is a local variable in your rewritten subroutine, isn't it? C_HANDLE has INTENT(IN), but you're not changing that. Certainly nothing you can do in any procedure automatically changes a dummy argument's INTENT to OUT.

[J_S_1]

Thanks again for the quick reply Steve.

C_HANDLE will not change, it is simply the pointer to the (Elsewhere defined) derived type. The question was rather now whether any operation I perform upon p_FAST will actually be reflected in the variable at the (original=) allocated memory slot which C_HANDLE points to.

[J_S_1]

Unfortunately even defining everything as a C_PTR does not get around the problem I mentioned in #28. When I try to define the pointers as a variable length Character:

CHARACTER, DIMENSION(*),  POINTER :: ErrMsg

exactly the same assue arises

error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.  

I can't imagine why CHARACTER, DIMENSION(*) does not work here... Perhaps you will have a better idea. Despite how much I search for a solution/ try to read through material, the problem appears to be very specific...

[IanH]

Putting aside the F2015 features, and guessing a little at what you are trying to do (things tend to get a bit murky when split across multiple posts in a thread), the following examples (which might now be const correct... thanks RO) show one approach that can be used to pass string data back and forth between C++ and Fortran.  In this example I am explicitly passing both a pointer to the character data that makes up the string, and the length of the string itself.  This is conceptually consistent with the basic_string type in the C++ standard library, but less so with C strings.  Alternative approaches, perhaps where Fortran understands the C practice of using a null character to terminate delimit a string, could also be used.

In this example I am also copying the character data provided by the C++ code on the Fortran side, as this is simpler in terms of management of object lifetime and avoiding issues with different character kinds.  For short strings I think this is reasonable.  Alternative approaches exist where the character data is access through a deferred length Fortran pointer, and these may be attractive if the amount of character data being passed is large.

An example is also show of how C++ can then query a string managed by Fortran.  This is done by two calls between the languages - one to get the length of the string data, so that C++ can allocate an appropriate buffer, the other to actually copy the data into that buffer.  This two step approach means it is clear which language owns a particular object.  Again, alternative approaches are available, what's best depends on your situation.

If you compile the Fortran with gfortran you will need quite a recent version (perhaps even current trunk), due to issues with its deferred length character support.

[J_S_1]

==> IanH: thanks very much! Once again this is a huge help.

As mentioned earlier, I will be passing all input values anyway as C_PTR types which point to the allocated locations of the data types, but this addition will allow me to determine the length of the input argument so that I can avoid the declarations with CHARACTER(*) etc, this will be my task for the weekend.

In case you have a chance before the weekend, one very important question which wasn't really addressed earlier.. I will be exporting the function with BIND(C, Name = ...), but each of the four dominant subroutines that I am exporting contains many many other calls to other subroutines (literally hundreds). Does every subroutines which is called also need to be declared with BIND(C) (and therewith, do I need to go through this sort of process with every single subroutines used), or is it sufficient to define ONLY the subrtouines I wish to call from C with BIND(C)?

[Lorri_M_Intel]

If I understand your question correctly ... you only need to use BIND(C) on the routines that will be called from C.

Calls inside of Fortran can stay "Fortran-y", no matter what the outer routine does.

              ---Lorri

[J_S_1]

==> Lorri: Thanks very much for the comment Lorri, that's a huge relief.

Ok, my thanks are out to everybody, I am making serious progress.

I have got a version of the Library written which is almost at the stage of working, however I am having a small issue with declaring allocatable arrays in C++. My export function has the folllowing form:

subroutine FAST_Start(...ErrMsg_c) BIND (C, NAME='FAST_Start')
   USE Type_Data_Module....
   IMPLICIT NONE 
  ...
   INTEGER(C_INT),         INTENT(  OUT) :: ErrStat_c      
   CHARACTER(KIND=C_CHAR), INTENT(  OUT) :: ErrMsg_c(IntfStrLen)    

Could anybody fill me in how to define the inputs for the character arrays on the C++ side? There are a LOT of similar questions online, but as far as I can tell they all relate to inputs with known string length. As we see the string length IS known here, and in the worst cast scenario I could write a small export function to give me the value of   #IntfStrLen#. I have attempted to define/call the caracter array as follows:

extern "C" typedef void (FAST_Start)(
        ...
        char* Name_data);

int main()

{

      #Load library
      # Get procedure address
      char* b
      FAST_Start(....&b)
}

 This however returns a segmentation fault. When I try to define the arraw length with char Name_data[1025], and the declare b[1025], it returns only the first letter of the string...

[IanH]

J S. wrote:

==> Lorri: Thanks very much for the comment Lorri, that's a huge relief.

Ok, my thanks are out to everybody, I am making serious progress.

I have got a version of the Library written which is almost at the stage of working, however I am having a small issue with declaring allocatable arrays in C++. My export function has the folllowing form:

subroutine FAST_Start(...ErrMsg_c) BIND (C, NAME='FAST_Start')
   USE Type_Data_Module....
   IMPLICIT NONE 
  ...
   INTEGER(C_INT),         INTENT(  OUT) :: ErrStat_c      
   CHARACTER(KIND=C_CHAR), INTENT(  OUT) :: ErrMsg_c(IntfStrLen)    

Could anybody fill me in how to define the inputs for the character arrays on the C++ side? There are a LOT of similar questions online, but as far as I can tell they all relate to inputs with known string length. As we see the string length IS known here, and in the worst cast scenario I could write a small export function to give me the value of   #IntfStrLen#. I have attempted to define/call the caracter array as follows:

extern "C" typedef void (FAST_Start)(
        ...
        char* Name_data);
/*          ^   subroutine expects pointer to char. */

int main()

{

      #Load library
      # Get procedure address
      char* b;
/*        ^  b is a pointer to char. */
      FAST_Start(....&b);
/*                   ^  Get the address of b, that is, get the 
                        address of a pointer to char.  This is 
                        a pointer to pointer to char. */
}

 This however returns a segmentation fault. When I try to define the arraw length with char Name_data[1025], and the declare b[1025], it returns only the first letter of the string...

Your C code has an indirection issue - as written above, your C argument does not match your C declaration.  Given the fragments of code shown, your Fortran subroutine does match its C declaration.

If you are asking about how to get string data from Fortran back to C, there is an example of this in the code attached in reply #35.  Beyond that, perhaps you could elaborate a little.

[gib]

Here is something that seems to work:

extern "C" void get_string(char **, int *);
...
    char *b;
    int nb;
    get_string(&b, &nb);
    b[nb] = '\0';

subroutine get_string(bufptr, buflen) bind(C)
!DEC$ ATTRIBUTES DLLEXPORT :: get_string
use, intrinsic :: iso_c_binding
type(c_ptr) :: bufptr
integer(c_int) :: buflen
character*(1024) :: string

string = 'A test string'
buflen = len(string)
bufptr = c_loc(string)
end subroutine

[IanH]

gib wrote:

Here is something that seems to work:

extern "C" void get_string(char **, int *);
...
    char *b;
    int nb;
    get_string(&b, &nb);
    b[nb] = '\0';

subroutine get_string(bufptr, buflen) bind(C)
!DEC$ ATTRIBUTES DLLEXPORT :: get_string
use, intrinsic :: iso_c_binding
type(c_ptr) :: bufptr
integer(c_int) :: buflen
character*(1024) :: string

string = 'A test string'
buflen = len(string)
bufptr = c_loc(string)
end subroutine

`string` is an unsaved local variable of the get_string procedure.  It ceases to exist each time that procedure returns.  The C address of ``string` that is being passed back in `bufptr` will therefore be the address of something that is no longer really there.  This might appear to work because, as an implementation detail, the compiler has set aside some static storage (storage that has the life of the entire program) for the `string` variable, but that is compiler specific behaviour that is very much subject to change.

Adding SAVE to the `string` variable, which ensures that the storage for the `string` variable is persistent over the life of the entire program, is a fix for this, but then you need to be mindful that the one storage area will be re-used for multiple .calls to `get_string`. 

A more general solution is to make `string` a Fortran pointer, allocate a target for the pointer (using ALLOCATE) to the desired size, and then return the C address of the target of that pointer..  However, then you need to consider how C then tells Fortran at some later stage that the target of the pointer can be deallocated.

There is also a buffer run with the code as shown - the C code will be whacking a null into the memory that corresponds to `string(1025)` (given C's zero based indexing.).  LEN_TRIM may have been intended instead of LEN, but the potential for buffer overrun still needs to be considered.

(The argument to C_LOC also needs to have the target attribute.)

[J_S_1]

My computer is about to die gentlemen so I will try to quickly write a response.

Thanks to both of you but it appears in both cases I need to also provide the string length as an input argument (also in comment #35). In my particular case the return string will always have 1025 characters, so I think I just need to pass a pointer to the function for a character array with 1025 characters. Would this then be

char* b [1025]

? And is it possible to not declare the length and allow C++ to figure this out for itself? I will take another look at both of your posts tomorrow, thankyou for taking a look.