Difficulties building/linking with ISO C binding with ifort on Windows

[Bob_A_1]

Big picture: I am trying to embed a Lua interpreter in Fortran code via the F2003 ISO C bindings (see https://bitbucket.org/haraldkl/aotus ).

I can get it to work with ifort and Windows, just not both at the same time ('it works' means 'a C library is built and example Fortran code using the Lua bridge builds and produces the expected results'). gfortran/gcc works on both Windows and Linux, ifort/gcc works on Linux, but neither ifort/msvc or ifort/gcc works on Windows. The C code is ANSI C and the Fortran is plain F2003; there's no screwy preprocessor stuff, platform-specific nonsense, or operating environment weirdness like Cygwin. This isn't exactly a trivial problem but as I said, I've successfully built this code with gfortran/gcc on both Windows & Linux and with ifort/gcc on Linux so the problem is not with the code or the operating systems. I expect it should build with ifort on Windows but I cannot figure out the magic words to make that happen.

The C portion of the project compiles without incident and is packed into a few static libraries (libaotus.a, libflu.a, liblualib.a) with ar which comes from the TDM distribution of gfortran and gcc for Windows. I'm not worried about this part since this code works fine if I build the code with gfortran.

The automated build tool used by the project (waf) fails with ifort on Windows so I'm trying to manually convert the compilation steps which use gfortran to ifort. I have a reasonable amount of success until I try linking with the C libraries which is where everything falls apart.

Here's an example of where conversion is failing - first the (successful) compilation step:

@REM [46/84] Compiling LuaFortran\examples\test.f90
@REM gfortran.exe -O3 -march=native -Wall -Wconversion -Wimplicit-interface -Wunderflow -W -frange-check -c -o .\LuaFortran\examples\test.f90.8.o ..\LuaFortran\examples\test.f90
ifort.exe /O3 /warn=all /fpe:0 /warn=interfaces /gen-interfaces -c -o .\LuaFortran\examples\test.f90.8.o ..\LuaFortran\examples\test.f90

Then a few lines later, the unsuccessful link step:

@REM [49/84] Linking build\flu_sample.exe
@REM gfortran.exe -Wl,--enable-auto-import -Wl,--enable-auto-import -O3 -march=native -Wall -Wconversion -Wimplicit-interface -Wunderflow -W -frange-check LuaFortran\examples\test.f90.8.o -o .\flu_sample.exe -L. -lflu
@REM -Wl,--enable-auto-import -Wl,--enable-auto-import ; superfluous linker directives; only suppresses warnings which may not exist with ifort
ifort.exe /O3 /warn=all /fpe:0 LuaFortran\examples\test.f90.8.o -o .\flu_sample.exe /LIBPATH:. -lflu

I'm almost certain the problem is in the unixy '-lflu' argument to ifort which (I believe) is passed directly through ifort to LINK.EXE, which, as far as I can tell, is not meant for humans to understand or manipulate directly. I believe if I can get the linking sorted out, the rest of the project will fall into place.

What's the right way to resolve this?

Is there a way to tell ifort to ignore Microsoft's infrastructure and just use C:\TDM-GCC-64\bin\ld.exe which (apparently) works?

Note that I typically work with either pure Fortran or interpreted languages. If I'm faced with a linking problem, it usually indicates I'm using the wrong language or platform. I've been fighting with this for a week or two now trying to get it to build with ifort on Windows, and passing whatever knowledge I gather back to the upstream project (I'm not directly involved in the Lua/Fortran bridge development). I've helped improve the automated build system's detection of ifort, but so far it's been a miserable slog on Windows. Any help you could provide would be greatly appreciated.

[JVanB]

I seem to recall having had success replacing "-lflu" with "libflu.a" in a similar situation. See if that helps any.

[mecej4]

On Windows, Ifort does not recognize the -l option. Instead, specify flu.lib (if that is the name that you gave to the library) in the command line used for linking. The linking will be attempted but, if you built the library using GNU tools, it is highly likely that the linking will fail because of the many incompatibilities between the runtime libraries of GCC and MSVC.

[Steven_L_Intel1]

/lLIBPATH is a linker option. To use that with "ifort", precede it with /LINK. That tells ifort that everything after that on the line is for the linker. As mecej4 says, you would give the actual name of the .lib. If you don't need to use /LIBPATH, you can skip that and just name the library on the ifort command line.

[Bob_A_1]

Thanks for the suggestions. The literal filename for the library is libflu.a so replacing '-lflu' with the library filename would make sense. Then the question is whether LINK.EXE could deal with a *.a library vs a *.lib library or (conversely) whether ld.exe could link the ifort-produced *.o file with the gcc-produced *.a file. Under Linux this is straightforward; ld handles everything, but on Windows, the toolchain seems really confused and unconfigurable and I'm not sure if what I'm trying to do is even possible.

I'll try adding /LINK to preface the linker arguments and specifying libflu.a in place of -lflu and see if that works.

[Steven_L_Intel1]

Windows is simply different from Linux, not more complicated. What looks complicated is when you try to shoehorn Linux commands into a Windows environment.

A .a library is not built for use with Microsoft tools.

[Bob_A_1]

Ok, that closes off one line of attack; the combination of gcc & ifort will work on Linux but not on Windows so I can't leverage any success I've had with gcc. Now the difficulty is back to getting MSVC to work; the automated build tool's inability to deal with MSVC or vice-versa was what caused me to abandon MSVC in the first place (using the build tool isn't my choice, it's how the library is shipped.)

[TimP]

The different naming of libraries is only part of the problem. In spite of both working on Windows, the run time libraries for gcc aren't compatible with those for visual studio and ifort.

[Bob_A_1]

@Tim Yeah, I figured as much but I thought I'd ask. I typically program at a much higher level than this; if I'm explicitly trying to control the linker, something has gone terribly, terribly wrong. Unfortunately, I need to link C & Fortran code. Getting anything to build reliably on Windows using actual Windows-specific tools has yielded nothing but bafflement and failure.

[TimP]

Cygwin should be an easier way to check suitability of Windows for a linux application. Going further to use ifort normally requires a corresponding c++.

[JVanB]

Why are run time library incompatibilities a problem? I thought the whole point of joining pieces of a program together as DLLs was that each could run using its own RTL.

! gfdll.f90
! Compile with:
! gfortran -shared gfdll.f90 -ogfdll.dll -Wl,--out-implib,libgfdll.a
module gfM
   use ISO_C_BINDING
   implicit none
   contains
      subroutine S(x,a,b,N) bind(C,name='gfS')
         !GCC$ attributes dllexport :: S
         integer(C_SIZE_T), value :: N
         real(C_DOUBLE) x(N)
         real(C_DOUBLE), value :: a, b
         real(C_DOUBLE) h
         integer(C_SIZE_T) i
         
         h = (b-a)/(N-1)
         x = BESSEL_J0([(a+h*i,i=0,N-1)])
      end subroutine S
end module gfM

! ifmain.f90
! Compile with:
! ifort ifmain.f90 libgfdll.a
module ifM
   use ISO_C_BINDING
   implicit none
   interface
      subroutine S(x,a,b,N) bind(C,name='gfS')
         import
         implicit none
         !DEC$ attributes dllimport :: S
         integer(C_SIZE_T), value :: N
         real(C_DOUBLE) x(N)
         real(C_DOUBLE), value :: a, b
      end subroutine S
   end interface
end module ifM

program ifP
   use ifM
   implicit none
   integer(C_SIZE_T), parameter :: N = 10
   real(C_DOUBLE) x(N), y(N)
   real(C_DOUBLE) a, b
   real(C_DOUBLE) h
   integer(C_SIZE_T) i

   a = 0
   b = 3
   h = (b-a)/(N-1)
   y = BESSEL_J0([(a+h*i,i=0,N-1)])
   call S(x,a,b,N)
   do i = 1, N
      write(*,*) x(i),y(i)
   end do
end program ifP

Output:

   1.00000000000000        1.00000000000000
  0.972414529115098       0.972414529115098
  0.891937468132727       0.891937468132727
  0.765197686557966       0.765197686557967
  0.602566169772492       0.602566169772492
  0.417208867196137       0.417208867196137
  0.223890779141236       0.223890779141236
  3.764579426748675E-002  3.764579426748669E-002
 -0.127560987606651      -0.127560987606650
 -0.260051954901933      -0.260051954901933

See? Different RTLs, so different results, but everyone was happy :)

[Bob_A_1]

@Tim: Cygwin is a non-starter; we make (literal) nuclear-grade code; we need native binaries for the systems we support. This isn't "linux" code; it's ANSI C and F2003 so presuming the compilers in question each support the language standards (importantly, the ISO C bindings from F2003), the code should just work. There are no OS dependencies so it should work on Linux, Windows, OSX, OpenVMS, whatever (unfortunately, I'm not joking about OpenVMS). gcc & ifort work together fine on Linux, gfortran & gcc work great on both Windows & Linux, and while I've managed to compile everything without errors using ifort and msvc on Windows, I'm hung up on the random, mystical combination of options and 'standard' libraries to use to get things to properly link. The correct answer seems to be use a compute cluster and combinatorical search to brute force the proper list of options and libraries.

@Repeat Offender: DLLs seem to be a needless complication and are probably a bad idea for this application for quality assurance purposes. The problem seems to boil down to:

a) how to convince msvc to make libraries for the correct architecture (ia32, x86) with all the symbols properly defined (MSVCRT seems to be the culprit and CL/LINK seem to gleefully ignore /NODEFAULTLIB directives; plenty of sources say "link against libcmt.lib" but provide zero working examples), and

b) how to convince ifort to compile for the correct architecture (again, ia32, x86), and to statically link against the libraries built in a)

I'd estimate 80% of the problem is due to a) but I can't tell if a) has succeeded until I attempt b).

As I said, both compilers are digesting the source code fine and emitting .obj files (read: the source code is not the problem), but despite telling both compilers to build for x86, something (probably the infernal linker) is mangling things into x64 mode. That, and the generated libraries end up referencing a bunch of undefined symbols like __strstr that are defined in standard libraries and have no business being either locally defined or unfound externalities. Half this behavior is implicitly set in environment variables where's it's effectively hidden state (e.g. via ifortvars.bat & vcvarsall.bat). It's virtually impossible to understand, let alone document and reproduce the build environment. So far I've been able to avoid issues with this for every combination of tools except MSVC and ifort. A lot is probably masked by the automated build tools, but I can at least get a detailed log of what steps the build tool took. So far it's been impossible to translate a working build process using gcc/gfortran to MSVC/ifort -- not the literal compiler flags, but the underlying steps. It's extremely frustrating because I know that ifort and MSVC are completely capable of doing what I need done, but there's a bottomless pit of hidden tribal knowledge on how to get things to properly link on Windows. Chasing down subtle nuances in gratuitous complexity is not how I want to spend my development effort; this shouldn't be so damned difficult. Yet it is.

[JVanB]

Ok, then with static linking:

! gflib.f90
! Compile with:
! gfortran -c gflib.f90
! ar cr libgflib.a gflib.o
module gfM
   use ISO_C_BINDING
   implicit none
   contains
      subroutine S(x,a,b,N) bind(C,name='gfS')
         integer(C_SIZE_T), value :: N
         real(C_DOUBLE) x(N)
         real(C_DOUBLE), value :: a, b
         real(C_DOUBLE) h
         integer(C_SIZE_T) i
         
         h = (b-a)/(N-1)
         x = BESSEL_J0([(a+h*i,i=0,N-1)])
      end subroutine S
end module gfM

! ifmain.f90
! Compile with:
! ifort ifmain.f90 libgflib.a
module ifM
   use ISO_C_BINDING
   implicit none
   interface
      subroutine S(x,a,b,N) bind(C,name='gfS')
         import
         implicit none
         integer(C_SIZE_T), value :: N
         real(C_DOUBLE) x(N)
         real(C_DOUBLE), value :: a, b
      end subroutine S
   end interface
end module ifM

program ifP
   use ifM
   implicit none
   integer(C_SIZE_T), parameter :: N = 10
   real(C_DOUBLE) x(N), y(N)
   real(C_DOUBLE) a, b
   real(C_DOUBLE) h
   integer(C_SIZE_T) i

   write(*,'(a,i0,a)') 'This is a ',bit_size(0_C_INTPTR_T),'-bit program.'
   a = 0
   b = 3
   h = (b-a)/(N-1)
   y = BESSEL_J0([(a+h*i,i=0,N-1)])
   call S(x,a,b,N)
   do i = 1, N
      write(*,*) x(i),y(i)
   end do
end program ifP

This is a 32-bit program.
   1.00000000000000        1.00000000000000
  0.972414529115098       0.972414529115098
  0.891937468132727       0.891937468132727
  0.765197686557967       0.765197686557967
  0.602566169772492       0.602566169772492
  0.417208867196137       0.417208867196137
  0.223890779141236       0.223890779141236
  3.764579426748669E-002  3.764579426748669E-002
 -0.127560987606650      -0.127560987606650
 -0.260051954901933      -0.260051954901933

Strange that both sides seem to be using ifort's BESSEL_J0 now. Maybe that's because gfortran MinGW uses MSVCRT whenever possible and ifort is also using MSVCRT?