mixed language: intrinsic functions missing at link time

[Jeremy_W_]

This is a mixed language C++ solution with several underlying fortran projects.  Builds on windows with VS2015 (toolset 140) and intel fortran and builds on mac and linux with GCC6 via makefiles.  Really want to build static executables for redistribution on mac, so I bought intel C++ and fortran for mac.  I've got all the supporting objects to build and all the .a files built with ar (with the same makefiles, just using ifort and icpc).  But I can't get the executable to link: looks like all of the fortran intrinsics are not found (here is the first of many):

Undefined symbols for architecture x86_64:
  "_for_adjustl", referenced from:
      _model_input_output_interface_mp_mio_store_instruction_set_ in libmio.a(mio.o)

What additional options do I need to invoke compile-and-link the executable? I'm sure its something simple, but I'm stumped.

Here are some makefile details:

ILIBDIR :='/opt/intel/lib/'

CFLAGS := '-pthread -std=c++11 -O2' 
FFLAGS := '-O2 -c -fpp -free'

LFLAGS := '-static -L$(ILIBDIR)'

make FC=${FC} CC=${CC} CXX=${CXX} CFLAGS=${CFLAGS} FFLAGS=${FFLAGS} INCLUDES=$(INCLUDES) LFLAGS=$(LFLAGS) LIBS=$(LIBS) ILIBDIR=$(ILIBDIR) LIBLDIR=$(LIBLDIR) LIBS=$(LIBS) -C programs/pest++ -f makefile_linux pestpp

Thanks in advance

[Kevin_D_Intel]

This may not be enough to get you over the hurdle, but you at least need to link libifcore.a to resolve that reference.

[Jeremy_W_]

Thanks Kevin!  

That got me further down the road - got all of the makefile-listed libraries linked, including mkl_blas and mkl_lapack. Now I getting "ld: library not found -lc++", which I tried to get past with -stdlib=libstdc++ and gcc6's libstdc++.a without success.  Any suggestions here? Is there some global option(s) I'm missing? 

Thanks again.

[mecej4]

You have not shown us the command used for linking, so I can only guess that you tried to link using icpc, icc or ld,  with a corresponding explicit rule in the makefile. The simplest solution is to use ifort to do the linking, with the -nofor_main option if the main program is not in Fortran.

[Kevin_D_Intel]

It is easier/recommended to drive the link as mecej4 writes using either icpc or icc or ifort. Using icc/icpc requires adding the needed Fortran libs. Using ifort adds the needed Fortran libs but requires adding linker options: -nofor-main -cxxlib  (Note: There's an alternative spelling of -nofor_main; -cxxlib includes C++ libs)

[Jeremy_W_]

Thanks for responses and sorry I didn't post the build command.  Here is the command that make ends up using to compiling and link the executable (I've attached the main make file):

icpc pest++.o -pthread -std=c++11 -O2 -Wl,-static,-L/opt/intel/compilers_and_libraries_2017.1.126/mac/compiler/lib/,-L/opt/intel/compilers_and_libraries_2017.1.126/mac/mkl/lib/,-L/Users/jwhite/Dev/pestpp/src/libs/opt/,-L/Users/jwhite/Dev/pestpp/src/libs/propack/,-L/Users/jwhite/Dev/pestpp/src/libs/pestpp_common/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/wrappers/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/yamr/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/serial,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/external/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/abstract_base/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/genie_wrapper/,-L/Users/jwhite/Dev/pestpp/src/libs/mio/,-L/Users/jwhite/Dev/pestpp/src/libs/common/,-L/Users/jwhite/Dev/pestpp/src/libs/linear_analysis/   -lopt -lpropack -lpestpp_com -lrm_wrappers -lrm_yamr -lrm_serial -lrm_external -lrm_genie_wrapper -lrm_abstract -lmio -lcommon -lmkl_lapack95 -lmkl_blas95 -llinear_analysis -o pestpp

Like I said, the underlying objects and libs (C++ and fortran) are compiling as is the executable, its the linking that's giving me trouble:

ld: library not found for -lc++

I thought since this is a C++ project, libc++ would be included implicitly?

[mecej4]

The makefile that you attached to #6 simply calls make again with -f makefile_linux for making pestpp. You need to look in makefile_linux, or possibly in another script used in a chained invocation of make,

The actual linking command was not issued directly based on the rules within makefile_mac_intel, so I cannot say any more about what went wrong.

[TimP]

As you indicated, icpc (linux) would link -lstdc++ automatically.  If the Mac icpc is based on clang++, does that possibly change this to -lc++ ?  If there is a bogus -lc++ link instruction, maybe it comes from your own link command (which seems to have things in an odd order).  I'm not sure whether icpc would interpret -pthread as -lpthread, but you could check this by adding -# to your link command build line.

In order to use ar and .a archives, you must take care that -ipo is never enabled in the Intel compilers.  If you do have any -ipo, you must use xiar in place of ar.  I suppose -ipo would not be compatible with any equivalent feature of clang++ (such as g++ -lto).

[Jeremy_W_]

Thanks for continuing to help me with this.  

Ok, so make_linux for pestpp is really simple:

OUT = pestpp
OBJECTS    := pest++.o

$(OUT): $(OBJECTS)
    $(CXX) $(OBJECTS) $(CFLAGS) $(LFLAGS) $(LIBLDIR) $(LIBS) -o $(OUT)

%.o: %.cpp
    $(CXX) $(CFLAGS) $(INCLUDES) $< -c $(input) -o $@

%.o: %.for
    $(FC) $(FFLAGS) $(INCLUDES) $< -c $(input) -o $@

clean:
    rm $(OBJECTS) $(OUT)

and the command I posted in #6 comes from stdout generated during the execution of makefile_mac_intel when trying to compile pestpp.o and link pestpp.o with the other objects and libs to yield the executable.  I'm not using $(LFLAGS) because I set the -Wl, option in the $(CFLAGS).  

[mecej4]

That makes it clear that you are using icpc for the link step, and explains why the Fortran RTL was not searched. Just follow Kevin Davis's prescription in #5 and you should be able to complete the linking.

[Jeremy_W_]

mecej4: I've gotten past the missing fortran RTL by adding the /opt intel compiler dir to the linker search path (see the -Wl option in #6 - thanks Kevin!), but now I'm stuck at missing libc++ (ld: library not found for -lc++).  Are you suggesting a separate link command using ifort after icpc compiles pestpp.o?

Kevin: I don't have any -ipo.  I'm still perplexed as to why the standard c++ lib is missing. "which seems to have things in an odd order" - what would be a more appropriate order?

[mecej4]

Jeremy W. wrote:

mecej4: I've gotten past the missing fortran RTL by adding the /opt intel compiler dir to the linker search path (see the -Wl option in #6 - thanks Kevin!), but now I'm stuck at missing libc++ (ld: library not found for -lc++).  Are you suggesting a separate link command using ifort after icpc compiles pestpp.o?

Yes. Replace CXX by FC in the fourth non-blank line of the makefile_linux in #9, and add -cxxlibs to LFLAGS. (I am going by what was written earlier, since I am not currently running Linux as I write).

[Jeremy_W_]

I tried the ifort link, but its still not seeing the fortran runtime libs.  Here is the stdout:

ifort pest++.o -Wl,-static,-L/opt/intel/compilers_and_libraries_2017.1.126/mac/compiler/lib/,-L/opt/intel/compilers_and_libraries_2017.1.126/mac/mkl/lib/,-L/Users/jwhite/Dev/pestpp/src/libs/opt/,-L/Users/jwhite/Dev/pestpp/src/libs/propack/,-L/Users/jwhite/Dev/pestpp/src/libs/pestpp_common/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/wrappers/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/yamr/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/serial,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/external/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/abstract_base/,-L/Users/jwhite/Dev/pestpp/src/libs/run_managers/genie_wrapper/,-L/Users/jwhite/Dev/pestpp/src/libs/mio/,-L/Users/jwhite/Dev/pestpp/src/libs/common/,-L/Users/jwhite/Dev/pestpp/src/libs/linear_analysis/ -cxxlib  -lopt -lpropack -lpestpp_com -lrm_wrappers -lrm_yamr -lrm_serial -lrm_external -lrm_genie_wrapper -lrm_abstract -lmio -lcommon -lmkl_lapack95 -lmkl_blas95 -llinear_analysis -o pestpp
ld: library not found for -lifportmt​

I've verified that libifportmt.a and libifportmt.dylib exist in opt/intel/compilers_and_libraries_2017.1.126/mac/compiler/lib/.  Are there any scripts that need to be executed after compiler installation?

Sorry I'm such amatuer with this stuff - I appreciate your help.

[mecej4]

At this point someone else who is familiar with OSX is needed to suggest what to try next. I am not at all familiar with the Mac customizations of the linker ld.

[Kevin_D_Intel · Accepted solution]

That was TimP writing earlier, not me. He's correct about using xiar if you introduce IPO. I didn't mention it earlier to avoid introducing other confusion but it is an important point if you introduce IPO later on.

The -lc++ error appears related to a lack of support for static linking on OS X from all I’ve read, although I'm no expert on it. The issue occurs even with g++ where one forces static linking via ld with -Wl,-static

$ cat h.cpp
#include <iostream>

int main() {
    std::cout << "Testing..." << std::endl;
    return 0;
}

$ g++ h.cpp && ./a.out
Testing...
$ otool -L a.out
a.out:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

$ g++ -static h.cpp
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ g++ -Wl,-static h.cpp
ld: library not found for -lc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)

There is a lot of discussion around static linking on OS X on stackoverflow. Google will locate many but this is just one. http://stackoverflow.com/questions/844819/how-to-static-link-on-os-x

All seem to lead to the same conclusion that static linking isn't recommended for OS X but I can't speak to that with any authority.

It seems for your interest the best approach might be to explicitly link static libraries using our compiler option (-static-intel) and explicit paths to other libs needed because throwing -Wl,-static will seemingly never overcome the missing -lc++ error from what I can determine. From my investigation it seems ld remains in static link mode when using -Wl,-static and there’s no means of switching it back with say -Wl,-dynamic, thus, you can never overcome the -lc++ error. The only success I had was linking with icpc with the fullpath to libifcore.a.

Assuming you only have the .a version of your libs in those various jwhite sub-directories, then you may be able to avoid throwing -Wl,-static and instead maybe something like this may work for you:

icpc pest++.o -pthread -std=c++11 -O2 -mkl -static-intel -o pestpp -L/Users/jwhite/Dev/pestpp/src/libs/opt/ -L/Users/jwhite/Dev/pestpp/src/libs/propack/ -L/Users/jwhite/Dev/pestpp/src/libs/pestpp_common/ -L/Users/jwhite/Dev/pestpp/src/libs/run_managers/wrappers/ -L/Users/jwhite/Dev/pestpp/src/libs/run_managers/yamr/ -L/Users/jwhite/Dev/pestpp/src/libs/run_managers/serial -L/Users/jwhite/Dev/pestpp/src/libs/run_managers/external/ -L/Users/jwhite/Dev/pestpp/src/libs/run_managers/abstract_base/ -L/Users/jwhite/Dev/pestpp/src/libs/run_managers/genie_wrapper/ -L/Users/jwhite/Dev/pestpp/src/libs/mio/ -L/Users/jwhite/Dev/pestpp/src/libs/common/ -L/Users/jwhite/Dev/pestpp/src/libs/linear_analysis/  -lopt -lpropack -lpestpp_com -lrm_wrappers -lrm_yamr -lrm_serial -lrm_external -lrm_genie_wrapper -lrm_abstract -lmio -lcommon  -llinear_analysis -llinear_analysis /opt/intel/compilers_and_libraries_2017.1.126/mac/compiler/lib/libifcore.a /opt/intel/compilers_and_libraries_2017.1.126/mac/mkl/lib/libmkl_lapack95.a /opt/intel/compilers_and_libraries_2017.1.126/mac/mkl/lib/libmkl_blas95.a

Note: Don't use -Wl at all above. Just allow our driver to pass all the -L and -l options and other static .a libs to the linker.

[Jeremy_W_]

Thanks Kevin - that was the ticket.  I had to provide a complete path to all project libs and a few additional mkl libs (sequential and core), doing away with -Wl and -l options and instead just providing the path to all the static (*.a) libs, including the "lib" in the name and the ".a" extension. I'm guessing (as you speculated) icpc sorted out the options and send the correct ones to ld.  

Much appreciation to all that help!

[Kevin_D_Intel]

You're welcome. Glad that helped.