I'm in the process of porting an application from Windows over to Linux, and have written my Makefile as follows:
INTEL_LIBS=/opt/intel/composer_xe_2013_sp1/lib/intel64 MKLROOT=/opt/intel/composer_xe_2013_sp1.3.174/mkl SRC_PATH = ../FortranSource SOURCES_OBJ = $(OBJECT_FOLDER)/BLAS.o \ #... and many others DEBUG_FLAGS = -g -debug parallel -debug-parameters all -warn declarations \ -auto -traceback -check pointer -check bounds -check uninit -check format -check output_conversion IFORT_FLAGS = $(DEBUG_FLAGS) -nologo -fpp -DUSE_MKL -recursive -reentrancy threaded -openmp -align rec8byte \ -threads -fp-model source -fp-model except -c -I $(MKLROOT)/include/intel64/lp64 -I $(MKLROOT)/include DBG_LIBS = -lirc -lsvml -limf $(TARGETFOLDER)/libintlc.so.5 $(TARGETFOLDER)/nCutilsd.so LINKLIBS = $(MKLROOT)/lib/intel64/libmkl_blas95_lp64.a $(MKLROOT)/lib/intel64/libmkl_lapack95_lp64.a \ --start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_core.a \ $(MKLROOT)/lib/intel64/libmkl_sequential.a --end-group -lpthread -lm $(DBG_LIBS) LINKFLAGS = $(DBG_FLAGS) LINKLIBS_PATH = -L $(INTEL_LIBS) F90 = /opt/intel/bin/ifort LINK_CMD=/opt/intel/bin/xild OBJECT_FOLDER = Debug TARGETFOLDER = ../../Builds TARGETNAME = nCored $(OBJECT_FOLDER)/%.o : $(SRC_PATH)/%.f90 $(F90) $< $(IFORT_FLAGS) -I $(MKLROOT)/include -I $(MKLROOT)/include/intel64/lp64 -o $@ -module $(OBJECT_FOLDER) $(OBJECT_FOLDER)/%.o : $(SRC_PATH)/%.F90 $(F90) $< $(IFORT_FLAGS) -I $(MKLROOT)/include -I $(MKLROOT)/include/intel64/lp64 -o $@ -module $(OBJECT_FOLDER) .PHONY: all clean all: $(OBJECT_FOLDER) $(MAKE) $(TARGETFOLDER)/nCored.so $(OBJECT_FOLDER) : mkdir -p $(OBJECT_FOLDER) clean: rm -rf Debug cd $(TARGETFOLDER) && rm -rf $(TARGETNAME).so $(TARGETFOLDER)/nCored.so : $(OBJECT_FOLDER)/nCore.o $(LINK_CMD) -o $@ $(SOURCES_OBJ) $(DBGLIBS) $(LINKLIBS) $(LINKLIBS_PATH) # The makefile ends with a long table of dependencies between my various modules...
When I try to build, all of my object files and module files build as expected, but during the link step I get literally thousands of error messages which are all of the pattern:
/home/Eos/Development/Makefiles/../FortranSource/Cal_Fitting.f90:666: undefined reference to `for__rtc_uninit_use'
...where the reference can be many different things. It seems self-evident that I've missed out some reference to the compiler's own libraries or that there is a path which is not defined, but I cannot find any ideas from the documentation for what this may be. Running:
...from the command line before typing 'make' doesn't seem to help, nor indeed does inserting the same line into the makefile itself on the line before $(LINK_CMD) - even though when I do the latter, the output from 'make' shows that the shell script is indeed being run.
So, please may I have some advice about what it is that I've missed out?
I have little experience with MKL, but I suspect that is the root of your issue. Have you tried using the MKL Link Line Advisor to help you come up with the correct flags/libs/etc? https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
What about the mklvars.sh script? Have you sourced that with the appropriate arguments? (https://software.intel.com/en-us/node/438548)
Also, is there a reason you are calling xild explicitly to link the shared object rather than using ifort?
It is a bit tough to debug only seeing the makefile; a copy of the link line could be useful. In general the more info available to trace and reproduce the problem the better (i.e., OS—including distro, architecture, ifort version, MKL version, compile/link lines where the problem occurs and the relevant error, any other info required to reproduce the problem)
Anyway, we’re starting to reach the limits of my usefulness on this matter, if you try the mklvars.sh script and also check https://software.intel.com/en-us/node/438542 to make sure everything else is set up correctly and it still is broken, then hopefully one of the folks at Intel will chime in. Also, it might be worth ensuring that MKL is correctly installed by attempting to build one of the examples distributed with MKL, discussed in the last link.
My hunch is that there is something wrong in your environment which is causing the linker to either not find the appropriate libraries at all, or to shadow a library with a different one. Output of the `env` command might be informative as well, to check variables like LD_LIBRARY_PATH etc.
Ah, converting from Windows to Linux. OK.
The undefined symbol you showed is from one of the Fortran runtime libraries (not MKL). If the other undefined symbols also start with the letters "for_" or perhaps "for__" then those, too, are from one of the Fortran runtime libraries.
One difference between Windows and Linux is that on Windows we are able to embed in the object file a request to LINK.EXE to bring in the Fortran runtime libraries. That is why it is possible to simply LINK FORTFILE.OBJ and have FORTFILE.EXE be created with all the Fortran runtime references resolved.
On Linux, either you need to list all the libraries yourself or you can do as suggested by Zaak, and use the verb "ifort" instead of "xild" to do the link stage. The ifort executable knows it needs to add the Fortran libraries and should also allow you to have all the other options listed here.
I hope this helps --
I concur with Zaak. You need to drive the link to produce the shared library using ifort with the -shared option. I also expect you need to compile with -fPIC. I can reproduce many unresolved externals related to the Fortran RTLs including for__rtc_uninit_use and attempting the link using xild directly. By using ifort to link you will not need to provide paths to the compiler libraries just the MKL libs.
Thank you Zaak, Lorri and Kevin.
I didn't think this was anything to do with the MKL, but I was about to embark on the task of commenting-out all my MKL dependencies just to check. I'm glad there's no need to do that now!
As for linking with ifort, I tried that yesterday: just replacing the definition of LINK_CMD in my makefile with /opt/intel/bin/ifort. The problem I encountered was that ifort didn't know what to do with the --start-group and --end-group definitions in the link options, so I got the errors:
ifort: command line warning #10006: ignoring unknown option '-fstart-group' ifort: command line warning #10006: ignoring unknown option '-fend-group'
... followed by a couple of hundred screenfuls of 'undefined reference' messages which this time definitely did relate to the MKL!
So far I have failed to find an option in ifort equivalent to gcc's "-Wl,", for specifying options that should be ignored by the compiler but passed on to the linker.
To do that you need to use multiple -Wl commands. Something of the form: -Wl,--start-group,libA.a,libB.a -Wl,--end-group
So for your LINKLIBS variable, try this:
LINKLIBS = $(MKLROOT)/lib/intel64/libmkl_blas95_lp64.a $(MKLROOT)/lib/intel64/libmkl_lapack95_lp64.a \ -Wl,--start-group,$(MKLROOT)/lib/intel64/libmkl_intel_lp64.a,$(MKLROOT)/lib/intel64/libmkl_core.a,\ $(MKLROOT)/lib/intel64/libmkl_sequential.a -Wl,--end-group -lpthread -lm $(DBG_LIBS)
I inserted -Wl in two places and added commas to the list of options associated with the -Wl option list.
Aha! Now we're making progress. It didn't occur to me (although it obviously should have done) that ifort would take exactly the same switches as gcc in this context!
I'm nearly out of the woods; now I'm just getting 'undefined reference' errors of the kind:
/home/Eos/Development/Makefiles/../FortranSource/Workspaces.F90:1820: undefined reference to `__kmpc_for_static_init_4'
...which look like they relate to a missing OpenMP library somewhere.
Thank you very much Kevin. The __kmp messages have gone away.
The next couple of hundred screenfuls are on the theme of:
/home/Eos/Development/Makefiles/../FortranSource/Calibration.F90:63: undefined reference to `__PDBX_isGuardedCall'
...and many other messages which all begin with __PDBX_. This looks debug-related, and sure enough they go away if I remove the:
-g -debug parallel -debug-parameters all
...from the compiler flags. However, I'd like to be able to do this compilation with debugging information included to help track down any problems on the new platform. Do you know what I need to add to enable this?
it's related to the "-debug parallel" switch.
Can you add that to your 'link' step; ie, pass that switch to ifort?
That way ifort will know to link against the libpdbx* libraries.
Yes, that worked; thank you Lorri.
I'm more or less out of the woods, now. I've got a few more errors relating to the import of functions from some other C libraries that I wrote myself, but that's just a question of Linux semantics. I'll try and close those down over the next few days, and once I'm done I'll record my final makefile here for posterity.
Here as promised is the makefile that successfully built my library:
INTEL_LIBS=/opt/intellib/intel64 MKLROOT=/opt/intel/mkl SRC_PATH = ../FortranSource SOURCES_OBJ = $(OBJECT_FOLDER)/BLAS.o \ # …and many others DEBUG_FLAGS = -g -debug parallel -debug-parameters all -warn declarations \ -auto -traceback -check pointer -check bounds -check uninit -check format -check output_conversion IFORT_FLAGS = $(DEBUG_FLAGS) -nologo -fpp -fPIC -DUSE_MKL -recursive -reentrancy threaded -openmp -align rec8byte -align array64byte \ -threads -fp-model source -fp-model except -c -I $(MKLROOT)/include/intel64/lp64 -I $(MKLROOT)/include # My own C library DBG_LIBS = -lnCutilsd NODBG_LIBS = -lnCutils LINKLIBS = $(MKLROOT)/interfaces/blas95/pic/lib/intel64/libmkl_blas95_lp64.a \ $(MKLROOT)/interfaces/lapack95/pic/lib/intel64/libmkl_lapack95_lp64.a \ -Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a \ $(MKLROOT)/lib/intel64/libmkl_core.a \ $(MKLROOT)/lib/intel64/libmkl_sequential.a \ -Wl,--end-group -liomp5 -lpthread -lm $(DBG_LIBS) LINKFLAGS = -shared -debug parallel -Wl,-rpath,/usr/local/lib/Intel -Wl,-rpath,$$ORIGIN $(DBG_FLAGS) LINKLIBS_PATH = -L $(INTEL_LIBS) -L $(TARGETFOLDER) F90 = /opt/intel/bin/ifort LINK_CMD=/opt/intel/bin/ifort # Defaults, may be modified for different cases OBJECT_FOLDER = Debug TARGETFOLDER = ../../Builds TARGETNAME = libnCored NCUTILS_LIB = $(TARGETFOLDER) # We need rules for both .f90 and .F90 files $(OBJECT_FOLDER)/%.o : $(SRC_PATH)/%.f90 $(F90) $< $(IFORT_FLAGS) -I $(MKLROOT)/include -I $(MKLROOT)/include/intel64/lp64 -o $@ -module $(OBJECT_FOLDER) $(OBJECT_FOLDER)/%.o : $(SRC_PATH)/%.F90 $(F90) $< $(IFORT_FLAGS) -I $(MKLROOT)/include -I $(MKLROOT)/include/intel64/lp64 -o $@ -module $(OBJECT_FOLDER) .PHONY: all clean all: $(OBJECT_FOLDER) $(MAKE) $(TARGETFOLDER)/libnCored.so TARGETNAME=libnCored OBJECT_FOLDER=Debug $(OBJECT_FOLDER) : mkdir -p $(OBJECT_FOLDER) clean: rm -rf Debug rm -rf Release cd $(TARGETFOLDER) && rm -rf $(TARGETNAME).so $(TARGETFOLDER)/libnCored.so : $(OBJECT_FOLDER)/nCore.o $(LINK_CMD) -o $@ $(SOURCES_OBJ) $(DBGLIBS) $(LINKFLAGS) $(LINKLIBS) $(LINKLIBS_PATH) # The makefile ends with a long table of dependencies between my different modules
There was a final 'gotcha' with respect to the MKL. Because what I'm writing here is a shared library that will be called from a main application written in C and Java, all of the libraries that I linked into it needed either to be shared libraries themselves or, if static, needed to have been built with the -fPIC option. This was not true of the libmkl_blas95_lp64.a and libmkl_lapack95_lp64.a libraries that came pre-built from Intel (although it was for the other MKL static libraries); neither are shared versions of these libraries provided. Therefore, with support from Intel via this post in the MKL forum, I had to rebuild these MKL static libraries after manually editing the Intel-supplied makefiles - slightly scary, but ultimately successful.