Intel® oneAPI Math Kernel Library
Ask questions and share information with other developers who use Intel® Math Kernel Library.

Yet another tale of LAPACK95 linker woe ("undefined reference")

Bob_A_1
Beginner
610 Views

Once again, Eclipse has died wiping out the magical settings I used to link my Fortran code against the LAPACK95 libraries. I've resuscitated Eclipse but I'm back where I was 8 weeks ago poking futilely at the Link Advisor and Eclipse's compile and link options. (i.e. this is a linker options problem, not an Eclipse problem.)

The environment is Linux Mint, 64-bit. Before running Eclipse, environment variables are set with:

source /opt/intel/bin/compilervars.sh intel64
source /opt/intel/mkl/bin/mklvars.sh intel64 mod

This sets MKLROOT to

/opt/intel/composer_xe_2013.2.146/mkl

and LD_LIBRARY_PATH to

/opt/intel/composer_xe_2013.2.146/compiler/lib/intel64:/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64:/opt/intel/composer_xe_2013.2.146/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.2.146/mpirt/lib/intel64:/opt/intel/composer_xe_2013.2.146/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.2.146/ipp/lib/intel64:/opt/intel/composer_xe_2013.2.146/compiler/lib/intel64:/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64:/opt/intel/composer_xe_2013.2.146/tbb/lib/intel64

This is not a new configuration; it's been a stable working setup for months. Well, except for Eclipse occasionally starting on fire.

I used the Link Advisor to find the options to create a dynamically-linked binary with 32-bit integers (lp64) for the Intel64 architecture.

The last relevant bit of console output before the build process dies is:

Building file: ../src/fate.f90
Invoking: Intel(R) Intel(R) 64 Fortran Compiler
ifort -g -O0 -fpp -DDEBUG -warn declarations -warn unused -warn uncalled -ftrapuv -save -fpe0 -fp-model source -traceback -c -pg -Ddebug -stand -heap-arrays -I/opt/intel/composer_xe_2013.2.146/mkl/include/intel64/lp64 -I/opt/intel/composer_xe_2013.2.146/mkl/include -axSSE4.1 -o "src/fate.o" "../src/fate.f90"
Finished building: ../src/fate.f90

Building target: F2064a
Invoking: Intel(R) Fortran Linker
ifort -L/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64 /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_blas95_lp64.a /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_lapack95_lp64.a -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -pg -o "F2064a" <giant redacted list of object files>
./src/m_numcon.o: In function `dsolv':
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1491: undefined reference to `dgesvx1_mkl95_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1506: undefined reference to `dgesv1_mkl95_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1521: undefined reference to `dgetrf_mkl95_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1527: undefined reference to `dgetrs1_mkl95_'
make: *** [F2064a] Error 1

I get the same results whether the options are put before the source files or after.

I tried again, this time using the suggested options for creating a statically-linked binary.

Similarly, the last relevant bit of console output before the build process dies is:

Building file: ../src/fate.f90
Invoking: Intel(R) Intel(R) 64 Fortran Compiler
ifort -g -O0 -fpp -DDEBUG -warn declarations -warn unused -warn uncalled -ftrapuv -save -fpe0 -fp-model source -traceback -c -pg -Ddebug -stand -heap-arrays -I/opt/intel/composer_xe_2013.2.146/mkl/include/intel64/lp64 -I/opt/intel/composer_xe_2013.2.146/mkl/include -axSSE4.1 -o "src/fate.o" "../src/fate.f90"
Finished building: ../src/fate.f90

Building target: F2064a
Invoking: Intel(R) Fortran Linker
ifort /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_blas95_lp64.a /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -pg -o "F2064a" <same giant list of redacted object files>
./src/m_numcon.o: In function `tridag':
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:695: undefined reference to `dgtsv_'
./src/m_numcon.o: In function `dsolv_sparse':
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1230: undefined reference to `dss_create_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1239: undefined reference to `dss_define_structure_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1249: undefined reference to `dss_reorder_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1258: undefined reference to `dss_factor_real_d_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1267: undefined reference to `dss_solve_real_d__'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1315: undefined reference to `dss_delete_'
./src/m_numcon.o: In function `dsolv':
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1462: undefined reference to `dgesvx_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1477: undefined reference to `dgesv_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1491: undefined reference to `dgesvx1_mkl95_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1506: undefined reference to `dgesv1_mkl95_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1521: undefined reference to `dgetrf_mkl95_'
/home/apthorpe/workspace/F2064a/Debug_Intel64/../src/m_numcon.f90:1527: undefined reference to `dgetrs1_mkl95_'
make: *** [F2064a] Error 1

I know what I'm missing is small but I cannot for the life of me remember what byzantine and unmemorable bit of configuration needs to go where, and it's nothing that a cursory search could find. I'm at the end of my rope with this so any help would be greatly appreciated.

To summarize:

1) A solution exists. This was working fine until a few hours ago

2) This is a problem with linker options, not with the IDE

3) I've tried using the Link Advisor suggestions for several different methods of linking against LAPACK95 and none of them work

Thanks much,

-- Bob

Aside: I'd have hoped by now that dev tools could scan the filesystem and sensibly sort out their dependencies by now; dynamic languages already do a pretty good job of this. It doesn't seem that compilers or linkers have gotten any smarter in the past 25 years or so, at least when it comes to chasing down libraries or include files. There is a reasonably small number of places these files can be and the compiler/linker should have a fair idea of what combinations of libraries are compatible with each other, especially those supplied by a vendor. A naive scan of my filesystem shows roughly 7200 shared objects and 500 static libraries. Is it really that Herculean of an effort to scan the files with find, nm, etc. and suggest compiler options or provide better guidance that "couldn't find it, didn't look too hard, gave up"? I'm not arguing that we should be ignorant of our toolset but I'd really rather be working on my code than playing "bring me a rock" with the linker (again). Or hosing down the charred remains of my IDE.

0 Kudos
9 Replies
mecej4
Honored Contributor III
610 Views

Building target: F2064a
Invoking: Intel(R) Fortran Linker
ifort -L/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64 /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_blas95_lp64.a

is wrong. The -L option specifies the directory wherein optional libraries are to be searched during linking. The -l option (lower case L) is for specifying which libraries to search. Your command line specifies where to search, but the list of optional libraries is null.

0 Kudos
Zhang_Z_Intel
Employee
610 Views

Bob,

In the step "Invoking: Intel(R) Fortran Linker", please try putting your <giant list of redacted object files> before the -L option. That is,

ifort <giant list of redacted object files> -L/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64 /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_blas95_lp64.a /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_lapack95_lp64.a -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -pg -o "F2064a"

I think this should take care of the problem. Let us know.

0 Kudos
Bob_A_1
Beginner
609 Views

mecej4 wrote:

Quote:

Building target: F2064a
Invoking: Intel(R) Fortran Linker
ifort -L/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64 /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_blas95_lp64.a

is wrong. The -L option specifies the directory wherein optional libraries are to be searched during linking. The -l option (lower case L) is for specifying which libraries to search. Your command line specifies where to search, but the list of optional libraries is null.

Maybe it's not clear from the formatting but further down the command line is the fragment

 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm

which is where the individual libraries are specified

0 Kudos
mecej4
Honored Contributor III
609 Views

Maybe it's not clear from the formatting but further down the command line is the fragment

 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm

It was not clear. The libraries listed in this fragment do not include the Lapack-95 or Blas-95 routines. To pull them in, you need to specify, in addition, -lmkl_lapack95_lp64 and/or -lmkl_blas95_lp64.

0 Kudos
Bob_A_1
Beginner
610 Views

Zhang Z (Intel) wrote:

Bob,

In the step "Invoking: Intel(R) Fortran Linker", please try putting your <giant list of redacted object files> before the -L option. That is,

ifort <giant list of redacted object files> -L/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64 /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_blas95_lp64.a /opt/intel/composer_xe_2013.2.146/mkl/lib/intel64/libmkl_lapack95_lp64.a -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -pg -o "F2064a"

I think this should take care of the problem. Let us know.

Counterintuitively reordering the options after the arguments seems to work when manually entered from the command line so in a very limited scope the problem is solved. I had seen this solution mentioned a few times before and I still can't understand why it makes any difference. Options should be position-independent just as a matter of human interface design; at least that's how options seem to work in my code and most of the other CLI code I use. Plus, this seems to happen often enough that putting a few weeks of intern coding time on improving the linker's option handling would save a lot of suffering in the long term. Believe me, I hate asking these questions as much as you probably hate answering them. Still, don't think I'm ungrateful; I really do appreciate the quick and clear response and please take my criticisms as suggestions to improve the products, not just as frustrated venting. Overall I've been very happy with the compiler; it's just dealing with the archaic and brittle linking process (not unique to Intel's linker FWIW) that drives me nuts. 

The second problem (not relevant here) is that I can't convince Eclipse/Photran to put linker options after the arguments. But at least I know what the solution to this problem is, even if I can't get the IDE to cooperate.

Thanks again,

-- Bob

0 Kudos
mecej4
Honored Contributor III
610 Views

Bob A. wrote:
 I still can't understand why it (order) makes any difference

It may help to recognize that some of the options are compiler options and some are linker options. Often, the compiler driver simply passes the linker options to the linker after it is done with compiling source files. IFort uses the system linker ld, which has well established rules according to which the order of some arguments matters.

For example:all.o files specified are used; libraries are searched in the order specified and, if a symbol occurs in more than one library, the first instance is used. Libraries are searched only for unsatisfied symbols. Libraries specified on the command line are searched before the implicitly specified system libraries are searched. There are good reasons for why things are this way.

0 Kudos
Bob_A_1
Beginner
610 Views

As a follow-up for other Eclipse/Photran users, the workaround for this issue is to go into

Project > Preferences > Settings > Tool Settings > Intel Fortran Linker > Command Line Pattern

and change it from the default of:

${COMMAND} ${FLAGS} ${OUTPUT_FLAG} ${OUTPUT_PREFIX}${OUTPUT} ${INPUTS}

to:

${COMMAND} ${INPUTS} ${FLAGS} ${OUTPUT_FLAG} ${OUTPUT_PREFIX}${OUTPUT}

to order commands in the special way the Intel linker expects.

Hopefully this will avert further frustration,

-- Bob

0 Kudos
Bob_A_1
Beginner
610 Views

mecej4 wrote:

Quote:

Bob A. wrote: I still can't understand why it (order) makes any difference

It may help to recognize that some of the options are compiler options and some are linker options. Often, the compiler driver simply passes the linker options to the linker after it is done with compiling source files. IFort uses the system linker ld, which has well established rules according to which the order of some arguments matters.

For example:all.o files specified are used; libraries are searched in the order specified and, if a symbol occurs in more than one library, the first instance is used. Libraries are searched only for unsatisfied symbols. Libraries specified on the command line are searched before the implicitly specified system libraries are searched. There are good reasons for why things are this way.

Ugh. So the linking operation is actually a conflation of what should be separate compilation and linking steps. And rather than clearly and explicitly separating those steps, the onus is put on end users and third-party tool developers to understand the rationale behind this counterintuitive behavior and work around it. This makes the behavior understandable in an evolutionary sense but I think it's a stretch to call this a "good" reason. :/

0 Kudos
mecej4
Honored Contributor III
610 Views

Whether or not you or I find the behavior of the linker ld intuitive, the behavior is documented and appears to be satisfactory to the majority of users.  Intuition is conditioned by experience, so something that is counterintuitive to one person may be obvious to another.

0 Kudos
Reply