known-good code fails to link using mpif90

[tlroche]

summary: mpif90 fails to link known-good code using icc/ifort 11.1 on 64-bit RHEL 5. Errors suggest lack of access to Intel libraries. How to fix?

details:

I am attempting to build CMAQ, an air-quality model widely used by the US Environmental Protection Agency (et al), on topsail, a Research Computing cluster @ UNC-Chapel Hill:

https://help.unc.edu/6214
> Login node : topsail.unc.edu 8 CPUs @ 2.3 GHz Intel EM64T with 2x4M
> L2 cache (Model E5345/Clovertown), 12 GB memory

> Compute nodes: 520 blade servers, each with 2 quad-core 2.3 GHz
> Intel EM64T processors, 2x4M L2 cache (Model E5345/Clovertown), and
> 12 GB memory

The compute nodes run RHEL 5, are managed with LSF, and show

[tr@topsail-login1 ~]$ bsub -q int -Ip uname -imprv
...
> 2.6.18-128.7.1.el5 #1 SMP Wed Aug 19 04:00:49 EDT 2009 x86_64 x86_64 x86_64

I am configured to run

[tr@topsail-login1 ~]$ bsub -q int -Ip icc --version
> ...
> icc (ICC) 11.1 20090827
[tr@topsail-login1 ~]$ bsub -q int -Ip ifort --version
> ...
> ifort (IFORT) 11.1 20090827

CMAQ builds widely in part because it supplies its own build scripts. I'm trying to run one (scripts/jproc/bldit.jproc), which first checks out fortran sources from a CVS repository, then compiles them. bldit.jproc is known to work on other platforms, but has not (to my knowledge--ICBW) been run on topsail. My bldit.jproc on topsail is able to check out and compile, but linking the built objects fails:

bldit.jproc runtime log
> /usr/mpi/intel/mvapich-1.1.0/bin/mpif90 -Bstatic calczen.o chj.o index2.o intavg.o interp.o jproc.o o3scal.o optics.o pntavg.o readcsqy.o readet.o reado2.o reado3.o readprof.o readtoms.o setaer.o setair.o setalb.o setcld.o srband.o subgrid.o tridiag.o twostr.o -L/ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort -lioapi -L/ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/netCDF/Linux2_x86_64ifort -lnetcdf -o JPROC_d1a_Linux2_x86_64intel
>
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(m3msg2.o): In function `m3msg2_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x29): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x40): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0xa4): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(m3msg2.o): In function `m3mesg_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x109): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x120): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x184): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(m3msg2.o): In function `m3prompt_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x208): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x21f): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x2f2): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(m3msg2.o): In function `m3parag_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x43b): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x451): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x563): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(m3msg2.o): In function `m3flush_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x5ab): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x5c2): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/m3msg2.F:(.text+0x5e0): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(shut3.o): In function `shut3_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/shut3.F:(.text+0x2a): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/shut3.F:(.text+0x40): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/shut3.F:(.text+0x20c): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(init3.o): In function `init3_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/init3.F:(.text+0x78): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/init3.F:(.text+0x91): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/init3.F:(.text+0x107): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/Linux2_x86_64ifort/libioapi.a(rdvars.o): In function `rdvars_':
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0x3d): undefined reference to `__kmpc_global_thread_num'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0x76): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0xc2): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0x2de): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0x32f): undefined reference to `__kmpc_end_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0x5ef): undefined reference to `__kmpc_critical'
> /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/lib/ioapi/ioapi/rdvars.F:(.text+0x686): undefined reference to `__kmpc_end_critical'
> /usr/lib64/libpthread.a(pthread_create.o): In function `pthread_create':
> (.text+0xe31): undefined reference to `_dl_stack_flags'
> /usr/lib64/libpthread.a(pthread_create.o): In function `pthread_create':
> (.text+0x1331): undefined reference to `_dl_stack_flags'
> /usr/lib64/libpthread.a(ptw-write.o): In function `__write_nocancel':
> (.text+0x18): undefined reference to `__syscall_error'
> /usr/lib64/libpthread.a(ptw-write.o): In function `__write_nocancel':
> (.text+0x6e): undefined reference to `__syscall_error'
> /usr/lib64/libpthread.a(ptw-read.o): In function `__read_nocancel':
> (.text+0x18): undefined reference to `__syscall_error'
> /usr/lib64/libpthread.a(ptw-read.o): In function `__read_nocancel':
> (.text+0x6e): undefined reference to `__syscall_error'
> /usr/lib64/libpthread.a(ptw-close.o): In function `__close_nocancel':
> (.text+0x18): undefined reference to `__syscall_error'
> /usr/lib64/libpthread.a(ptw-close.o):(.text+0x5a): more undefined references to `__syscall_error' follow
> /usr/lib64/libpthread.a(init.o): In function `__pthread_initialize_minimal':
> (.text+0x145): undefined reference to `__libc_setup_tls'
> /usr/lib64/libpthread.a(init.o): In function `__pthread_initialize_minimal':
> (.text+0x18a): undefined reference to `_dl_cpuclock_offset'
> /usr/lib64/libpthread.a(init.o): In function `__pthread_initialize_minimal':
> (.text+0x35d): undefined reference to `_dl_init_static_tls'
> /usr/lib64/libpthread.a(init.o): In function `__pthread_initialize_minimal':
> (.text+0x368): undefined reference to `_dl_wait_lookup_done'
...
> mv JPROC_d1a_Linux2_x86_64intel /ifs1/home/tr/scr/CMAQ/CMAQ_4.7/scripts/jproc
> mv: cannot stat `JPROC_d1a_Linux2_x86_64intel': No such file or directory

A CMAQ community member who writes/maintains an important component states

> A misconfigured "mpif90" is the problem: it knows nothing about the
> Intel libraries, in this case "libomp"

I'd appreciate your assistance: does that diagnosis make sense? If so, how to fix it? If not, is there a better diagnosis? Note that I have only user rights on topsail, but will forward suggestions to the admins.

TIA, Tom Roche

[Ron_Green]

Yes, that makes perfect sense - the kmp_ symbols are part of the OpenMP runtime library, libiomp5.a.

If you build the code on a cluster node using LSF you have to make sure that the cluster nodes set the environment to find the compiler libraries. The first command in your batch script should be to source the ifortvars.sh and/or iccvars.sh scripts:


source /opt/intel/Compiler/11.1/072/ifortvars.sh intel64
source /opt/intel/Compiler/11.1/072/iccvars.sh intel64


You can test this with a simple batch command to dump the environment

bsub -q int -Ip env

and look for LD_LIBRARY_PATH, PATH, INCLUDE. They need to have the paths to your compiler:
PATH=/opt/intel/Compiler/11.1/072/bin/intel64: for example. Your path may vary, as your sysadmins may have some root other than "/opt/intel"

Also, I would recommend compiling with -static-intel to the compiler/link options so that you statically put the Intel runtime libs into your executable.

AND, a little advice: K.I.S.S., if you know what that means. Don't start out trying to build some monster code. First try to build a hello world using mpif90 on the cluster nodes. Succeed there first then worry about building the big application.

ron

[tlroche]

Thanks for your suggestions--I'm about to embark upon them. Just to clarify:

Ronald W. Green (Intel) wrote:
> Don't start out trying to build some monster code.

Well, CMAQ *is* monster code, but it is somewhat componentized, and I've been building it following its README.txt. JPROC is fairly far down the chain, and I have been exercising mpif90 along the way. OTOH, this process has also found some topsail config problems (e.g.) and "undocumented features," and may find more ...

[tlroche]

Ronald W. Green (Intel) April 27, 2010 7:01 PM PDT (rearranged)
> kmp_ symbols are part of the OpenMP runtime library, libiomp5.a.

> If you build the code on a cluster node using LSF you have to make sure that the cluster nodes set the environment to find the compiler libraries. [...] You can test this with a simple batch command to dump the environment

> bsub -q int -Ip env

> and look for LD_LIBRARY_PATH, PATH, INCLUDE.

I get

[tr@topsail-login1 ~]$ bsub -q int -Ip "echo -e ${LD_LIBRARY_PATH} | tr ':' '\n' | sort | uniq | xargs -i ls -al {}/libiomp5.a 2> /dev/null"
> -rw-rw-r-- 1 root root 1056924 Aug 27 2009 /ifs1/opt/intel-11.1/cc/lib/intel64/libiomp5.a
> -rw-rw-r-- 1 root root 1056924 Aug 27 2009 /ifs1/opt/intel-11.1/fortran_11.1.056/lib/intel64/libiomp5.a
> -rw-rw-r-- 1 root root 1056924 Aug 27 2009 /ifs1/opt/intel-11.1/fortran/lib/intel64/libiomp5.a
> -r-xr-xr-x 1 root root 1058736 Aug 14 2009 /ifs1/opt/intel-11.1/mkl/lib/em64t/libiomp5.a

> They need to have the paths to your compiler

I know the compilers are in $PATH:

[tr@topsail-login1 ~]$ bsub -q int -Ip which icc
> /ifs1/opt/intel-11.1/cc/bin/intel64/icc
[tr@topsail-login1 ~]$ bsub -q int -Ip which ifort
> /ifs1/opt/intel-11.1/fortran_11.1.056/bin/intel64/ifort

So given libiomp5.a is there, and the fortran objects are compiling, why are they not linking?
I guess the next step is to rebuild:

> The first command in your [build] script should be to source the ifortvars.sh and/or iccvars.sh scripts:

>
> source /opt/intel/Compiler/11.1/072/ifortvars.sh intel64
> source /opt/intel/Compiler/11.1/072/iccvars.sh intel64
>

> Also, I would recommend compiling with -static-intel to the
> compiler/link options so that you statically put the Intel runtime
> libs into your executable.

OK, so currently I have

- buildJproc.sh (http://tinyurl.com/buildJproc-sh-20100427), written by me. It sets up the environment for bldit.jproc, calls it using LSF, and redirects its output to a logfile.
- bldit.jproc (http://tinyurl.com/bldit-jproc-20100427), written by the CMAQ folks. I edited one var to point to the local ifort.
- bldit.jproc has 'set LINK_FLAGS = "-Bstatic"'

It sounds like I need to

+ call buildJproc.sh from LSF
+ edit buildJproc.sh to source *vars.sh before calling bldit.jproc
+ in bldit.jproc, change to 'set LINK_FLAGS = "-static-intel"'

Am I missing anything? or including anything incorrectly?

your assistance is appreciated, Tom Roche