topic Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint' in Intel® oneAPI Math Kernel Library

MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Tue, 13 Aug 2024 19:00:59 GMT

In attempting to compile MUMPS, it fails on the examples with the below errors.

Here are the modules loaded on RHEL 9 cluster. Is there some incompatibility with Intel oneAPI 2024 mkl? Is it possible to skip compiling all of the examples? Same exact errors at this other thread or even this one from 2003?

scotch/7.0.4, parmetis/4.0.3, scalapack/2.2.0, oneapi/mkl/2024.0, oneapi/mpi/2021.11

Warning: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)).
ar vr ../lib/libzmumps.a zana_aux.o zana_aux_par.o zana_lr.o zfac_asm_master_ELT_m.o zfac_asm_master_m.o zfac_front_aux.o zfac_front_LU_type1.o zfac_front_LU_type2.o zfac_front_LDLT_type1.o zfac_front_LDLT_type2.o zfac_front_type2_aux.o zfac_sispointers_m.o zfac_lr.o zfac_mem_dynamic.o zfac_omp_m.o zfac_par_m.o zfac_sol_l0omp_m.o zlr_core.o zlr_stats.o zlr_type.o zmumps_comm_buffer.o zmumps_config_file.o zmumps_load.o zmumps_lr_data_m.o zmumps_ooc_buffer.o zmumps_ooc.o zmumps_sol_es.o zmumps_save_restore.o zmumps_save_restore_files.o zmumps_struc_def.o zomp_tps_m.o zsol_lr.o zsol_omp_m.o zstatic_ptr_m.o zmumps_mpi3_mod.o zini_driver.o zana_driver.o zfac_driver.o zsol_driver.o zsol_distrhs.o zsol_distsol.o zend_driver.o zana_aux_ELT.o zana_dist_m.o zana_LDLT_preprocess.o zana_reordertree.o zarrowheads.o zbcast_int.o zfac_asm_ELT.o zfac_asm.o zfac_b.o zfac_diag.o zfac_distrib_distentry.o zfac_dist_arrowheads_omp.o zfac_distrib_ELT.o zfac_lastrtnelind.o zfac_mem_alloc_cb.o zfac_mem_compress_cb.o zfac_mem_free_block_cb.o zfac_mem_stack_aux.o zfac_mem_stack.o zfac_process_band.o zfac_process_blfac_slave.o zfac_process_blocfacto_LDLT.o zfac_process_blocfacto.o zfac_process_bf.o zfac_process_end_facto_slave.o zfac_process_contrib_type1.o zfac_process_contrib_type2.o zfac_process_contrib_type3.o zfac_process_maprow.o zfac_process_master2.o zfac_process_message.o zfac_process_root2slave.o zfac_process_root2son.o zfac_process_rtnelind.o zfac_root_parallel.o zfac_scalings.o zfac_determinant.o zfac_scalings_simScaleAbs.o zfac_scalings_simScale_util.o zfac_sol_pool.o zfac_type3_symmetrize.o zini_defaults.o
ar: creating ../lib/libzmumps.a
a - zana_aux.o
a - zana_aux_par.o
a - zana_lr.o
a - zfac_asm_master_ELT_m.o
a - zfac_asm_master_m.o
a - zfac_front_aux.o
a - zfac_front_LU_type1.o
a - zfac_front_LU_type2.o
a - zfac_front_LDLT_type1.o
a - zfac_front_LDLT_type2.o
a - zfac_front_type2_aux.o
a - zfac_sispointers_m.o
a - zfac_lr.o
a - zfac_mem_dynamic.o
a - zfac_omp_m.o
a - zfac_par_m.o
a - zfac_sol_l0omp_m.o
a - zlr_core.o
a - zlr_stats.o
a - zlr_type.o
a - zmumps_comm_buffer.o
a - zmumps_config_file.o
a - zmumps_load.o
a - zmumps_lr_data_m.o
a - zmumps_ooc_buffer.o
a - zmumps_ooc.o
a - zmumps_sol_es.o
a - zmumps_save_restore.o
a - zmumps_save_restore_files.o
a - zmumps_struc_def.o
a - zomp_tps_m.o
a - zsol_lr.o
a - zsol_omp_m.o
a - zstatic_ptr_m.o
a - zmumps_mpi3_mod.o
a - zini_driver.o
a - zana_driver.o
a - zfac_driver.o
a - zsol_driver.o
a - zsol_distrhs.o
a - zsol_distsol.o
a - zend_driver.o
a - zana_aux_ELT.o
a - zana_dist_m.o
a - zana_LDLT_preprocess.o
a - zana_reordertree.o
a - zarrowheads.o
a - zbcast_int.o
a - zfac_asm_ELT.o
a - zfac_asm.o
a - zfac_b.o
a - zfac_diag.o
a - zfac_distrib_distentry.o
a - zfac_dist_arrowheads_omp.o
a - zfac_distrib_ELT.o
a - zfac_lastrtnelind.o
a - zfac_mem_alloc_cb.o
a - zfac_mem_compress_cb.o
a - zfac_mem_free_block_cb.o
a - zfac_mem_stack_aux.o
a - zfac_mem_stack.o
a - zfac_process_band.o
a - zfac_process_blfac_slave.o
a - zfac_process_blocfacto_LDLT.o
a - zfac_process_blocfacto.o
a - zfac_process_bf.o
a - zfac_process_end_facto_slave.o
a - zfac_process_contrib_type1.o
a - zfac_process_contrib_type2.o
a - zfac_process_contrib_type3.o
a - zfac_process_maprow.o
a - zfac_process_master2.o
a - zfac_process_message.o
a - zfac_process_root2slave.o
a - zfac_process_root2son.o
a - zfac_process_rtnelind.o
a - zfac_root_parallel.o
a - zfac_scalings.o
a - zfac_determinant.o
a - zfac_scalings_simScaleAbs.o
a - zfac_scalings_simScale_util.o
a - zfac_sol_pool.o
a - zfac_type3_symmetrize.o
a - zini_defaults.o
ranlib ../lib/libzmumps.a
make[2]: Leaving directory '/path/to/me/MUMPS_5.7.3/src'
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/src'
cd examples; make z
make[1]: Entering directory '/path/to/me/MUMPS_5.7.3/examples'
mpif90 -O -std=legacy -DBLR_MT -fopenmp -fallow-argument-mismatch -I. -I../include -I../src -I../libseq -c zsimpletest.F -o zsimpletest.o
mpif90 -o zsimpletest -O -fopenmp -fallow-argument-mismatch zsimpletest.o -L../lib -lzmumps -lmumps_common -L/insomnia001/shared/apps/metis/5.1.0/lib/ -lmetis -L../PORD/lib/ -lpord -L/insomnia001/shared/apps/scotch/7.0.4/lib -lesmumps -lscotch -lscotcherr -L/insomnia001/shared/apps/oneapi/mkl/2024.0 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L../libseq -lmpiseq -L/usr/lib64 -lblas -lpthread
/usr/bin/ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x4f): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x5f4): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x628): undefined reference to `zmumps_'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_critical_with_hint'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_critical'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_master'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_fini'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_reduce'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_flush'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_fork_call'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_push_num_threads'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_ordered'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task_alloc'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_barrier'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_cmplx8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_global_thread_num'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_critical'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_serialized_parallel'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_fixed4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task_with_deps'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_reduce_nowait'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float4_max'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_master'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_single'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_fini_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_fixed8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_cmplx4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_ok_to_fork'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_single'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_ordered'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_reduce_nowait'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_serialized_parallel'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float8_max'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_reduce'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:41: zsimpletest] Error 1
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

Re:MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Vipin_S_Intel — Wed, 21 Aug 2024 13:18:13 GMT

Hi Rob, we would like to inform you that we are routing your query to the dedicated team for further assistance.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Wed, 28 Aug 2024 02:37:56 GMT

Hello @RobbieTheK, could you provide detailed instructions on how you compiled the examples? So, I can reproduce them on my end? Thank you.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Wed, 28 Aug 2024 14:13:43 GMT

Modules loaded:

1) metis/5.1.0 oneapi/intel_ippcp_intel64/2021.9 15) oneapi/dev-utilities/2024.0.0
2) oneapi/tbb/2021.11 9) oneapi/intel_ipp_intel64/2021.10 16) oneapi/debugger/2024.0.0
3) oneapi/compiler-rt/2024.0.2 10) oneapi/hpctoolkit/hpctoolkit-2024.0.0 17) oneapi/dal/2024.0.0
4) oneapi/mkl/2024.0 11) oneapi/hpctoolkit/tbb32/2021.11 18) oneapi/compiler/2024.0.2
5) oneapi/mpi/2021.11 12) oneapi/dpl/2022.3 19) oneapi/ccl/2021.11.2
6) oneapi/oclfpga/2024.0.0 13) oneapi/dpct/2024.0.0 20) oneapi/advisor/2024.0
7) oneapi/vtune/2024.0 14) oneapi/dnnl/3.3.0 21) scalapack/2.2.0

Makefile.inc. contents:

#
# This file is part of MUMPS 5.7.3, released
# on Mon Jul 15 11:44:21 UTC 2024
#
#Begin orderings

# NOTE that PORD is distributed within MUMPS by default. It is recommended to
# install other orderings. For that, you need to obtain the corresponding package
# and modify the variables below accordingly.
# For example, to have Metis available within MUMPS:
# 1/ download Metis and compile it
# 2/ uncomment (suppress # in first column) lines
# starting with LMETISDIR, LMETIS
# 3/ add -Dmetis in line ORDERINGSF
# ORDERINGSF = -Dpord -Dmetis
# 4/ Compile and install MUMPS
# make clean; make (to clean up previous installation)
#
# Metis/ParMetis and SCOTCH/PT-SCOTCH (ver 6.0 and later) orderings are recommended.
#
#topdir = .
topdir = /path/to/me/MUMPS_5.7.3
#SCOTCHDIR = ${HOME}/scotch_6.0
#ISCOTCH = -I$(SCOTCHDIR)/include
SCOTCHDIR = /path/shared/apps/scotch/7.0.5
ISCOTCH = -I$(SCOTCHDIR)/include
#
# You have to choose one among the following two lines depending on
# the type of analysis you want to perform. If you want to perform only
# sequential analysis choose the first (remember to add -Dscotch in the ORDERINGSF
# variable below); for both parallel and sequential analysis choose the second
# line (remember to add -Dptscotch in the ORDERINGSF variable below)

#LSCOTCH = -L$(SCOTCHDIR)/lib -lesmumps -lscotch -lscotcherr
LSCOTCH = -L$(SCOTCHDIR)/lib64 -lptesmumps -lptscotch -lptscotcherr

LPORDDIR = $(topdir)/PORD/lib/
IPORD = -I$(topdir)/PORD/include/
LPORD = -L$(LPORDDIR) -lpord$(PLAT)

#LMETISDIR = /opt/metis-5.1.0/build/Linux-x86_64/libmetis
#IMETIS = /opt/metis-5.1.0/include
LMETISDIR = /path/shared/apps/metis/5.1.0/lib
IMETIS = /opt/metis-5.1.0/include

# You have to choose one among the following two lines depending on
# the type of analysis you want to perform. If you want to perform only
# sequential analysis choose the first (remember to add -Dmetis in the ORDERINGSF
# variable below); for both parallel and sequential analysis choose the second
# line (remember to add -Dparmetis in the ORDERINGSF variable below)

#LMETIS = -L$(LMETISDIR) -lmetis
LMETIS = -L$(LMETISDIR) -lmetis

# The following variables will be used in the compilation process.
# Please note that -Dptscotch and -Dparmetis imply -Dscotch and -Dmetis respectively.
# If you want to use Metis 4.X or an older version, you should use -Dmetis4 instead of -Dmetis
# or in addition with -Dparmetis (if you are using parmetis 3.X or older).
#ORDERINGSF = -Dscotch -Dmetis -Dpord -Dptscotch -Dparmetis
ORDERINGSF = -Dpord -Dparmetis
ORDERINGSC = $(ORDERINGSF)

LORDERINGS = $(LMETIS) $(LPORD) $(LSCOTCH)
IORDERINGSF = $(ISCOTCH)
IORDERINGSC = $(IMETIS) $(IPORD) $(ISCOTCH)

#End orderings
########################################################################
################################################################################

PLAT =
LIBEXT = .a
LIBEXT_SHARED = .so
SONAME = -soname
SHARED_OPT = -shared
FPIC_OPT = -fPIC
# Adapt/uncomment RPATH_OPT to avoid modifying
# LD_LIBRARY_PATH in case of shared libraries
# RPATH_OPT = -Wl,-rpath,/path/to/MUMPS_x.y.z/lib/
OUTC = -o
OUTF = -o
RM = /bin/rm -f
#CC = mpiicc
CC = mpiicx
FC = mpiifort
FL = mpiifort
AR = ar vr
#RANLIB = ranlib
RANLIB = echo
# Make this variable point to the path where the Intel MKL library is
# installed. It is set to the default install directory for Intel MKL.
#MKLROOT=/opt/intel/mkl/lib/intel64
MKLROOT=/insomnia001/shared/apps/oneapi/mkl/2024.0/lib
LAPACK = -L$(MKLROOT) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
SCALAP = -L$(MKLROOT) -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64

LIBPAR = $(SCALAP) $(LAPACK)

INCSEQ = -I$(topdir)/libseq
LIBSEQ = $(LAPACK) -L$(topdir)/libseq -lmpiseq$(PLAT)

LIBBLAS = -L$(MKLROOT) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
LIBOTHERS = -lpthread

#Preprocessor defs for calling Fortran from C (-DAdd_ or -DAdd__ or -DUPPER)
CDEFS = -DAdd_

#Begin Optimized options
OPTF = -O -nofor-main -qopenmp -DGEMMT_AVAILABLE -diag-disable=10448
OPTL = -O -nofor-main -qopenmp -diag-disable=10448
OPTC = -O -qopenmp -diag-disable=10448
#End Optimized options

INCS = $(INCPAR)
LIBS = $(LIBPAR)
LIBSEQNEEDED =

Running make z fails here:

cd examples; make z
make[1]: Entering directory '/path/to/me/MUMPS_5.7.3/examples'
mpiifort -o zsimpletest -O -nofor-main -qopenmp -diag-disable=10448 zsimpletest.o -L/path/to/me/MUMPS_5.7.3/lib -lzmumps -lmumps_common -L/insomnia001/shared/apps/metis/5.1.0/lib -lmetis -L/insomnia001/home/rk3199/MUMPS_5.7.3/PORD/lib/ -lpord -L/path/shared/apps/scotch/7.0.5/lib64 -lptesmumps -lptscotch -lptscotcherr -L/path/shared/apps/oneapi/mkl/2024.0/lib -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/path/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/path/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread
ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x47): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x4aa): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x8cb): undefined reference to `zmumps_'
make[1]: *** [Makefile:44: zsimpletest] Error 1
make[1]: Leaving directory '/insomnia001/home/rk3199/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Wed, 28 Aug 2024 20:20:03 GMT

Hello @RobbieTheK,

It's not an endorsement, but I just tried this build, by i) cloning this GitHub, ii) sourcing the latest oneAPI and iii) "cmake -B build; cmake --build build". This successfully built everything. You can try this too.

I will try your way. As you can see, the official Makefile build points to a linking error: "undefined reference to `zmumps_'". Maybe, you can check LD_LIBRARY_PATH environment variable to include the MUMPS library path.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Wed, 28 Aug 2024 20:24:47 GMT

Yes I tried that but try the examples lower in the page those fail with missing some scalapack library.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Wed, 28 Aug 2024 21:06:12 GMT

"missing ... scalapack"? I only see "undefined reference to `zmumps_'" above?

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Wed, 28 Aug 2024 21:12:20 GMT

The scalapack error happens with the git package when trying to cmake on the examples.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Wed, 28 Aug 2024 21:53:56 GMT

cmake --install build

cmake -S example -B example/build -DMUMPS_ROOT=build/local

cmake --build example/build

[ 25%] Building C object CMakeFiles/d_example_c.dir/d_example.c.o
[ 50%] Linking C executable d_example_c
[ 50%] Built target d_example_c
[ 75%] Building CXX object CMakeFiles/d_example_cpp.dir/d_example.cpp.o
[100%] Linking CXX executable d_example_cpp
[100%] Built target d_example_cpp

>mumps$

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Thu, 29 Aug 2024 00:58:16 GMT

a build with MUMPS_5.7.3:

1. cp Make.inc/Makefile.INTEL.PAR ./Makefile.inc

2. replaced old compilers with LLVM compilers in Makefile.inc

#CC = mpiicc
CC = mpiicx
#FC = mpiifort
#FL = mpiifort
FC = mpiifx
FL = mpiifx

3. sourced oneAPI tools

4. make z

a - zsol_fwd.o a - zsol_matvec.o a - zsol_root_parallel.o a - ztools.o a - ztype3_root.o echo ../lib/libzmumps.a ../lib/libzmumps.a make[2]: Leaving directory '/scratch/.../MUMPS_5.7.3/src' make[1]: Leaving directory '/scratch/.../MUMPS_5.7.3/src' cd examples; make z make[1]: Entering directory '/scratch/.../MUMPS_5.7.3/examples' mpiifx -O -nofor_main -qopenmp -DGEMMT_AVAILABLE -I. -I../include -I../src -c zsimpletest.F -o zsimpletest.o ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o -L../lib -lzmumps -lmumps_common -L../PORD/lib/ -lpord -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead mpiifx -O -nofor_main -qopenmp -DGEMMT_AVAILABLE -I. -I../include -I../src -c zsimpletest_save_restore.F -o zsimpletest_save_restore.o ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead mpiifx -o zsimpletest_save_restore -O -nofor_main -qopenmp zsimpletest_save_restore.o -L../lib -lzmumps -lmumps_common -L../PORD/lib/ -lpord -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead make[1]: Leaving directory '/scratch/.../MUMPS_5.7.3/examples' /scratch/.../MUMPS_5.7.3$ ls examples c_example.c input_simpletest_real zsimpletest.F c_example_save_restore.c Makefile zsimpletest.o csimpletest.F multiple_arithmetics_example.F zsimpletest_save_restore csimpletest_save_restore.F README zsimpletest_save_restore.F dsimpletest.F ssimpletest.F zsimpletest_save_restore.o dsimpletest_save_restore.F ssimpletest_save_restore.F input_simpletest_cmplx zsimpletest

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Thu, 29 Aug 2024 01:20:15 GMT

Can you share how to install the oneapi LLVM, as a module like how the other oneapi modules are installed?

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Thu, 29 Aug 2024 06:00:18 GMT

What I call LLVM compilers are the new compilers that should be already available when you load oneAPI compiler modules. The "old" or "classic" compilers have been deprecated. So for example, the classic icc compiler is not even available anymore:

ls /opt/intel/oneapi/compiler/2024.2/bin aocl-ioc64 fortcom icx ifort.cfg opencl-aot run-clang-tidy xiar.cfg codecov fpp icx-cc ifx profdcg sycl-ls xild compiler git-clang-format icx.cfg ifx.cfg profmerge sycl-trace xild.cfg dpcpp icpx icx-cl ioc64 profmergesampling tselect dpcpp-cl icpx.cfg ifort map_opts proforder xiar mlubin@ortce-skl:~$

instead, you have to use the icx (LLVM-based) new compiler. In the case of Fortran, as you can see, ifort is still available, but it is better to use the ifx compiler. Since I was building with MPI, I used MPI wrappers that call new compilers. Therefore, you can see above mpiicx instead of mpiicc, and mpiifx instead of mpiifort. Just check after you load modules, "which icx", etc., to make sure that you got the new compilers. Or consult your cluster admin. Hope this helps.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Thu, 29 Aug 2024 16:37:02 GMT

Can you clarify your comment here:

Maybe, you can check
LD_LIBRARY_PATH
environment variable to include the MUMPS library path.

If I'm compiling MUMPS how would I add the library path?

Here's the scalapack error I get when using the Git MUMPS you provided:

cmake -S example -B example/build -DMUMPS_ROOT=build/local
CMake Error at /insomnia001/home/rk3199/mumps/build/cmake/MUMPSConfig.cmake:31 (include):
include could not find requested file:

/insomnia001/home/rk3199/mumps/build/cmake/MUMPS-targets.cmake
Call Stack (most recent call first):
CMakeLists.txt:15 (find_package)

CMake Error at /cluster/shared/apps/cmake/3.30.2/share/cmake-3.30/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
By not providing "FindSCALAPACK.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"SCALAPACK", but CMake did not find one.

Could not find a package configuration file provided by "SCALAPACK" with
any of the following names:

SCALAPACKConfig.cmake
scalapack-config.cmake

Add the installation prefix of "SCALAPACK" to CMAKE_PREFIX_PATH or set
"SCALAPACK_DIR" to a directory containing one of the above files. If
"SCALAPACK" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/cluster/path/to/me/mumps/build/cmake/MUMPSConfig.cmake:51 (find_dependency)
CMakeLists.txt:15 (find_package)

-- Configuring incomplete, errors occurred!

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Thu, 29 Aug 2024 16:39:27 GMT

You didn't mention changing the path to MKLROOT

MKLROOT=/cluster/shared/apps/oneapi/mkl/2024.0/lib

Also your comment:

"Since I was building with MPI, I used MPI wrappers that call new compilers."

Did you create this wrapper? the mpiifx is in:

/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpiifx

Still fails:

cd examples; make z
make[1]: Entering directory '/cluster/path/to/me/MUMPS_5.7.3/examples'
mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o -L/cluster/home/rk3199/MUMPS_5.7.3/lib -lzmumps -lmumps_common -L/insomnia001/home/rk3199/MUMPS_5.7.3/PORD/lib/ -lpord -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread
/cluster/shared/apps/oneapi/mpi/2021.11/bin/mpiifx: line 702: ifx: command not found
make[1]: *** [Makefile:44: zsimpletest] Error 127
make[1]: Leaving directory '/cluster/path/to/me/MUMPS_5.7.3/examples'

I see a newer mpiifx in the HPCtoolkit

grep -R mpiifx /cluster/shared/apps/oneapi/hpctoolkit/
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/.toolkit_linking_tool/intel.oneapi.lin.mpi.devel-2021.11.0-49511.filelist:bin/mpiifx
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/bin/mpiifort:TARGET_WRAPPER="mpiifx"
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/lib/release: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/lib/release_mt: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib32/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib/ia32/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib/intel64: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/compiler/modulefiles/oclfpga: No such file or directory
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/.toolkit_linking_tool/intel.oneapi.lin.mpi.devel-2021.11.0-49511.filelist:binmpiifx
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/bin/mpiifort:TARGET_WRAPPER="mpiifx"
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/lib/release: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/lib/release_mt: warning: recursive directory loop
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/.toolkit_linking_tool/intel.oneapi.lin.mpi.devel-2021.11.0-49511.filelist:bin/mpiifx
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpiifort:TARGET_WRAPPER="mpiifx"
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/lib/release: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/lib/release_mt: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/2024.0/lib32/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/2024.0/modulefiles/oclfpga: No such file or directory
/cluster/shared/apps/oneapi/hpctoolkit/2024.0/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/2024.0/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/2024.0/bin/mpiifort:TARGET_WRAPPER="mpiifx"

Also:

which icx
/cluster/shared/apps/oneapi/hpctoolkit/compiler/2024.0/bin/icx

What am I doing wrong here?

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Thu, 05 Sep 2024 21:08:05 GMT

Hello @RobbieTheK,

Answering your questions above:

q.> In my environment, when I source oneAPI, /opt/intel/oneapi/setvars.sh, the MKLROOT is automatically set: echo $MKLROOT /opt/intel/oneapi/mkl/2024.2
q.>"Since I was building with MPI, I used MPI wrappers that call new compilers." Did you create this wrapper? -- No, the mpiifx is already supplied with IMPI distribution, for example, in my case:
which mpiifx /opt/intel/oneapi/mpi/2021.13/bin/mpiifx
q.: "/cluster/shared/apps/oneapi/mpi/2021.11/bin/mpiifx: line 702: ifx: command not found: suggest that ifx was not located on your system, In my case: which ifx /opt/intel/oneapi/compiler/2024.2/bin/ifx
please check if ifx is available on your system.
I would not try to use 2 different distributions at the same time (for example, what you call a newer distribution)

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Fri, 06 Sep 2024 16:40:23 GMT

ml
Currently Loaded Modulefiles:
1) oneapi/hpctoolkit/hpctoolkit-2024.0.0 4) oneapi/compiler-rt/2024.0.2 7) scalapack/2.2.0 
2) oneapi/hpctoolkit/mpi/2021.11 5) oneapi/mkl/2024.0  scotch/7.0.5 
3) oneapi/tbb/2021.11 6) metis/5.1.0

Fri Sep 06 me@node $ which mpiifx
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpiifx
Fri Sep 06me@node $ which ifx
/cluster/shared/apps/oneapi/hpctoolkit/compiler/2024.0/bin/ifx
Fri Sep 06me@node $ echo $MKLROOT
/cluster/shared/apps/oneapi/mkl/2024.0

Same errors:

make[1]: Entering directory '/cluster/home/me/MUMPS_5.7.3/examples'
mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o -L/cluster/home/me/MUMPS_5.7.3/lib -lzmumps -lmumps_common -L/cluster/home/rk3199/MUMPS_5.7.3/PORD/lib/ -lpord -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x47): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x4aa): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x8cb): undefined reference to `zmumps_'
make[1]: *** [Makefile:44: zsimpletest] Error 1
make[1]: Leaving directory '/cluster/home/me/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Fri, 06 Sep 2024 18:04:05 GMT

@RobbieTheK,

In the comment before the last one, you had an "ifx not found" issue. I don't see it anymore. The undefined reference to zmumps_ when you compile examples means that there is a linking problem—the system cannot find the mumps library (that should be built at this point) while attempting to compile examples. This is not oneMKL issue.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Fri, 06 Sep 2024 19:49:27 GMT

OK so I went back and used the Git MUMPS and that compiled correctly. However we need MUMPS for compiling another application that uses nonlinear inversion algorithm (NLI) for Magnetic resonance elastography (MRE). That 'make fails with:

 make 
mpif90 -o MRE-Zone.discov -w -O3 -fno-range-check -ffixed-line-length-none -ffree-line-length-none -ffixed-line-length-0 -std=legacy -ldl MRE-Zone.o CGupdatescale.o FEgaussnewton.o fdchecks.o sensitivityanalysis.o FEmaterial.o FEmesh.o FEsolution.o exteriorconstraint.o gaussnewton.o hex27.o joachimowiczreg.o mcgarrybounds.o optimizationparams.o reporterror.o spatialfiltering.o tikhonovreg.o totalvariation.o zonefunctions.o FEforward.o FEgradient.o FEmatrix.o FEproblem.o fileio.o globalvariables.o hex8.o marquardtreg.o mpiparameters.o regularizationparams.o softpriorreg.o tet4.o timing.o vanhoutenreg.o zonevariables.o -L/cluster/shared/apps/metis/5.1.0/lib -lmetis -lmkl_sequential -limf -lmkl_core -ldl -lmkl_intel_lp64 -lmkl_gnu_thread -lpthread -lm -ldl 
/usr/bin/ld: MRE-Zone.o: in function `worldslave.37':
MRE-Zone.f90:(.text+0x40905): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40bd9): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40c0e): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40c57): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40ccb): undefined reference to `zmumps_'
/usr/bin/ld: FEgaussnewton.o:FEgaussnewton.f90:(.text+0xef3e): more undefined references to `zmumps_' follow
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_ordered_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_dynamic_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_guided_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_atomic_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_single_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_name_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_static_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_barrier'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_atomic_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_dynamic_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_num_procs'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_parallel'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_max_threads'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_dynamic_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_end_nowait'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_ordered_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_max_active_levels'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_in_parallel'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_name_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_parallel_sections'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_guided_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_end_nowait'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_static_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_thread_num'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_dynamic_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_num_threads'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_set_max_active_levels'
collect2: error: ld returned 1 exit status
make: *** [Makefile:161: link]

Any idea why?

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

Mark_L_Intel — Wed, 11 Sep 2024 19:17:32 GMT

Hello @RobbieTheK,

"OK so I went back and used the Git MUMPS and that compiled correctly. " -- glad to see it.

The error messages like this: "libmkl_gnu_thread.so: undefined reference to omp_get_thread_num'" typically indicate that there's an issue with linking the OpenMP library when using oneMKL with GNU compilers. Indeed, I see you used this flag

-lmkl_sequential

This means you ask for a non-threaded version of oneMKL while the application has OpenMP statements. Please use oneMKL Link Line Advisor to determine the correct build options.

For example, I picked up Intel tools in Link Advisor and got the following link line and compiler options:

If I uncheck "Link with Inlte oneMKL libraries explicitly:", you get simplified version:

As a side comment, I'm not sure why you switched to GNU compilers -- since you used mpif90. To use Intel compilers, please use mpiifx.

Re: MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

RobbieTheK — Wed, 11 Sep 2024 20:16:19 GMT

OK I used -lmetis -limf -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl but alas still a zumps error, and I also changed to using mpiifx :

mpiifx -o MRE-Zone.discov -w -O3 -fno-range-check -ffixed-line-length-none -ffree-line-length-none -ffixed-line-length-0 -std=legacy -ldl MRE-Zone.o CGupdatescale.o FEgaussnewton.o fdchecks.o sensitivityanalysis.o FEmaterial.o FEmesh.o FEsolution.o exteriorconstraint.o gaussnewton.o hex27.o joachimowiczreg.o mcgarrybounds.o optimizationparams.o reporterror.o spatialfiltering.o tikhonovreg.o totalvariation.o zonefunctions.o FEforward.o FEgradient.o FEmatrix.o FEproblem.o fileio.o globalvariables.o hex8.o marquardtreg.o mpiparameters.o regularizationparams.o softpriorreg.o tet4.o timing.o vanhoutenreg.o zonevariables.o -L/cluster/shared/apps/metis/5.1.0/lib -lmetis -limf -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl 
ld: MRE-Zone.o: in function `mre_zone_IP_mumpsslavebuild_':
MRE-Zone.f90:(.text+0x6adc8): undefined reference to `zmumps_'
ld: MRE-Zone.o: in function `mre_zone_IP_mumpsslavesolve_':
MRE-Zone.f90:(.text+0x6bc2a): undefined reference to `zmumps_'
ld: MRE-Zone.f90:(.text+0x6bcc8): undefined reference to `zmumps_'
ld: MRE-Zone.f90:(.text+0x6bd8a): undefined reference to `zmumps_'
ld: MRE-Zone.f90:(.text+0x6be18): undefined reference to `zmumps_'
ld: FEgaussnewton.o:FEgaussnewton.f90:(.text+0x606): more undefined references to `zmumps_' follow
make: *** [Makefile:165: link] Error 1