MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

[RobbieTheK]

In attempting to compile MUMPS, it fails on the examples with the below errors. 

Here are the modules loaded on RHEL 9 cluster. Is there some incompatibility with Intel oneAPI 2024 mkl? Is it possible to skip compiling all of the examples? Same exact errors at this other thread or even this one from 2003?

scotch/7.0.4, parmetis/4.0.3, scalapack/2.2.0, oneapi/mkl/2024.0, oneapi/mpi/2021.11

Warning: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)).
ar vr ../lib/libzmumps.a zana_aux.o zana_aux_par.o zana_lr.o zfac_asm_master_ELT_m.o zfac_asm_master_m.o zfac_front_aux.o zfac_front_LU_type1.o zfac_front_LU_type2.o zfac_front_LDLT_type1.o zfac_front_LDLT_type2.o zfac_front_type2_aux.o zfac_sispointers_m.o zfac_lr.o zfac_mem_dynamic.o zfac_omp_m.o zfac_par_m.o zfac_sol_l0omp_m.o zlr_core.o zlr_stats.o zlr_type.o zmumps_comm_buffer.o zmumps_config_file.o zmumps_load.o zmumps_lr_data_m.o zmumps_ooc_buffer.o zmumps_ooc.o zmumps_sol_es.o zmumps_save_restore.o zmumps_save_restore_files.o zmumps_struc_def.o zomp_tps_m.o zsol_lr.o zsol_omp_m.o zstatic_ptr_m.o zmumps_mpi3_mod.o zini_driver.o zana_driver.o zfac_driver.o zsol_driver.o zsol_distrhs.o zsol_distsol.o zend_driver.o zana_aux_ELT.o zana_dist_m.o zana_LDLT_preprocess.o zana_reordertree.o zarrowheads.o zbcast_int.o zfac_asm_ELT.o zfac_asm.o zfac_b.o zfac_diag.o zfac_distrib_distentry.o zfac_dist_arrowheads_omp.o zfac_distrib_ELT.o zfac_lastrtnelind.o zfac_mem_alloc_cb.o zfac_mem_compress_cb.o zfac_mem_free_block_cb.o zfac_mem_stack_aux.o zfac_mem_stack.o zfac_process_band.o zfac_process_blfac_slave.o zfac_process_blocfacto_LDLT.o zfac_process_blocfacto.o zfac_process_bf.o zfac_process_end_facto_slave.o zfac_process_contrib_type1.o zfac_process_contrib_type2.o zfac_process_contrib_type3.o zfac_process_maprow.o zfac_process_master2.o zfac_process_message.o zfac_process_root2slave.o zfac_process_root2son.o zfac_process_rtnelind.o zfac_root_parallel.o zfac_scalings.o zfac_determinant.o zfac_scalings_simScaleAbs.o zfac_scalings_simScale_util.o zfac_sol_pool.o zfac_type3_symmetrize.o zini_defaults.o
ar: creating ../lib/libzmumps.a
a - zana_aux.o
a - zana_aux_par.o
a - zana_lr.o
a - zfac_asm_master_ELT_m.o
a - zfac_asm_master_m.o
a - zfac_front_aux.o
a - zfac_front_LU_type1.o
a - zfac_front_LU_type2.o
a - zfac_front_LDLT_type1.o
a - zfac_front_LDLT_type2.o
a - zfac_front_type2_aux.o
a - zfac_sispointers_m.o
a - zfac_lr.o
a - zfac_mem_dynamic.o
a - zfac_omp_m.o
a - zfac_par_m.o
a - zfac_sol_l0omp_m.o
a - zlr_core.o
a - zlr_stats.o
a - zlr_type.o
a - zmumps_comm_buffer.o
a - zmumps_config_file.o
a - zmumps_load.o
a - zmumps_lr_data_m.o
a - zmumps_ooc_buffer.o
a - zmumps_ooc.o
a - zmumps_sol_es.o
a - zmumps_save_restore.o
a - zmumps_save_restore_files.o
a - zmumps_struc_def.o
a - zomp_tps_m.o
a - zsol_lr.o
a - zsol_omp_m.o
a - zstatic_ptr_m.o
a - zmumps_mpi3_mod.o
a - zini_driver.o
a - zana_driver.o
a - zfac_driver.o
a - zsol_driver.o
a - zsol_distrhs.o
a - zsol_distsol.o
a - zend_driver.o
a - zana_aux_ELT.o
a - zana_dist_m.o
a - zana_LDLT_preprocess.o
a - zana_reordertree.o
a - zarrowheads.o
a - zbcast_int.o
a - zfac_asm_ELT.o
a - zfac_asm.o
a - zfac_b.o
a - zfac_diag.o
a - zfac_distrib_distentry.o
a - zfac_dist_arrowheads_omp.o
a - zfac_distrib_ELT.o
a - zfac_lastrtnelind.o
a - zfac_mem_alloc_cb.o
a - zfac_mem_compress_cb.o
a - zfac_mem_free_block_cb.o
a - zfac_mem_stack_aux.o
a - zfac_mem_stack.o
a - zfac_process_band.o
a - zfac_process_blfac_slave.o
a - zfac_process_blocfacto_LDLT.o
a - zfac_process_blocfacto.o
a - zfac_process_bf.o
a - zfac_process_end_facto_slave.o
a - zfac_process_contrib_type1.o
a - zfac_process_contrib_type2.o
a - zfac_process_contrib_type3.o
a - zfac_process_maprow.o
a - zfac_process_master2.o
a - zfac_process_message.o
a - zfac_process_root2slave.o
a - zfac_process_root2son.o
a - zfac_process_rtnelind.o
a - zfac_root_parallel.o
a - zfac_scalings.o
a - zfac_determinant.o
a - zfac_scalings_simScaleAbs.o
a - zfac_scalings_simScale_util.o
a - zfac_sol_pool.o
a - zfac_type3_symmetrize.o
a - zini_defaults.o
ranlib ../lib/libzmumps.a
make[2]: Leaving directory '/path/to/me/MUMPS_5.7.3/src'
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/src'
cd examples; make z
make[1]: Entering directory '/path/to/me/MUMPS_5.7.3/examples'
mpif90 -O -std=legacy -DBLR_MT -fopenmp -fallow-argument-mismatch -I. -I../include -I../src -I../libseq -c zsimpletest.F -o zsimpletest.o
mpif90 -o zsimpletest -O -fopenmp -fallow-argument-mismatch zsimpletest.o -L../lib -lzmumps -lmumps_common -L/insomnia001/shared/apps/metis/5.1.0/lib/ -lmetis -L../PORD/lib/ -lpord -L/insomnia001/shared/apps/scotch/7.0.4/lib -lesmumps -lscotch -lscotcherr -L/insomnia001/shared/apps/oneapi/mkl/2024.0 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L../libseq -lmpiseq -L/usr/lib64 -lblas -lpthread
/usr/bin/ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x4f): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x5f4): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x628): undefined reference to `zmumps_'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_critical_with_hint'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_critical'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_master'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_fini'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_reduce'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_flush'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_fork_call'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_push_num_threads'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_ordered'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task_alloc'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_barrier'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_cmplx8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_global_thread_num'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_critical'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_serialized_parallel'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_fixed4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task_with_deps'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_reduce_nowait'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float4_max'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_master'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_single'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_fini_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_fixed8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_cmplx4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_ok_to_fork'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_single'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_ordered'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_reduce_nowait'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_serialized_parallel'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float8_max'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_reduce'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:41: zsimpletest] Error 1
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

[Vipin_S_Intel]

Hi Rob, we would like to inform you that we are routing your query to the dedicated team for further assistance.

[Mark_L_Intel]

Hello @RobbieTheK, could you provide detailed instructions on how you compiled the examples? So, I can reproduce them on my end? Thank you.   

[RobbieTheK]

Modules loaded:

1) metis/5.1.0 oneapi/intel_ippcp_intel64/2021.9 15) oneapi/dev-utilities/2024.0.0
2) oneapi/tbb/2021.11 9) oneapi/intel_ipp_intel64/2021.10 16) oneapi/debugger/2024.0.0
3) oneapi/compiler-rt/2024.0.2 10) oneapi/hpctoolkit/hpctoolkit-2024.0.0 17) oneapi/dal/2024.0.0
4) oneapi/mkl/2024.0 11) oneapi/hpctoolkit/tbb32/2021.11 18) oneapi/compiler/2024.0.2
5) oneapi/mpi/2021.11 12) oneapi/dpl/2022.3 19) oneapi/ccl/2021.11.2
6) oneapi/oclfpga/2024.0.0 13) oneapi/dpct/2024.0.0 20) oneapi/advisor/2024.0
7) oneapi/vtune/2024.0 14) oneapi/dnnl/3.3.0 21) scalapack/2.2.0

Makefile.inc. contents:

#
# This file is part of MUMPS 5.7.3, released
# on Mon Jul 15 11:44:21 UTC 2024
#
#Begin orderings

# NOTE that PORD is distributed within MUMPS by default. It is recommended to
# install other orderings. For that, you need to obtain the corresponding package
# and modify the variables below accordingly.
# For example, to have Metis available within MUMPS:
# 1/ download Metis and compile it
# 2/ uncomment (suppress # in first column) lines
# starting with LMETISDIR, LMETIS
# 3/ add -Dmetis in line ORDERINGSF
# ORDERINGSF = -Dpord -Dmetis
# 4/ Compile and install MUMPS
# make clean; make (to clean up previous installation)
#
# Metis/ParMetis and SCOTCH/PT-SCOTCH (ver 6.0 and later) orderings are recommended.
#
#topdir = .
topdir = /path/to/me/MUMPS_5.7.3
#SCOTCHDIR = ${HOME}/scotch_6.0
#ISCOTCH = -I$(SCOTCHDIR)/include
SCOTCHDIR = /path/shared/apps/scotch/7.0.5
ISCOTCH = -I$(SCOTCHDIR)/include
#
# You have to choose one among the following two lines depending on
# the type of analysis you want to perform. If you want to perform only
# sequential analysis choose the first (remember to add -Dscotch in the ORDERINGSF
# variable below); for both parallel and sequential analysis choose the second
# line (remember to add -Dptscotch in the ORDERINGSF variable below)

#LSCOTCH = -L$(SCOTCHDIR)/lib -lesmumps -lscotch -lscotcherr
LSCOTCH = -L$(SCOTCHDIR)/lib64 -lptesmumps -lptscotch -lptscotcherr

LPORDDIR = $(topdir)/PORD/lib/
IPORD = -I$(topdir)/PORD/include/
LPORD = -L$(LPORDDIR) -lpord$(PLAT)

#LMETISDIR = /opt/metis-5.1.0/build/Linux-x86_64/libmetis
#IMETIS = /opt/metis-5.1.0/include
LMETISDIR = /path/shared/apps/metis/5.1.0/lib
IMETIS = /opt/metis-5.1.0/include

# You have to choose one among the following two lines depending on
# the type of analysis you want to perform. If you want to perform only
# sequential analysis choose the first (remember to add -Dmetis in the ORDERINGSF
# variable below); for both parallel and sequential analysis choose the second
# line (remember to add -Dparmetis in the ORDERINGSF variable below)

#LMETIS = -L$(LMETISDIR) -lmetis
LMETIS = -L$(LMETISDIR) -lmetis

# The following variables will be used in the compilation process.
# Please note that -Dptscotch and -Dparmetis imply -Dscotch and -Dmetis respectively.
# If you want to use Metis 4.X or an older version, you should use -Dmetis4 instead of -Dmetis
# or in addition with -Dparmetis (if you are using parmetis 3.X or older).
#ORDERINGSF = -Dscotch -Dmetis -Dpord -Dptscotch -Dparmetis
ORDERINGSF = -Dpord -Dparmetis
ORDERINGSC = $(ORDERINGSF)

LORDERINGS = $(LMETIS) $(LPORD) $(LSCOTCH)
IORDERINGSF = $(ISCOTCH)
IORDERINGSC = $(IMETIS) $(IPORD) $(ISCOTCH)

#End orderings
########################################################################
################################################################################

PLAT =
LIBEXT = .a
LIBEXT_SHARED = .so
SONAME = -soname
SHARED_OPT = -shared
FPIC_OPT = -fPIC
# Adapt/uncomment RPATH_OPT to avoid modifying
# LD_LIBRARY_PATH in case of shared libraries
# RPATH_OPT = -Wl,-rpath,/path/to/MUMPS_x.y.z/lib/
OUTC = -o
OUTF = -o
RM = /bin/rm -f
#CC = mpiicc
CC = mpiicx
FC = mpiifort
FL = mpiifort
AR = ar vr
#RANLIB = ranlib
RANLIB = echo
# Make this variable point to the path where the Intel MKL library is
# installed. It is set to the default install directory for Intel MKL.
#MKLROOT=/opt/intel/mkl/lib/intel64
MKLROOT=/insomnia001/shared/apps/oneapi/mkl/2024.0/lib
LAPACK = -L$(MKLROOT) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
SCALAP = -L$(MKLROOT) -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64

LIBPAR = $(SCALAP) $(LAPACK)

INCSEQ = -I$(topdir)/libseq
LIBSEQ = $(LAPACK) -L$(topdir)/libseq -lmpiseq$(PLAT)

LIBBLAS = -L$(MKLROOT) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
LIBOTHERS = -lpthread

#Preprocessor defs for calling Fortran from C (-DAdd_ or -DAdd__ or -DUPPER)
CDEFS = -DAdd_

#Begin Optimized options
OPTF = -O -nofor-main -qopenmp -DGEMMT_AVAILABLE -diag-disable=10448
OPTL = -O -nofor-main -qopenmp -diag-disable=10448
OPTC = -O -qopenmp -diag-disable=10448
#End Optimized options

INCS = $(INCPAR)
LIBS = $(LIBPAR)
LIBSEQNEEDED =

Running make z fails here:

cd examples; make z
make[1]: Entering directory '/path/to/me/MUMPS_5.7.3/examples'
mpiifort -o zsimpletest -O -nofor-main -qopenmp -diag-disable=10448 zsimpletest.o -L/path/to/me/MUMPS_5.7.3/lib -lzmumps -lmumps_common -L/insomnia001/shared/apps/metis/5.1.0/lib -lmetis -L/insomnia001/home/rk3199/MUMPS_5.7.3/PORD/lib/ -lpord -L/path/shared/apps/scotch/7.0.5/lib64 -lptesmumps -lptscotch -lptscotcherr -L/path/shared/apps/oneapi/mkl/2024.0/lib -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/path/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/path/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread
ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x47): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x4aa): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x8cb): undefined reference to `zmumps_'
make[1]: *** [Makefile:44: zsimpletest] Error 1
make[1]: Leaving directory '/insomnia001/home/rk3199/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

[Mark_L_Intel]

Hello @RobbieTheK,

     It's not an endorsement, but I just tried this build, by i) cloning this GitHub, ii) sourcing the latest oneAPI and iii)  "cmake -B build; cmake --build build".  This successfully built everything. You can try this too. 

     I will try your way.  As you can see, the official Makefile build points to a linking error: "undefined reference to `zmumps_'".  Maybe, you can check LD_LIBRARY_PATH environment variable to include the MUMPS library path. 

[RobbieTheK]

Yes I tried that but try the examples lower in the page those fail with missing some scalapack library.

[Mark_L_Intel]

"missing ... scalapack"? I only see "undefined reference to `zmumps_'" above? 

[RobbieTheK]

The scalapack error happens with the git package when trying to cmake on the examples.

[Mark_L_Intel]

cmake --install build

cmake -S example -B example/build -DMUMPS_ROOT=build/local

cmake --build example/build

[ 25%] Building C object CMakeFiles/d_example_c.dir/d_example.c.o
[ 50%] Linking C executable d_example_c
[ 50%] Built target d_example_c
[ 75%] Building CXX object CMakeFiles/d_example_cpp.dir/d_example.cpp.o
[100%] Linking CXX executable d_example_cpp
[100%] Built target d_example_cpp

>mumps$

[RobbieTheK]

Can you clarify your comment here:

Maybe, you can check
LD_LIBRARY_PATH
environment variable to include the MUMPS library path.

If I'm compiling MUMPS how would I add the library path?

Here's the scalapack error I get when using the Git MUMPS you provided:

cmake -S example -B example/build -DMUMPS_ROOT=build/local
CMake Error at /insomnia001/home/rk3199/mumps/build/cmake/MUMPSConfig.cmake:31 (include):
include could not find requested file:

/insomnia001/home/rk3199/mumps/build/cmake/MUMPS-targets.cmake
Call Stack (most recent call first):
CMakeLists.txt:15 (find_package)

CMake Error at /cluster/shared/apps/cmake/3.30.2/share/cmake-3.30/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
By not providing "FindSCALAPACK.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"SCALAPACK", but CMake did not find one.

Could not find a package configuration file provided by "SCALAPACK" with
any of the following names:

SCALAPACKConfig.cmake
scalapack-config.cmake

Add the installation prefix of "SCALAPACK" to CMAKE_PREFIX_PATH or set
"SCALAPACK_DIR" to a directory containing one of the above files. If
"SCALAPACK" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/cluster/path/to/me/mumps/build/cmake/MUMPSConfig.cmake:51 (find_dependency)
CMakeLists.txt:15 (find_package)

-- Configuring incomplete, errors occurred!

[Mark_L_Intel]

a build with MUMPS_5.7.3:

1. cp Make.inc/Makefile.INTEL.PAR ./Makefile.inc

2. replaced old compilers with LLVM compilers in Makefile.inc

#CC = mpiicc
CC = mpiicx
#FC = mpiifort
#FL = mpiifort
FC = mpiifx
FL = mpiifx

3. sourced oneAPI tools

4. make z

a - zsol_fwd.o
a - zsol_matvec.o
a - zsol_root_parallel.o
a - ztools.o
a - ztype3_root.o
echo ../lib/libzmumps.a
../lib/libzmumps.a
make[2]: Leaving directory '/scratch/.../MUMPS_5.7.3/src'
make[1]: Leaving directory '/scratch/.../MUMPS_5.7.3/src'
cd examples; make z
make[1]: Entering directory '/scratch/.../MUMPS_5.7.3/examples'
mpiifx -O -nofor_main -qopenmp -DGEMMT_AVAILABLE -I. -I../include -I../src  -c zsimpletest.F -o zsimpletest.o
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o  -L../lib -lzmumps -lmumps_common  -L../PORD/lib/ -lpord  -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -lpthread
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
mpiifx -O -nofor_main -qopenmp -DGEMMT_AVAILABLE -I. -I../include -I../src  -c zsimpletest_save_restore.F -o zsimpletest_save_restore.o
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
mpiifx -o zsimpletest_save_restore -O -nofor_main -qopenmp zsimpletest_save_restore.o  -L../lib -lzmumps -lmumps_common  -L../PORD/lib/ -lpord  -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -lpthread
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
make[1]: Leaving directory '/scratch/.../MUMPS_5.7.3/examples'
/scratch/.../MUMPS_5.7.3$ ls examples
c_example.c                 input_simpletest_real           zsimpletest.F
c_example_save_restore.c    Makefile                        zsimpletest.o
csimpletest.F               multiple_arithmetics_example.F  zsimpletest_save_restore
csimpletest_save_restore.F  README                          zsimpletest_save_restore.F
dsimpletest.F               ssimpletest.F                   zsimpletest_save_restore.o
dsimpletest_save_restore.F  ssimpletest_save_restore.F
input_simpletest_cmplx      zsimpletest

[RobbieTheK]

Can you share how to install the oneapi LLVM, as a module like how the other oneapi modules are installed?

[Mark_L_Intel]

What I call LLVM compilers are the new compilers that should be already available when you load oneAPI compiler modules. The "old" or  "classic" compilers have been deprecated.  So for example, the classic icc compiler is not even available anymore:

 ls /opt/intel/oneapi/compiler/2024.2/bin
aocl-ioc64  fortcom           icx      ifort.cfg  opencl-aot         run-clang-tidy  xiar.cfg
codecov     fpp               icx-cc   ifx        profdcg            sycl-ls         xild
compiler    git-clang-format  icx.cfg  ifx.cfg    profmerge          sycl-trace      xild.cfg
dpcpp       icpx              icx-cl   ioc64      profmergesampling  tselect
dpcpp-cl    icpx.cfg          ifort    map_opts   proforder          xiar
mlubin@ortce-skl:~$

instead, you have to use the icx (LLVM-based) new compiler. In the case of Fortran, as you can see, ifort is still available, but it is better to use the ifx compiler. Since I was building with MPI, I used MPI wrappers that call new compilers. Therefore, you can see above mpiicx instead of mpiicc, and mpiifx instead of mpiifort. Just check after you load modules, "which icx", etc., to make sure that you got the new compilers. Or consult your cluster admin. Hope this helps.    

[RobbieTheK]

You didn't mention changing the path to MKLROOT

MKLROOT=/cluster/shared/apps/oneapi/mkl/2024.0/lib

Also your comment:

"Since I was building with MPI, I used MPI wrappers that call new compilers."

Did you create this wrapper? the mpiifx is in:

/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpiifx

Still fails:

cd examples; make z
make[1]: Entering directory '/cluster/path/to/me/MUMPS_5.7.3/examples'
mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o -L/cluster/home/rk3199/MUMPS_5.7.3/lib -lzmumps -lmumps_common -L/insomnia001/home/rk3199/MUMPS_5.7.3/PORD/lib/ -lpord -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread
/cluster/shared/apps/oneapi/mpi/2021.11/bin/mpiifx: line 702: ifx: command not found
make[1]: *** [Makefile:44: zsimpletest] Error 127
make[1]: Leaving directory '/cluster/path/to/me/MUMPS_5.7.3/examples'

I see a newer mpiifx in the HPCtoolkit

grep -R mpiifx /cluster/shared/apps/oneapi/hpctoolkit/
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/.toolkit_linking_tool/intel.oneapi.lin.mpi.devel-2021.11.0-49511.filelist:bin/mpiifx
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/bin/mpiifort:TARGET_WRAPPER="mpiifx"
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/lib/release: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/mpi/lib/release_mt: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib32/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib/ia32/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib/intel64: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/tbb/lib/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/.toolkit_linking_tool/.envs/2024.0/hpckit/compiler/modulefiles/oclfpga: No such file or directory
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/.toolkit_linking_tool/intel.oneapi.lin.mpi.devel-2021.11.0-49511.filelist:binmpiifx
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/bin/mpiifort:TARGET_WRAPPER="mpiifx"
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/lib/release: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/latest/lib/release_mt: warning: recursive directory loop
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/.toolkit_linking_tool/intel.oneapi.lin.mpi.devel-2021.11.0-49511.filelist:bin/mpiifx
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpiifort:TARGET_WRAPPER="mpiifx"
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/lib/release: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/lib/release_mt: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/2024.0/lib32/gcc4.8: warning: recursive directory loop
grep: /cluster/shared/apps/oneapi/hpctoolkit/2024.0/modulefiles/oclfpga: No such file or directory
/cluster/shared/apps/oneapi/hpctoolkit/2024.0/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" ;;
/cluster/shared/apps/oneapi/hpctoolkit/2024.0/bin/mpifc: ifort|ifx) "$dir"/mpiifx "$@" $opt_args ;;
/cluster/shared/apps/oneapi/hpctoolkit/2024.0/bin/mpiifort:TARGET_WRAPPER="mpiifx"

Also:

which icx
/cluster/shared/apps/oneapi/hpctoolkit/compiler/2024.0/bin/icx

What am I doing wrong here?

[Mark_L_Intel]

Hello @RobbieTheK,

    Answering your questions above:

• q.> In my environment, when I source oneAPI, /opt/intel/oneapi/setvars.sh, the MKLROOT is automatically set:  

  echo $MKLROOT
  /opt/intel/oneapi/mkl/2024.2​

• q.>"Since I was building with MPI, I used MPI wrappers that call new compilers." Did you create this wrapper? -- No, the mpiifx is already supplied with IMPI distribution, for example, in my case: 

  which mpiifx
  /opt/intel/oneapi/mpi/2021.13/bin/mpiifx

• q.: "/cluster/shared/apps/oneapi/mpi/2021.11/bin/mpiifx: line 702: ifx: command not found: suggest that ifx was not located on your system, In my case: 

  which ifx
  /opt/intel/oneapi/compiler/2024.2/bin/ifx

   please check if ifx is available on your system. 

• I would not try to use 2 different distributions at the same time (for example, what you call a newer distribution)

[RobbieTheK]

ml
Currently Loaded Modulefiles:
1) oneapi/hpctoolkit/hpctoolkit-2024.0.0 4) oneapi/compiler-rt/2024.0.2 7) scalapack/2.2.0 
2) oneapi/hpctoolkit/mpi/2021.11 5) oneapi/mkl/2024.0  scotch/7.0.5 
3) oneapi/tbb/2021.11 6) metis/5.1.0

Fri Sep 06 me@node $ which mpiifx
/cluster/shared/apps/oneapi/hpctoolkit/mpi/2021.11/bin/mpiifx
Fri Sep 06me@node $ which ifx
/cluster/shared/apps/oneapi/hpctoolkit/compiler/2024.0/bin/ifx
Fri Sep 06me@node $ echo $MKLROOT
/cluster/shared/apps/oneapi/mkl/2024.0

Same errors:

make[1]: Entering directory '/cluster/home/me/MUMPS_5.7.3/examples'
mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o -L/cluster/home/me/MUMPS_5.7.3/lib -lzmumps -lmumps_common -L/cluster/home/rk3199/MUMPS_5.7.3/PORD/lib/ -lpord -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/cluster/shared/apps/oneapi/mkl/2024.0/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x47): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x4aa): undefined reference to `zmumps_'
ld: zsimpletest.F:(.text+0x8cb): undefined reference to `zmumps_'
make[1]: *** [Makefile:44: zsimpletest] Error 1
make[1]: Leaving directory '/cluster/home/me/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

[Mark_L_Intel]

@RobbieTheK,

   In the comment before the last one, you had an "ifx not found" issue. I don't see it anymore. The undefined reference to zmumps_ when you compile examples means that there is a linking problem—the system cannot find the mumps library (that should be built at this point) while attempting to compile examples.  This is not oneMKL issue. 

[RobbieTheK]

OK so I went back and used the Git MUMPS and that compiled correctly. However we need MUMPS for compiling another application that uses nonlinear inversion algorithm (NLI) for Magnetic resonance elastography (MRE). That 'make fails with:

 make 
mpif90 -o MRE-Zone.discov -w -O3 -fno-range-check -ffixed-line-length-none -ffree-line-length-none -ffixed-line-length-0 -std=legacy -ldl MRE-Zone.o CGupdatescale.o FEgaussnewton.o fdchecks.o sensitivityanalysis.o FEmaterial.o FEmesh.o FEsolution.o exteriorconstraint.o gaussnewton.o hex27.o joachimowiczreg.o mcgarrybounds.o optimizationparams.o reporterror.o spatialfiltering.o tikhonovreg.o totalvariation.o zonefunctions.o FEforward.o FEgradient.o FEmatrix.o FEproblem.o fileio.o globalvariables.o hex8.o marquardtreg.o mpiparameters.o regularizationparams.o softpriorreg.o tet4.o timing.o vanhoutenreg.o zonevariables.o -L/cluster/shared/apps/metis/5.1.0/lib -lmetis -lmkl_sequential -limf -lmkl_core -ldl -lmkl_intel_lp64 -lmkl_gnu_thread -lpthread -lm -ldl 
/usr/bin/ld: MRE-Zone.o: in function `worldslave.37':
MRE-Zone.f90:(.text+0x40905): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40bd9): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40c0e): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40c57): undefined reference to `zmumps_'
/usr/bin/ld: MRE-Zone.f90:(.text+0x40ccb): undefined reference to `zmumps_'
/usr/bin/ld: FEgaussnewton.o:FEgaussnewton.f90:(.text+0xef3e): more undefined references to `zmumps_' follow
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_ordered_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_dynamic_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_guided_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_atomic_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_single_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_name_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_static_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_barrier'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_atomic_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_dynamic_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_num_procs'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_parallel'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_max_threads'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_dynamic_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_end_nowait'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_ordered_end'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_max_active_levels'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_in_parallel'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_critical_name_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_parallel_sections'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_guided_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_end_nowait'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_ordered_static_start'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_thread_num'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_loop_dynamic_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `GOMP_sections_next'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_get_num_threads'
/usr/bin/ld: /cluster/shared/apps/oneapi/mkl/2024.0/lib/libmkl_gnu_thread.so: undefined reference to `omp_set_max_active_levels'
collect2: error: ld returned 1 exit status
make: *** [Makefile:161: link]

Any idea why? 

[Mark_L_Intel]

Hello @RobbieTheK,

   "OK so I went back and used the Git MUMPS and that compiled correctly. " -- glad to see it.

The error messages like this: "libmkl_gnu_thread.so: undefined reference to omp_get_thread_num'"  typically indicate that there's an issue with linking the OpenMP library when using oneMKL with GNU compilers. Indeed, I see you used this flag

-lmkl_sequential

This means you ask for a non-threaded version of oneMKL while the application has OpenMP statements. Please use oneMKL Link Line Advisor to determine the correct build options. 

For example, I picked up Intel tools in Link Advisor and got the following link line and compiler options:

If I uncheck "Link with Inlte oneMKL libraries explicitly:", you get simplified version:

 

As a side comment, I'm not sure why you switched to GNU compilers --  since you used mpif90. To use Intel compilers, please use mpiifx. 

[RobbieTheK]

OK I used -lmetis -limf -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl  but alas still a zumps error, and I also changed to using mpiifx :

mpiifx -o MRE-Zone.discov -w -O3 -fno-range-check -ffixed-line-length-none -ffree-line-length-none -ffixed-line-length-0 -std=legacy -ldl MRE-Zone.o CGupdatescale.o FEgaussnewton.o fdchecks.o sensitivityanalysis.o FEmaterial.o FEmesh.o FEsolution.o exteriorconstraint.o gaussnewton.o hex27.o joachimowiczreg.o mcgarrybounds.o optimizationparams.o reporterror.o spatialfiltering.o tikhonovreg.o totalvariation.o zonefunctions.o FEforward.o FEgradient.o FEmatrix.o FEproblem.o fileio.o globalvariables.o hex8.o marquardtreg.o mpiparameters.o regularizationparams.o softpriorreg.o tet4.o timing.o vanhoutenreg.o zonevariables.o -L/cluster/shared/apps/metis/5.1.0/lib -lmetis -limf -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl 
ld: MRE-Zone.o: in function `mre_zone_IP_mumpsslavebuild_':
MRE-Zone.f90:(.text+0x6adc8): undefined reference to `zmumps_'
ld: MRE-Zone.o: in function `mre_zone_IP_mumpsslavesolve_':
MRE-Zone.f90:(.text+0x6bc2a): undefined reference to `zmumps_'
ld: MRE-Zone.f90:(.text+0x6bcc8): undefined reference to `zmumps_'
ld: MRE-Zone.f90:(.text+0x6bd8a): undefined reference to `zmumps_'
ld: MRE-Zone.f90:(.text+0x6be18): undefined reference to `zmumps_'
ld: FEgaussnewton.o:FEgaussnewton.f90:(.text+0x606): more undefined references to `zmumps_' follow
make: *** [Makefile:165: link] Error 1

[Mark_L_Intel]

"undefined reference to `zmumps_'" is not coming from oneAPI tools.  As I outlined before, it means that when you build examples, the library zmumps is not linked correctly -- maybe the link path is missing, etc.