MUMPS many undefined references to `zmumps_' `__kmpc_critical_with_hint'

[RobbieTheK]

In attempting to compile MUMPS, it fails on the examples with the below errors. 

Here are the modules loaded on RHEL 9 cluster. Is there some incompatibility with Intel oneAPI 2024 mkl? Is it possible to skip compiling all of the examples? Same exact errors at this other thread or even this one from 2003?

scotch/7.0.4, parmetis/4.0.3, scalapack/2.2.0, oneapi/mkl/2024.0, oneapi/mpi/2021.11

Warning: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)).
ar vr ../lib/libzmumps.a zana_aux.o zana_aux_par.o zana_lr.o zfac_asm_master_ELT_m.o zfac_asm_master_m.o zfac_front_aux.o zfac_front_LU_type1.o zfac_front_LU_type2.o zfac_front_LDLT_type1.o zfac_front_LDLT_type2.o zfac_front_type2_aux.o zfac_sispointers_m.o zfac_lr.o zfac_mem_dynamic.o zfac_omp_m.o zfac_par_m.o zfac_sol_l0omp_m.o zlr_core.o zlr_stats.o zlr_type.o zmumps_comm_buffer.o zmumps_config_file.o zmumps_load.o zmumps_lr_data_m.o zmumps_ooc_buffer.o zmumps_ooc.o zmumps_sol_es.o zmumps_save_restore.o zmumps_save_restore_files.o zmumps_struc_def.o zomp_tps_m.o zsol_lr.o zsol_omp_m.o zstatic_ptr_m.o zmumps_mpi3_mod.o zini_driver.o zana_driver.o zfac_driver.o zsol_driver.o zsol_distrhs.o zsol_distsol.o zend_driver.o zana_aux_ELT.o zana_dist_m.o zana_LDLT_preprocess.o zana_reordertree.o zarrowheads.o zbcast_int.o zfac_asm_ELT.o zfac_asm.o zfac_b.o zfac_diag.o zfac_distrib_distentry.o zfac_dist_arrowheads_omp.o zfac_distrib_ELT.o zfac_lastrtnelind.o zfac_mem_alloc_cb.o zfac_mem_compress_cb.o zfac_mem_free_block_cb.o zfac_mem_stack_aux.o zfac_mem_stack.o zfac_process_band.o zfac_process_blfac_slave.o zfac_process_blocfacto_LDLT.o zfac_process_blocfacto.o zfac_process_bf.o zfac_process_end_facto_slave.o zfac_process_contrib_type1.o zfac_process_contrib_type2.o zfac_process_contrib_type3.o zfac_process_maprow.o zfac_process_master2.o zfac_process_message.o zfac_process_root2slave.o zfac_process_root2son.o zfac_process_rtnelind.o zfac_root_parallel.o zfac_scalings.o zfac_determinant.o zfac_scalings_simScaleAbs.o zfac_scalings_simScale_util.o zfac_sol_pool.o zfac_type3_symmetrize.o zini_defaults.o
ar: creating ../lib/libzmumps.a
a - zana_aux.o
a - zana_aux_par.o
a - zana_lr.o
a - zfac_asm_master_ELT_m.o
a - zfac_asm_master_m.o
a - zfac_front_aux.o
a - zfac_front_LU_type1.o
a - zfac_front_LU_type2.o
a - zfac_front_LDLT_type1.o
a - zfac_front_LDLT_type2.o
a - zfac_front_type2_aux.o
a - zfac_sispointers_m.o
a - zfac_lr.o
a - zfac_mem_dynamic.o
a - zfac_omp_m.o
a - zfac_par_m.o
a - zfac_sol_l0omp_m.o
a - zlr_core.o
a - zlr_stats.o
a - zlr_type.o
a - zmumps_comm_buffer.o
a - zmumps_config_file.o
a - zmumps_load.o
a - zmumps_lr_data_m.o
a - zmumps_ooc_buffer.o
a - zmumps_ooc.o
a - zmumps_sol_es.o
a - zmumps_save_restore.o
a - zmumps_save_restore_files.o
a - zmumps_struc_def.o
a - zomp_tps_m.o
a - zsol_lr.o
a - zsol_omp_m.o
a - zstatic_ptr_m.o
a - zmumps_mpi3_mod.o
a - zini_driver.o
a - zana_driver.o
a - zfac_driver.o
a - zsol_driver.o
a - zsol_distrhs.o
a - zsol_distsol.o
a - zend_driver.o
a - zana_aux_ELT.o
a - zana_dist_m.o
a - zana_LDLT_preprocess.o
a - zana_reordertree.o
a - zarrowheads.o
a - zbcast_int.o
a - zfac_asm_ELT.o
a - zfac_asm.o
a - zfac_b.o
a - zfac_diag.o
a - zfac_distrib_distentry.o
a - zfac_dist_arrowheads_omp.o
a - zfac_distrib_ELT.o
a - zfac_lastrtnelind.o
a - zfac_mem_alloc_cb.o
a - zfac_mem_compress_cb.o
a - zfac_mem_free_block_cb.o
a - zfac_mem_stack_aux.o
a - zfac_mem_stack.o
a - zfac_process_band.o
a - zfac_process_blfac_slave.o
a - zfac_process_blocfacto_LDLT.o
a - zfac_process_blocfacto.o
a - zfac_process_bf.o
a - zfac_process_end_facto_slave.o
a - zfac_process_contrib_type1.o
a - zfac_process_contrib_type2.o
a - zfac_process_contrib_type3.o
a - zfac_process_maprow.o
a - zfac_process_master2.o
a - zfac_process_message.o
a - zfac_process_root2slave.o
a - zfac_process_root2son.o
a - zfac_process_rtnelind.o
a - zfac_root_parallel.o
a - zfac_scalings.o
a - zfac_determinant.o
a - zfac_scalings_simScaleAbs.o
a - zfac_scalings_simScale_util.o
a - zfac_sol_pool.o
a - zfac_type3_symmetrize.o
a - zini_defaults.o
ranlib ../lib/libzmumps.a
make[2]: Leaving directory '/path/to/me/MUMPS_5.7.3/src'
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/src'
cd examples; make z
make[1]: Entering directory '/path/to/me/MUMPS_5.7.3/examples'
mpif90 -O -std=legacy -DBLR_MT -fopenmp -fallow-argument-mismatch -I. -I../include -I../src -I../libseq -c zsimpletest.F -o zsimpletest.o
mpif90 -o zsimpletest -O -fopenmp -fallow-argument-mismatch zsimpletest.o -L../lib -lzmumps -lmumps_common -L/insomnia001/shared/apps/metis/5.1.0/lib/ -lmetis -L../PORD/lib/ -lpord -L/insomnia001/shared/apps/scotch/7.0.4/lib -lesmumps -lscotch -lscotcherr -L/insomnia001/shared/apps/oneapi/mkl/2024.0 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L../libseq -lmpiseq -L/usr/lib64 -lblas -lpthread
/usr/bin/ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x4f): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x5f4): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x628): undefined reference to `zmumps_'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_critical_with_hint'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_critical'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_master'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_fini'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_reduce'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_flush'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_fork_call'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_push_num_threads'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_ordered'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task_alloc'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_8u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_barrier'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_cmplx8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_global_thread_num'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_4'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_critical'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_serialized_parallel'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_fixed4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_omp_task_with_deps'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_reduce_nowait'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float4_max'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_master'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_single'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_fini_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_for_static_init_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_init_8'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_fixed8_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_cmplx4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_ok_to_fork'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_single'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_dispatch_next_4u'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_ordered'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_end_reduce_nowait'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_serialized_parallel'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float4_add'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_atomic_float8_max'
/usr/bin/ld: /path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined reference to `__kmpc_reduce'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:41: zsimpletest] Error 1
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

[RobbieTheK]

Any suggestions on where to look? In the compiled MUMPS 5.7.3 I have in ~/MUMPS/5.7.3/include and MUMPS/5.7.3/examples/include these zmumps header files: 

-rwxrwxr-- 1 me user 4133 May 2 06:15 zmumps_c.h
-rwxrwxr-- 1 me user 2312 May 2 06:15 zmumps_root.h
-rwxrwxr-- 1 me user 12294 Jun 10 23:26 zmumps_struc.h

[Mark_L_Intel]

These are header files. You have issues with linking to mumps library. All I do is "make z" from the main distribution directory MUMPS_5.7.3. By the time you go into examples, you must have library files built: 

MUMPS_5.7.3$ ls lib
libmumps_common.a  libpord.a  libzmumps.a  README

  After make process builds libary, it goes into examples directory and builds examples:

echo ../lib/libzmumps.a
../lib/libzmumps.a
make[2]: Leaving directory '/scratch/users/mlubin/triage/MUMPS_5.7.3/src'
make[1]: Leaving directory '/scratch/users/mlubin/triage/MUMPS_5.7.3/src'
cd examples; make z
make[1]: Entering directory '/scratch/users/mlubin/triage/MUMPS_5.7.3/examples'
mpiifx -O -nofor_main -qopenmp -DGEMMT_AVAILABLE -I. -I../include -I../src  -c zsimpletest.F -o zsimpletest.o
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o  -L../lib -lzmumps -lmumps_common  -L../PORD/lib/ -lpord  -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -lpthread
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
mpiifx -O -nofor_main -qopenmp -DGEMMT_AVAILABLE -I. -I../include -I../src  -c zsimpletest_save_restore.F -o zsimpletest_save_restore.o
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
mpiifx -o zsimpletest_save_restore -O -nofor_main -qopenmp zsimpletest_save_restore.o  -L../lib -lzmumps -lmumps_common  -L../PORD/lib/ -lpord  -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -lpthread
ifx: command line warning #10435: option '-nofor_main' use with underscore is deprecated; use '-nofor-main' instead
make[1]: Leaving directory '/scratch/users/mlubin/triage/MUMPS_5.7.3/examples'

as you can see you should have commands like this:

mpiifx -o zsimpletest -O -nofor_main -qopenmp zsimpletest.o  -L../lib -lzmumps -lmumps_common  -L../PORD/lib/ -lpord  -L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core  -lpthread

with options:

 -L../lib -lzmumps -lmumps_common 

that link example with mumps library. I'm looking into your build log you provided I don't see these commands. I don't know why. Reiterating, I simply source oneAPI tools and then do "make z" -- and everything builds smoothly.

[RobbieTheK]

I believe I got MUMPS to compile. However the actual application that requires MUMPS to compile is where I'm gettin the error. See attached go into the MREv9p37a directory and run make.

[Mark_L_Intel]

Hello @RobbieTheK,

        There is no error associated with oneMKL in your latest submission.  

[RobbieTheK]

Were you able to download the source code I uploaded and attempt to compile with MUMPS?

[Mark_L_Intel]

Hello @RobbieTheK,

    I downloaded your source code and it is clear (as I mentioned above) that the errors you are seeing now are asscoiated with MUMPS distribution itself rather than with oneMKL. This Forum is dedicated to oneMKL issues. Therefore, this post will no longer will be monitored by Intel.  I'd like to thank you for your contributions to this Forum.