Windows: libiomp5md: reports multiple initialisation

[Terry]

The context:

I am developing a small cmake C++ project with multiple dependencies including mkl; the project has a very heavy scientific workload and is intended to scale to large server contexts when running. 

The (final) problem: 
OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

To the best of my knowledge my code - which has two internal components, an executable and a single (home brewed) dll, and I believe it links exactly once to the libiomp5md as well as to the MKL dynamic libraries.

The omp is used by the Eigen3 headers only library and also by the MKL. 

I attach the two MAP files from linking. I have successfully excluded the VCOMP library from the build of each of these components etc.

This is a breaking problem for me.  I need to scale the calculations and cannot have an unreliable solution (the results are maths!!). I need the parallelization working.

It is clear that this duplicate initialization problem is widespread over time with reports on this web site from 2012 but there are no "sensible" proposed solutions and the links are dead.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I am using

Microsoft Visual Studio Enterprise 2019
Version 16.11.9
VisualStudio.16.Release/16.11.9+32106.194
Microsoft .NET Framework
Version 4.8.04084

Installed Version: Enterprise

Visual C++ 2019 00435-60000-00000-AA626
Microsoft Visual C++ 2019

Intel Libraries for oneAPI Package ID: w_oneAPI_2021.4.0.364
Intel Libraries for oneAPI – toolkit version: 2021.4.0, extension version 21.1.0.16, Package ID: w_oneAPI_2021.4.0.364, Copyright © 2019-2021 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® Advisor 2021.4 610727
Intel® Advisor 2021.4, (build 610727), Copyright © 2009-2021 Intel Corporation. All rights reserved.

Intel® C++ Compiler Package ID: w_comp_lib_2020.4.311
Intel® C++ Compiler – extension version 19.1.0.16, Package ID: w_comp_lib_2020.4.311, Copyright © 2002-2020 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® C++ Compiler Package ID: w_oneAPI_2021.4.0.364
Intel® C++ Compiler – toolkit version: 2021.4.0, extension version 21.0.16, Package ID: w_oneAPI_2021.4.0.364, Copyright © 2002-2021 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® DPC++ Compatibility Tool Package ID: w_oneAPI_2021.4.0.398
Intel® DPC++ Compatibility Tool – toolkit version: 2021.4.0, extension version 2021.1.0.16, Package ID: w_oneAPI_2021.4.0.398, Copyright © 2019-2021 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® oneAPI DPC++ Compiler Package ID: w_oneAPI_2021.4.0.364
Intel® oneAPI DPC++ Compiler – toolkit version: 2021.4.0, extension version 21.1.0.16, Package ID: w_oneAPI_2021.4.0.364, Copyright © 2019-2021 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® oneAPI Menu & Samples 1.1.419
Intel® oneAPI Visual Studio Menu & Samples Extension

Intel® Performance Libraries Package ID: w_comp_lib_2020.4.311
Intel® Performance Libraries – extension version 19.1.0.16, Package ID: w_comp_lib_2020.4.311, Copyright © 2002-2020 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® Visual Fortran Compiler Package ID: w_comp_lib_2020.4.311
Intel® Visual Fortran Compiler – extension version 19.1.0057.16, Package ID: w_comp_lib_2020.4.311, Copyright © 2002-2020 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.

Intel® VTune™ Profiler 2021 619561
Intel® VTune™ Profiler 2021, (build 619561), Copyright © 2009-2021 Intel Corporation. All rights reserved.

[ShanmukhS_Intel]

Hi,

Thanks for posting on Intel Communities.

We would like to request you to share a sample reproducer and steps(if any) so that we could try investigating your issue at our end.

In addition,  We would like to recommend you to clean the solution file and build the same after performing any configuration changes so that they would get reflected in your project configuration.

Best Regards,

Shanmukh.SS

[Terry]

Hi Shanmukh.SS

Thanks for your response. I will try. The problem seems very consistent.

Below is my basic setup from which all else derives. If I run code using the MKL all is well - including using code run from Eigen with MKL switched on in Eigen; however, as soon as one uses Eigen with MKL switched off or the simplest OMP omp parallel for in my code outside the MKL/LAPACK, while also using MKL/LAPACK, the fated errors of duplicate initialization appear. The uses are not nested. 

Is this enough info for you - the cmake is cut and pasted from my base cmakelists.txt file and works for me!  It assumes UnitTest++, Eigen3, MKL are available (I use vcpkg to achieve this). The problems occur with VS2019 and VS2022 (Enterprise). The failing examples are framed as unit tests. I can probably send you code that breaks this but only if you can get a basic multi-package framework up and running. Let me know.

Best Regards

Terry

cmake_minimum_required(VERSION 3.20)
project(RootProject)

if(MSVC)
#string(APPEND CMAKE_CXX_FLAGS " /Zc:__cplusplus")
endif()

message(STATUS "Final project output is placed in ${CMAKE_BINARY_DIR}")
# collect binaries together in common location so that modules and dlls find each other at build time
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)

######################################packages#############################################
# provides unit test capability via UnitTest++
find_package(UnitTest++ CONFIG REQUIRED)
# target_link_libraries(main PRIVATE UnitTest++)
find_path(UnitTest++_INCLUDE_DIR "UnitTest++/UnitTest++.h")
set(UnitTest++_INCLUDE_DIRS ${UnitTest++_INCLUDE_DIR})

# provides boost functionality via Boost::boost
set(Boost_NO_WARN_NEW_VERSIONS 1)
find_package(Boost COMPONENTS filesystem iostreams chrono random serialization REQUIRED)

# provides access to the Intel/MKL version of lapack via LAPACK::LAPACK as Eigen needs to see mkl.h
set (BLA_SIZEOF_INTEGER 4) # required for eigen
set(BLA_VENDOR Intel10_64lp)
set(BLA_STATIC 0 CACHE BOOL "force nonstatic linking in lapack" FORCE)
find_package(LAPACK)
If(LAPACK_FOUND AND MSVC )
## global flags to stop mixed omp versioning
## check correct library chosen (must have the vcomp entries)
add_compile_options(/MD$<$<CONFIG:Debug>:d>)
add_compile_definitions(_OPENMP_NOFORCE_MANIFEST)
add_link_options(/nodefaultlib:vcomp$<$<CONFIG:Debug>:d>)
endif()

If(LAPACK_FOUND)
## add mkl headers include to LAPACK::LAPACK
## ensure they are consistent with the discovered version from mkl_blas
cmake_path(GET BLAS_mkl_core_dll_LIBRARY PARENT_PATH MKL_LIB_PATH)
cmake_path(APPEND_STRING MKL_LIB_PATH "/../../include" OUTPUT_VARIABLE MKL_HEADER_PATH)
cmake_path(SET MKL_HEADER_PATH NORMALIZE "${MKL_HEADER_PATH}")
## validate the path
find_path(MKL_H mkl.h PATHS "${MKL_HEADER_PATH}" REQUIRED NO_DEFAULT_PATH)

target_include_directories(LAPACK::LAPACK INTERFACE
"$<BUILD_INTERFACE:${MKL_HEADER_PATH}/../include>"
"$<INSTALL_INTERFACE:include>" # <prefix>/include
)
endif()

# provides OpenMP via OpenMP::OpenMP_CXX
find_package(OpenMP)
if(OpenMP_CXX_FOUND AND MSVC)
## allow simd extension to MS OMP v2.0
set_property(TARGET OpenMP::OpenMP_CXX
PROPERTY INTERFACE_COMPILE_OPTIONS $<$<COMPILE_LANGUAGE:CXX>:-openmp:experimental>)
endif()

# provides C++ templated linear algebra Eigen3::Eigen
find_package(Eigen3 CONFIG REQUIRED)
if(Eigen3_FOUND AND LAPACK_FOUND)
# use mkl for the linear algbra over the fp numbers
add_compile_definitions(EIGEN_USE_MKL_ALL)
endif()

[ShanmukhS_Intel]

Hi,

>>I can probably send you code that breaks this but only if you can get a basic multi-package framework up and running.

We have downloaded the mentioned dependencies(UnitTest++, Eigen3, MKL) for Windows. Could you please share the code and environmental settings to be configured in your Visual Studio(if possible please share your vcproj file), so that we could try reproducing it at our end.

>>If I run code using the MKL all is well - including using code run from Eigen with MKL switched on in Eigen; however, as soon as one uses Eigen with MKL switched off

As suggested, Could you please let us know any progress after cleaning the solution file and build the same after performing any configuration changes.

In addition, Could you please cross verify if there is some sort of environmental issue at your end. 

Best regards.

Shanmukh.SS

[Terry]

Hi, Sorry for the delay; thanks for making progress. I am afraid I had given up on you and did not see this progress until now. I am happy to share my troublesome code but it will take a little time.

As the code is (although managed in visual studio) cmake based, and ninja is the underlying build system, there is no vcproj, and the build is clean every time - so I do not really recognise the remark you are mentioning about the sln file.  

I will try try upload exactly the breaking code. Is there a private way of doing this?

I have been trying to produce a minimal reduced version of the problem for you; it seems this is not predictable or straight forward. It looks to me as if there is an issue that is similar to that of unpredictable order of static initialisation in C++ and as if the exact configuration of the binary plays a role in the trouble. Is that possible.

Best

Terry

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I also found the following on the web (what is the windows equivalent of the suggested work around):

Multiple OpenMP Loadings

If the application uses the Inference Engine with third-party components that depend on Intel® OpenMP, multiple loadings of the libiomp library may occur and cause OpenMP runtime initialization conflicts. This might happen if the application uses the Intel® Math Kernel Library (Intel® MKL) through the “Single Dynamic Library” (libmkl_rt.so) mechanism and calls Intel® MKL after loading the Inference Engine plugin.

Error log report:

OMP: Error #15: Initializing libiomp5.so, but found libiomp5.so already initialized.OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, see http://www.intel.com/software/products/support/.

Possible workarounds:

• Preload the OpenMP runtime using the LD_PRELOAD variable:
  This eliminates multiple loadings of libiomp, and makes all components use this specific version of OpenMP.

LD_PRELOAD=<path_to_libiomp5.so] <path_to your_executable]

• Set KMP_DUPLICATE_LIB_OK=TRUE. This option might result in performance degradation or incorrect results.

[ShanmukhS_Intel]

Hi,

Reminder:

Has the information provided helped? Is your issue resolved?

Kindly let us know so that we could look into your issue further.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi,

>>Is there a private way of doing this?

Thanks for getting back to us. As requested by you, we will be connecting with you privately further on this case.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi,

We have not heard back from you. This thread will no longer be monitored by Intel. If you need further assistance, please post a new question. 

Best Regards,

Shanmukh.SS