Intel 15.0.2 OMP Runtime Error

[LM11]

I received the following error while running a Fortran code in parallel:

OMP: Error #13: Assertion failure at kmp_runtime.c(1592).

OMP: Hint: Please submit a bug report with this message, compiler and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/product/support/.

I am not able to share the source code but my configurations are as follows:

Compiler Version: 15.0.2

Native System Compiler: gcc-4.4.7

MPI Version: OpenMPI-1.8.2

OS: RHEL 6.5  2.6.32-431.20.3

[Steven_L_Intel1]

Sorry, but there's nothing we could tell you without a test case. You can report through Intel Premier Support if you need privacy.

[jimdempseyatthecove]

Does a simple "MPI/parallel hello world" build and run on that system/cluster? (using the same build options and configuration settings)

Jim Dempsey

[LM11]

I am able to run multiple jobs with the listed Compiler/MPI versions, ranging from simple hello world examples to production level codes.  I have also been able to run this particular code all the way through the test case, but it happened to crash with the error listed above.  In addition, I have seen some inconsistencies with the results, which can be prevented by setting MKL_NUM_THREADS=1 instead of or default value of 4.  It almost seems like there is a threading conflict in the code for certain race conditions.

We have also tried using MPICH-3.1.4 and that appears to run fine with the MKL_NUM_THREADS=4, and all the results are consistent.

I recently tried OpenMPI-1.8.4 , but the inconsistencies in the results are still present.  Along with the consistency the code does periodically hang and prevent MPI from crashing and exiting, which is another reason I am interested the KMP_runtime error since it would be controlling the threads to an extent.

[TimP]

With Mkl set to 4 threads each mpi process will take 4 cores. Might your error be due to oversubscription?

[LM11]

I don't believe so, we also make use of hardware locality tools using --bind-to and --map-by to ensure the proper placement of MPI procs in conjunction with the MKL_NUM_THREADS settings.