OMP: Error #13: Assertion failure at kmp_csupport.c(492)

[rafadix08]

Hi,
I had the following error message running my fortran code on a UNIX cluster:

OMP: Error #13: Assertion failure at kmp_csupport.c(492).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.

I am using Intel Fortran version 11.0, MKL libraries and OpenMP for parallelization. I am optimizing a function that is computed by simulation. Parallelization is used in order to faster compute the function value at any point. The function was successfully computed at several different points proposed by the optimization algorithm (about 3000 iterations) before I received this message.

Please let me know if you need further information.

Many thanks,
Rafael

[Steven_L_Intel1]

Please supply all the information listed in the error message, including a test case that shows the problem. You may also want to try with the current 11.1.

[rafadix08]

Hi Steve,

The error message reads:

OMP: Error #13: Assertion failure at kmp_csupport.c(492).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.

I am attaching the source files together with the necessary data. I also included the executable obtaining with the following command:

ifort trsapp.f bigden.f newuoa.f update.f biglag.f newuob.f Global_Data.f90 LinReg_MOD.f90 minim.f90 Parallel_Emax_MOD.f90 Loss_Function_IndInf_MOD.f90 calfun.f90 Main.f90 -o estimation_IndInf -L$MKLPATH -I$MKLINCLUDE -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -openmp


I repeatedly ran the same source code and the program crashes at different points. There is an optimization routine that iterates over possible parameters and the program crashes at different iterations, everything else equal (starting point and other constants).

Please let me know if you need more information.

Many thanks,
Rafael

[rafadix08]

I inlcuded the wrong executable in the post above (in the post above I disabled the parallelization). Here is the folder with the right executable, which uses the parallelization.

[Ron_Green]

Quoting - rafadix08

I inlcuded the wrong executable in the post above (in the post above I disabled the parallelization). Here is the folder with the right executable, which uses the parallelization.

Rafael,

How many cores did you run this on? And did you set up MKL to use 1 thread or as many threads as available? Since both MKL is threaded and your code is threaded, how did you control the number of threads used by MKL versus how many threads were used by the OMP regions - or do the OMP regions not call MKL?

ron

[rafadix08]

Quoting - Ronald W. Green (Intel)

Rafael,

How many cores did you run this on? And did you set up MKL to use 1 thread or as many threads as available? Since both MKL is threaded and your code is threaded, how did you control the number of threads used by MKL versus how many threads were used by the OMP regions - or do the OMP regions not call MKL?

ron

Hi Ron,

I am not sure I understood your question, but I will try to list the information that may be relevant for you.

The cluster has 8 available cores.

The parallization is conductedwithin a function located in module Parallel_Emax_MOD.f90 and I set:
Call omp_set_num_threads(4)

In thisfunction I call the same function but with different arguments in parallel. This function called in the parallel regionuses the MKL library internally. However, I do not explicitly call MKL functions within the parallel region.

I do not set up the number of threads MKL should be using. I am not familiar with this. Should I be doing something differently?

Many thanks,
Rafael

[TimP]

When you call threaded MKL inside an OpenMP parallel region, as you appear to have done, it seems to me MKL should not start additional threads unless you set OMP_NESTED, and additional threads are available within your requested number. If you don't intend for MKL to start additional threads, it might be better to link mkl_sequential. It shouldn't be wrong to do as you have done; the whole thing should be made safer if possible. I'm sure Ron is asking just to get a clue what to look for.

[Ron_Green]

Quoting - tim18

When you call threaded MKL inside an OpenMP parallel region, as you appear to have done, it seems to me MKL should not start additional threads unless you set OMP_NESTED, and additional threads are available within your requested number. If you don't intend for MKL to start additional threads, it might be better to link mkl_sequential. It shouldn't be wrong to do as you have done; the whole thing should be made safer if possible. I'm sure Ron is asking just to get a clue what to look for.

Rafael,

I need to know the specific version of the compiler you are using:

ifort -V

will give this.

I compiled using the MKL Linkage wizard which is here:
http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/

which gave me this compile/linkage line:

ifort trsapp.f bigden.f newuoa.f update.f biglag.f newuob.f Global_Data.f90 LinReg_MOD.f90 minim.f90 Parallel_Emax_MOD.f90 Loss_Function_IndInf_MOD.f90 calfun.f90 Main.f90 -o estimation_IndInf -L$MKLPATH -I$MKLINCLUDE $MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -openmp -lpthread -openmp

I used the 11.1 compiler. I also had to unlimit the stack:

ulimit -s unlimited

OR you can compile with option -heap-arrays

I ran this out 300 iterations or so, about 20 minutes. But you said the fault is way out in iteration 3000? How long does it take to get there?

Can you also compile with -g -traceback ? This will give us more information where the program is blowing up.

ron

[Ron_Green]

I ran with the latest 11.1 compiler all afternoon. It got to iteration 2156 and then it restarted at iteration 1 - is this expected?

No crashes, however. I can try again with an older 11.0 compiler once you have told me the exact version you are using.

ron