Libiomp5.so Segmentation Fault

[necktweaker]

Hello,

I am attempting to compile and run the EPA's CMAQ air quality model. It compiles properly, however when I attempt to run the model, I get a segmentation fault in Libiomp5.so:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libiomp5.so 00002B07B6634AEA Unknown Unknown Unknown

I have read through some of the other posts regarding segmentation faults with libiomp5.so, and tried the information presented in those posts.

My stacksize is unlimited:
mturner@prospero:limit stacksize
stacksize unlimited

I have tried compiling with -heap-arrays as per http://software.intel.com/en-us/articles/openmp-option-no-pragmas-causes-segmentation-fault/

I have increased both OMP_STACKSIZE and KMP_STACKSIZE to 16M.

None of these have fixed my problem. I have also tried using the intel debugger (idbc) to run my executable however it gives me no useful information.

I am compiling with the most recent intel compilers for Linux:
Intel Fortran Intel 64 Compiler Professional for applications running on Intel 64, Version 11.1 Build 20100806 Package ID: l_cprof_p_11.1.073
Intel C Intel 64 Compiler Professional for applications running on Intel 64, Version 11.1 Build 20100806 Package ID: l_cproc_p_11.1.073

I am using the following compile and link flags, and libraries:
set FSTD = "-extend_source 132 -vec-report0 -nodefines -cm -w95 -traceback -CB -c -g -gen-interfaces -check -fpe0"
set LINK_FLAGS = "-lpthread -liomp5"
-L/usr/local/lib -lnetcdf
-L/data/libraries/pnetcdf/lib -lpnetcdf -L/data/libraries/hdf5/parallel/lib -lhdf5_hl -
lhdf5 -L/data/libraries/zlib/parallel/lib -lz -lm -L/usr/lib64 -lcurl -ldl -lgssapi_krb5 -lkrb5 -lk5
crypto -lkrb5support -lcom_err -lidn -lssl -lcrypto -lz -lresolv -lselinux -lsepol -lkeyutils

Also, when I compile CMAQ to run in parallel (with intel MPI library, mpiifort, and mpiicc) the model runs without error (using the same flags and include files). However, compiling and running in serial causes the segmentation fault.

Does anybody have any ideas?

Thanks,
Matt

[mecej4]

You might obtain more useful information by compiling your code with the -traceback option. That will enable you to find out which part of the CMAQ code made the call that caused libiomp5 to crash.

If the code was written with the assumption that there would be more than one thread, I can certainly understand that running it on a single thread would cause a crash. Consider an application with n threads, of which one is the master thread, which calls/coordinates n-1 slave threads to do various tasks. If you now run the code in serial mode, the number of slave threads becomes zero. If the code was not written to check for and cope with this eventuality, it may well crash.

You will have to think why it is so important to get the code to run in serial mode, and whether it is worthwhile to fix it so that it can function in serial mode.

[Martyn_C_Intel]

You describe your application as serial, but are linking to the OpenMP library libiomp5 ? Which part of the application or libraries uses OpenMP? If any of it uses OpenMP, you should include -openmp in your compile flags, even for code that contains no OpenMP directives. This is needed for thread safety.
When your app fails, are you running a single thread or multiple threads?

I would also try building without -check, as this adds to the complexity of the generated code.

[necktweaker]

Thank you both for your responses.

I figured out what the problem was (based on both responses). I had compiled IOAPI with -openmp. After recompiling IOAPI without -openmp, everything runs well now. The install instructions for CMAQ mention that IOAPI should be compiled after removing the -openmp flag from the Makeinclude file...I just didn't notice.

Thanks again for the responses.
Matt