Segfault in omp_destroy_lock

[Pieter_Vancraeyveld]

Hi all,

Compiling the following code with icpc -openmp -ggdb results in a segmentation fault.

[cpp]
#include 
#include 

class Test
{
private:
  omp_lock_t lock_;
public:
  Test() {
    std::cout << "Init " << &lock_ << std::endl;
    omp_init_lock( &lock_ );
  }
  ~Test() {
    std::cout << "Destroy " << &lock_ << std::endl;
    omp_destroy_lock( &lock_ );
  }
};

const Test test[2];

int main()
{
  return 0;
}[/cpp]

The backtrace is

[bash]Program received signal SIGSEGV, Segmentation fault.
0x00002ac861c81cfb in __kmpc_destroy_lock () from /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so
(gdb) bt                                                                                              
#0  0x00002ac861c81cfb in __kmpc_destroy_lock () from /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so
#1  0x00002ac861c91282 in omp_destroy_lock () from /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so   
#2  0x0000000000401390 in Test::~Test (this=0x601e88) at test.cpp:16                                      
#3  0x00000000004015e8 in __$U0 ()                                                                        
#4  0x00002ac862561242 in __run_exit_handlers (status=0, listp=0x2ac8628854a8, run_list_atexit=true) at exit.c:78
#5  0x00002ac862561295 in exit (status=0) at exit.c:100                                                          
#6  0x00002ac86254b924 in __libc_start_main (main=, argc=, ubp_av=, 
    init=, fini=, rtld_fini=, stack_end=0x7fff51e1cd48) at libc-start.c:252
#7  0x0000000000401219 in _start () at ../sysdeps/x86_64/elf/start.S:113[/bash]

Valgrind reports an error in Test::Test that possible leads to the invalid read in the destructor

[bash]Init 0x601e80                                                                                                                         
==3785== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)                                                            
==3785==    at 0x58B8B69: syscall (syscall.S:39)                                                                                          
==3785==    by 0x40C7379: __kmp_affinity_determine_capable (in /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so)                      
==3785==    by 0x40AF2ED: __kmp_env_initialize(char const*) (in /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so)                     
==3785==    by 0x40A5DFC: __kmp_serial_initialize (in /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so)                               
==3785==    by 0x4092CBA: omp_init_lock (in /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so)                                         
==3785==    by 0x40132F: Test::Test() (test.cpp:12)                                                                                       
==3785==    by 0x401440: __sti__$E (test.cpp:0)                                                                                           
==3785==    by 0x4016E5: ??? (in /home/pieter/strangeProd/codeDev/test/testOpenmp/test_icc)                                               
==3785==    by 0x401092: ??? (in /home/pieter/strangeProd/codeDev/test/testOpenmp/test_icc)                                               
==3785==  Address 0x0 is not stack'd, malloc'd or (recently) free'd                                                                       
==3785==                                                                                                                                  
Init 0x601e88                                                                                                                         
Destroy 0x601e88                                                                                                                       
==3785== Invalid read of size 8                                                                                                           
==3785==    at 0x4082CFB: __kmpc_destroy_lock (in /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so)                                   
==3785==    by 0x4092281: omp_destroy_lock (in /opt/intel/Compiler/11.1/064/lib/intel64/libiomp5.so)                                      
==3785==    by 0x40138F: Test::~Test() (test.cpp:16)                                                                                      
==3785==    by 0x4015E7: __$U0 (test.cpp:0)                                                                                               
==3785==    by 0x5819241: __run_exit_handlers (exit.c:78)                                                                                 
==3785==    by 0x5819294: exit (exit.c:100)                                                                                               
==3785==    by 0x5803923: (below main) (libc-start.c:252)                                                                                 
==3785==  Address 0x10 is not stack'd, malloc'd or (recently) free'd[/bash]

Compiling the same code with gcc doesn't result in a segmentation fault. Is something wrong with the sample code or am I missing something else?

I use icc version 11.1 20091130 and gcc version 4.4.1.

[Michael_K_Intel2]

Hi!

I can confirm your segmentation fault with my compiler version. The problem seems to ocurr when Test is a global variable that is destroyed by the exit handlers. If you move the test variable into main, the code starts working.

I'll forward your problem to one of the OpenMP runtime guys and ask him to get back to you with further advice.

Cheers,

-michael

[Lev_N_Intel · Accepted solution]

Hi Pieter,

It is a known issue: Intel OpenMP RTL shuts down at the end of main function. When destructor of global instance of class Test is invoked, OpenMP RTL is already uninitialized, so omp_destroy_lock causes segmentation fault.

The problem will be fixed in Intel Compiler 11.1 update 6.

For now, you may workaround the problem by moving main function into a separate file, which is compiled without -openmp option.

[jimdempseyatthecove]

This may be a case of order in which the atexit functions are linked into the list of exit functions

My guess is omp_init_lock in your ctor is not initializing the OpenMP runtime system, rather simply setting the initial lock contents. Then at program end, the omp_destroy_lock is calling OpenMP runtime system functions that _require_ OpenMP runtime not having exited.

There may be a potential work around.

In your ctor try

...
#pragma omp parallel
{
if(omp_get_thread_num() == 0)
omp_init_lock(&lock);
}
...

What this will do (in theory) is start up OpenMP _prior_ to main. Then later, exit OpenMP _after_ dtor in static object.

If this works, let the rest of us know.

Jim Dempsey

[Lev_N_Intel]

> This may be a case of order in which the atexit functions are linked into the list of exit functions

No. Intel compiler at the end of main generates implicit call to non-documented OpenMP RTL function, which shuts down the library.

> My guess is omp_init_lock in your ctor is not initializing the OpenMP runtime system...

No. omp_init_lock properly initializes the OpenMP RTL.

> In your ctor try

> ...
> #pragma omp parallel
> {
> if(omp_get_thread_num() == 0)
> omp_init_lock(&lock);
> }
> ...

No. As I said above, omp_init_lock properly initializes OpenMP RTL, no need in parallel region. This is overkill.

The problem is not in initialization, the problem is in early termination of the library. Implicit library shut down at the end of main can be effectively disabled by compiling main without -openmp option, as I recommended in the previous answer.

[jimdempseyatthecove]

Lev,

I hope the OpenMP shutdown call gets properly sequenced in next (later) release. A static object should be able to make an omp call that initialized the library and then have the library destroy itself on the way back out. Similar to how static objects get destroyed in reverse order from contstruction. The support routined for how this is done is in the CPP runtime library startup function. It should be QED to model the OpenMP shutdown off of this.

Jim