OpenMP code works on 11.1.056 but don't on 14.0.3

[Luiz_Felipe_S_]

Hi guys!

   I have a simple code that depends on OpenMP locks like this.

#include <stdio.h>
#include <unistd.h>
#include <stdarg.h>
#include <sys/wait.h>
#include <omp.h>

static omp_lock_t _log_omp_lock;
static volatile int _log_omp_lock_initialized = 0;

static void _ensure_log_omp_lock_initialized()
{
        if( _log_omp_lock_initialized ) return;

#pragma omp critical (_log_lock_init_critical_name)
    {
        if( ! _log_omp_lock_initialized ) {
            omp_init_lock(&_log_omp_lock);
            _log_omp_lock_initialized = 1;
        }
    }
}

static void _omplock()
{
    _ensure_log_omp_lock_initialized();
    omp_set_lock(&_log_omp_lock);
}

static void _ompunlock()
{
    omp_unset_lock(&_log_omp_lock);
}

static int _log(const char *format, ...)
{
        int err;
        va_list ap;

        va_start (ap, format);

        _ensure_log_omp_lock_initialized();
        _omplock();

        err = vfprintf (stderr, format, ap);

        _ompunlock();

        va_end(ap);

        return err;
}

int main (void)
{
        int pid, status;

        _log ("test\n");

        pid = fork();
        if (!pid)
        {
                _log ("test child\n");
                exit (0);
        }

        waitpid (pid, &status, 0);
        if (status != 0)
                _log ("child returned err=%d\n", status);
        exit (0);
}

It works on 11.1.056:

$ icc -V
Intel(R) C Intel(R) 64 Compiler Professional for applications running on Intel(R) 64, Version 11.1    Build 20090827 Package ID: l_cproc_p_11.1.056
Copyright (C) 1985-2009 Intel Corporation.  All rights reserved.

$ icc -Wall -openmp -g a.c -o a
tgcn5x64:/tmp$ ./a
test
test child

But don't work on 14.0.3

$ icc -V

Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.3.174 Build 20140422
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

$ icc -Wall -openmp -g a.c -o a

$ ./a
teste
child returned err=139

Using GDB I get

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2aaaaade4800 (LWP 5984)]
0x00002aaaaab26bc9 in __kmp_lookup_user_lock (user_lock=0x601408, func=0x0) at ../../src/kmp_lock.cpp:3470
3470    ../../src/kmp_lock.cpp: No such file or directory.
        in ../../src/kmp_lock.cpp
(gdb) bt
#0  0x00002aaaaab26bc9 in __kmp_lookup_user_lock (user_lock=0x601408, func=0x0) at ../../src/kmp_lock.cpp:3470
#1  0x00002aaaaab0f352 in __kmpc_set_lock (loc=0x601408, gtid=0, user_lock=0x601408) at ../../src/kmp_csupport.c:1486
#2  0x0000000000400995 in _omplock () at a.c:26
#3  0x0000000000400a59 in _log (format=0x400c60 "test child\n") at a.c:42
#4  0x0000000000400b00 in main () at a.c:62

What am I doing wrong?

[Vladimir_P_1234567890]

Hello,

We don't support inter-process synchronization, so the parent and child will never be synchronized using same lock variable. you need to call omp_init_lock() for the child process.

there was a bug in lock tables so child locks worked. Now lock tables are being cleaned up during forking and the code above stopped working.

So if you put omp_init_lock(&_log_omp_lock); before _log ("test child\n"); call it starts to work.

--Vladimir 

[Luiz_Felipe_S_]

Even doing that change, the program still segfaults.... It works only if I move the log entry to a place after the fork(). This behavior leads to another question, Is it safe to use OpenMP code before a fork() call?

#include <stdio.h>
#include <unistd.h>
#include <stdarg.h>
#include <sys/wait.h>
#include <omp.h>

static omp_lock_t _log_omp_lock;
static volatile int _log_omp_lock_initialized = 0;

static void _ensure_log_omp_lock_initialized()
{
        if( _log_omp_lock_initialized ) return;

#pragma omp critical (_log_lock_init_critical_name)
    {
        if( ! _log_omp_lock_initialized ) {
            fprintf (stderr, "Init lock!\n");
            omp_init_lock(&_log_omp_lock);
            _log_omp_lock_initialized = 1;
        }
    }
}

static void _omplock()
{
    _ensure_log_omp_lock_initialized();
    omp_set_lock(&_log_omp_lock);
}

static void _ompunlock()
{
    omp_unset_lock(&_log_omp_lock);
}

static int _log(const char *format, ...)
{
        int err;
        va_list ap;

        va_start (ap, format);

        _omplock();

        err = vfprintf (stderr, format, ap);

        _ompunlock();

        va_end(ap);

        return err;
}

int main (void)
{
        int pid, status;

        _log ("test start\n");

        pid = fork();
        if (!pid)
        {

                // _log_omp_lock_initialized = 0;
                omp_init_lock(&_log_omp_lock);

                _log ("test child\n");

                exit (0);
        }


        waitpid (pid, &status, 0);
        if (status != 0)
                _log ("child returned err=%d\n", status);
        else
                _log ("test end\n");
        exit (0);

}

[Feilong_H_Intel]

Hi Luiz,

I'm entering this issue to our problem-tracking database, asking an OpenMP runtime library expert to take a look.  Will let you know once I hear from him.

Thanks.

[Feilong_H_Intel · Accepted solution]

Engineering team said that this (using OpenMP code in a forked process) is a undefined behavior which is not guaranteed by any standard, although it happened to work in 11.1 compiler.  The OpenMP standard says nothing about the behavior of OpenMP code in a forked process (the only mention of "fork" is to say "OpenMP uses a fork-join model"; the fork system call is not mentioned at all).  So, the only advice I can give is not to write code in this way.

Thanks.

[Luiz_Felipe_S_]

Thanks for the help!

So, newer ICC has the same behavior of GCC. It was noted before that ICC behaved different, allowing this openmp-fork thing as this URL observes http://bisqwit.iki.fi/story/howto/openmp/#OpenmpAndFork

[jimdempseyatthecove]

From http://man7.org/linux/man-pages/man2/fork.2.html

Note the following further points:

       *  The child process is created with a single thread—the one that
          called fork().  The entire virtual address space of the parent is
          replicated in the child, including the states of mutexes,
          condition variables, and other pthreads objects; the use of
          pthread_atfork(3) may be helpful for dealing with problems that
          this can cause.

Therefore, if a child process receive a copy of the application with a locked omp_lock, it cannot unlock the lock.

Also, the OpenMP state tables would be copied with threads created in the parent thread pool, the child process would not have these threads established. OpenMP would come tumbling down.

Jim Dempsey