C/Fortran Multithreading Seg Faults

[phil_losie]

Hi all,

I have a main program, written in C, that utilizes a threadpool of 12 threads (using pthreads) to perform parallel processing of some science data. Our group has a legacy Fortran subroutine that we need to call from each of thread to do the actual processing on a subset of the data. I posted previously on guidelines for thread-safe Fortran code (see http://software.intel.com/en-us/forums/showthread.php?t=79889) and incorporated those guidelines into my code. However, I'm stilling getting SIGSEGV errors in the Fortran section of my code. I've traced the problem to file I/O.

Here is the main C program code:

[bash]/*
 * Multithreaded C-Fortran mixed language test program
 */

#include 
#include 
#include 
#include 
#include 
#include "threadpool.h"

#define NUM_THREADS 12
#define NUM_JOBS 10

extern void fpa_(char *filename, int *io);
void job_thread(void *arg1, void *arg2);

int main(int argc, char **argv)
{
    threadpool tp;
    int i;
    char filenames[NUM_JOBS][100];
    int io[NUM_JOBS];

    /* Create a thread pool of 12 threads */
    tp = create_threadpool(NUM_THREADS);

    for (i = 0; i < NUM_JOBS; i++)
    {
        /* Create a unique file name i.e. "./Files/file1.txt" */
        sprintf(&filenames[0], "/home/1060447/Test/Files/file%d.txt", i+1);

        /* Create a unique file I/O unit */
        io = 10+i;

        /* Add file name and I/O unit to threadpool job queue */
        printf("Main: queueing %s\n", &filenames[0]);
        dispatch_job(tp, job_thread, (void *) &filenames[0], &io);
    }

    /* Wait a while for all the threads to do their thing */
    printf("Main: sleeping 20 secs...\n");
    sleep(20);
    printf("Main: waking up and destroying threadpool...\n");

    /* Clean up threadpool */
    destroy_threadpool(tp);

    printf("Main: exiting...\n");
    return 0;
}

/* Thread job function */
void job_thread(void *arg1, void *arg2)
{
    char *filename = (char *) arg1;
    int *io = (int *) arg2;
    unsigned int len = strlen(filename);

    char string[len+1];
    sprintf(string, "%s", filename);

    fpa_(string, io);
    printf("->Thread %ld: Opened %s (len=%d) on I/O unit %d\n", pthread_self(), filename, len, *io);
}
[/bash]

Fortran subroutine code:

[bash]SUBROUTINE fpa(filename, io)

integer                  :: io
character*(*)            :: filename

open(io,file=filename,form='formatted',status='old')
close(io)

END SUBROUTINE fpa[/bash]

It creates a threadpool of 12 threads, each thread calls the job_thread function which in turn executes the Fortran fpa_ subroutine. The fpa_ subroutine simply opens and closes a unique file on a unique unit number. This program runs fine if I limit the threadpool to just 1 thread.

I compile the Fortran code using ifort (version 11.1) and options -openmp -pthread. The C code is compiled using gcc (version 4.1.2 in RedHat). The entire project is linked using gcc with the following options: -lpthread -mt -lm -L/opt/intel/Compiler/11.1/059/lib/intel64

Here is the program output (run from gdb debugger):

[plain](gdb) run
Starting program: /home/1060447/Test/test
[Thread debugging using libthread_db enabled]
[New Thread 0x40a00940 (LWP 5703)]
[New Thread 0x41401940 (LWP 5704)]
[New Thread 0x41e02940 (LWP 5705)]
[New Thread 0x42803940 (LWP 5706)]
[New Thread 0x43204940 (LWP 5707)]
[New Thread 0x43c05940 (LWP 5708)]
[New Thread 0x44606940 (LWP 5709)]
[New Thread 0x45007940 (LWP 5710)]
[New Thread 0x45a08940 (LWP 5711)]
[New Thread 0x46409940 (LWP 5712)]
[New Thread 0x46e0a940 (LWP 5713)]
[New Thread 0x4780b940 (LWP 5714)]
Main: queueing /home/1060447/Test/Files/file1.txt
Main: queueing /home/1060447/Test/Files/file2.txt
Main: queueing /home/1060447/Test/Files/file3.txt
Main: queueing /home/1060447/Test/Files/file4.txt
Main: queueing /home/1060447/Test/Files/file5.txt
Main: queueing /home/1060447/Test/Files/file6.txt
Main: queueing /home/1060447/Test/Files/file7.txt
Main: queueing /home/1060447/Test/Files/file8.txt
Main: queueing /home/1060447/Test/Files/file9.txt
Main: queueing /home/1060447/Test/Files/file10.txt
Main: sleeping 20 secs...
->Thread 1178638656: Opened /home/1060447/Test/Files/file2.txt (len=34) on I/O unit 11
->Thread 1157658944: Opened /home/1060447/Test/Files/file4.txt (len=34) on I/O unit 13
->Thread 1126189376: Opened /home/1060447/Test/Files/file5.txt (len=34) on I/O unit 14
->Thread 1094719808: Opened /home/1060447/Test/Files/file1.txt (len=34) on I/O unit 10
->Thread 1136679232: Opened /home/1060447/Test/Files/file6.txt (len=34) on I/O unit 15
->Thread 1147169088: Opened /home/1060447/Test/Files/file7.txt (len=34) on I/O unit 16

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x45a08940 (LWP 5711)]
0x000000335480a020 in pthread_mutex_unlock () from /lib64/libpthread.so.0[/plain]

A backtrace from the debugger:

[plain](gdb) bt
#0  0x000000335480a020 in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1  0x00002aaaaab3861f in for__release_lun () from /opt/intel/Compiler/11.1/059/lib/intel64/libifcoremt.so.5
#2  0x00002aaaaab3ba58 in for_open () from /opt/intel/Compiler/11.1/059/lib/intel64/libifcoremt.so.5
#3  0x0000000000401d45 in fpa (filename=Quit[/plain]

I have converted the Fortran subroutine to C and the main program runs fine, regardless of the number of threads used. It seems that there is a mixed language issue between C and Fortran, or an incompatibility between the versions of ifort and gcc that I'm using. This is really frustrating!

Any help would be greatly appreciated!

Thanks,
Phil Losie

[Ron_Green]

where to begin ...

First, you cannot mix threading runtimes. So if you are using pthreads in C to create and manage the threadpool, DON'T compile the Fortran -openmp -pthreads

BUT you will need to use ifort -c -nofor-main -auto to tell the Fortran compiler that the main runtime is not Fortran AND -auto says to use stack for local variables (each thread will have a local copy)

Next, yes, mixed language issues. A C string is null terminated, a Fortran character variable has a length attribute and the characters. I don't have your create_threadpool so I can test everything, but I think the Fortran ought to look like this:

SUBROUTINE fpa(filename, io) bind(C, name="fpa")
use iso_c_binding
integer (kind=c_int), value, intent(in) :: io
type (C_PTR), intent(in), value :: filename

character (kind=C_CHAR), POINTER :: fort_filename(:)

call C_F_POINTER( filename, fort_filename )

open(io,file=fort_filename(1:),form='formatted',status='old')
close(io)

END SUBROUTINE fpa


and on the C side, call "fpa" and NOT "fpa_" - relying on name decoration is a bad practice.

We may need to tweak the fortran a bit, like I said, I haven't tested this.

[TimP]

You could use openmp threading in gcc compatibly with ifort, if you would use a recent enough gcc to support it. You would require the libiomp5 which comes with ifort, as that supports all the libgomp function calls as well as the Intel OpenMP calls.

[phil_losie]

Ronald,

Thansk for your quick reply! Attached is all my source code. I incorporated your suggested changes and I'm still getting a seg fault.

-Phil

[Ron_Green]

Hi Phil,

I've uploaded a working copy. You know, there are a TON of examples of passing Fortran strings to C, but going the opposite way there is very little. Here are some of the keys:

On the C side, I arbitrarily swapped the order of arguments to pass IO before the FILENAME. You can change those back, I debugged passing the int first since it was simpler, then added the string argument. Also, note the extern declaration and how the args are passed:

extern void fpa(int io, char *filename);
...
fpa(*io, string);


Now, the fortran:

SUBROUTINE fpa(io, filename) bind(C, name="fpa")
use iso_c_binding
implicit none
integer (kind=c_int), value, intent(in) :: io
type(C_PTR), value :: filename

character*1 , pointer :: ffilename(:)
integer, parameter :: MAX_FILENAME_LENGTH=200
character (len=MAX_FILENAME_LENGTH) :: ofile = " "

!...interface to C strlen function ...
interface
function strlen(str) bind(C,name='strlen')
use ISO_C_BINDING
implicit none
type(C_PTR),value :: str
integer(C_SIZE_T) strlen
end function strlen
end interface

integer :: j,fnsize

!...find the input C string length
fnsize = strlen(filename)

!...create a Fortran point to an array of characters
call c_f_pointer( filename, ffilename, [fnsize] )

!...copy the individual chars to a character variable
do j=1,fnsize
ofile(j:j) = ffilename(j)
end do

open(io,file=trim(ofile),form='formatted',status='unknown')
close(io)

END SUBROUTINE fpa


The tricks were to determine the C string size calling the C lib strlen, then creating an equivalent Fortran pointer to the array of characters passed in by C, then copying those individual characters to a "normal" Fortran character variable.

In the attached code, you'll need to modify test.c to put your path into the filename prototype, as I left my path in there.

ron

[phil_losie]

Ron,

I'm still getting a seg fault with your updated code. Sometimes, it runs correctly. It runs correctly 100% of the time if I change the number of threads to 1.

[Ron_Green]

A colleague pointed out to me that when we have a C main calling IO routines in the Fortran runtime library, it is necessary to call for_rtl_init_ () prior to using the FRTL, then set reentrancy property, and finally at the end of the C main, call for_rtl_finish_ () to clean up.

I've attached a new sample. Note the changes to test.c. Also, the C code needed an include from the Fortran include dir for the symbolic name for reeentrancy setting. Check the makefile. Also, in the makefile I have a non-default path to my compiler so fix the FINC and LFLAGS to point to your compiler.

ron

[phil_losie]

Ron,

I tried this out and it works! Thanks so much for your help. I would have never figured this out on my own. By the way, how did your colleague know to call FRTL prior to invoking Fortran? Is there any documentation in case I run into other problems down the road?

Thanks again!

-Phil

[Steven_L_Intel1]

This is documented in the "Building Applications" manual on mixed-language programming. The for_rtl_init routine is documented in the A-Z Reference part of the Language Reference.