Segmentation fault when passing internal subroutine as an argument, compiler version 19.1.1.217

[vganine]

Dear Intel Community Members

After switching to Intel compiler version 19.1.1.217 with Intel MPI version 2019.7.217 our code breaks down with "Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7fffffff3cd0".

The issue arises when calling internal procedures passed as arguments. The following example seems to well replicate the problem:

module lib_mod
implicit none
public
interface library_subroutine
module procedure lsubr
end interface library_subroutine
contains

subroutine lsubr(dummysubr)
implicit none
interface
subroutine dummysubr(i1,i2)
implicit none
integer, intent(in) :: i1,i2
end subroutine dummysubr
end interface
integer :: i1, i2
i1 = 1
i2 = 10
call dummysubr(i1,i2)
return
end subroutine lsubr
end module lib_mod

program test
use lib_mod
implicit none

real :: vector(10)
integer :: ierr
call mpi_init(ierr)
vector = 0.0d0
call library_subroutine(internal_subroutine)
call mpi_finalize(ierr)
contains
subroutine internal_subroutine(i1,i2)
implicit none
integer, intent(in) :: i1,i2
integer :: i
do i = i1,i2
vector(i) = 1.0d0
end do
return
end subroutine internal_subroutine
end program test

The code is built using Fortran wrapper compiler :  mpif90 -g -O0 test.f90

Below is the command line exectuted:

ifort -g '-O0' 'test.f90' -I/sw/installed/impi/2018.5.288-iccifort-2019.5.281/intel64/include/gfortran -I/sw/installed/impi/2018.5.288-iccifort-2019.5.281/intel64/include -L/sw/installed/impi/2018.5.288-iccifort-2019.5.281/intel64/lib/debug_mt -L/sw/installed/impi/2018.5.288-iccifort-2019.5.281/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /sw/installed/impi/2018.5.288-iccifort-2019.5.281/intel64/lib/debug_mt -Xlinker -rpath -Xlinker /sw/installed/impi/2018.5.288-iccifort-2019.5.281/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lrt -lpthread

The system specs and compiler versions are as follows:

ifort version 19.1.1.217

gcc version 9.3.0

Intel MPI version 2019.7.217

Linux 3.10.0-1127.19.1.el7.x86_64 x86_64

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"

Intel Haswell microarchitecture, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

I have tried to build the same example without MPI, and it worked just fine.

Could you please suggest any workarounds to run it with Intel MPI?

Many thanks

Vlad Ganine

[Steve_Lionel]

I don't know what Intel Fortran does "under the hood" to enable passing internal procedures as actual arguments. In the distant past, this involved using a chunk of stack to write instructions that established the context and then passing the stack address. Nowadays, where stacks are made non-executable, a different mechanism is used. I discuss this in Doctor Fortran in "Think, Thank, Thunk" - Doctor Fortran (stevelionel.com)

It could be that the MPI initialization somehow breaks this mechanism. In any case, I think you're on very thin ice trying to do uplevel references in a routine called from outside the current scope while in the context of MPI. I don't know why it doesn't work, but it seems unnecessarily complex to me. Being able to pass an internal procedure at all was new in F2008.

[Ron_Green]

the wrapper mpif90 will by default invoke GFORTRAN. I notice your details shows a mix of GFORTRAN libraries and IFORT. This could happen if you have env var FC or F90 or similar set to IFORT. Trouble here is the MPI libraries built for gfortran are not compatible with IFORT.

You want to use Intel MPI wrapper 'mpiifort'. I can show you. I added a print*, "done" at the end of the main program.

This works

 module list

Currently Loaded Modulefiles:

 1) comp/19.1.1(default)  

rwgreen@orcsle147: mpiifort -g -O0 intproc.f90 -o intproc

rwgreen@orcsle147:~/quad/u1254993$ mpirun -n 2 ./intproc

 done

 done

rwgreen@orcsle147:~/quad/u1254993$ 

now if I do what you do with wrong wrapper:

rwgreen@orcsle147:~/quad/u1254993$ mpif90 -g -O0 -o intproc intproc.f90

rwgreen@orcsle147:~/quad/u1254993$ mpirun -n 2 ./intproc

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

#0 0x7F13760D3467

#1 0x7F13760D3AAE

#2 0x7F13755DA64F

#3 0x7FFF20BD9E08

#0 0x7FBF2BEA6467

#1 0x7FBF2BEA6AAE

#2 0x7FBF2B3AD64F

#3 0x7FFC9391B198

===================================================================================

=  BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=  RANK 0 PID 54152 RUNNING AT orcsle147

=  KILLED BY SIGNAL: 11 (Segmentation fault)

===================================================================================

===================================================================================

=  BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=  RANK 1 PID 54153 RUNNING AT orcsle147

=  KILLED BY SIGNAL: 9 (Killed)

===================================================================================

rwgreen@orcsle147:~/quad/u1254993$ 

So solution is to use the mpiifort wrapper from Intel MPI. You cannot use a MPI built for gfortran, it will cause runtime errors like you and I have seen.