ifort internal compiler error starting with 2022.3.0 and 2022.3.1

[rem0]

Started to get this error after moving from 2022.2 to 2022.3:

mpiifort -DPACKAGE_NAME=\"GFDL\ FMS\ Library\" -DPACKAGE_TARNAME=\"FMS\" -DPACKAGE_VERSION=\"2022.04.00-dev\" -DPACKAGE_STRING=\"GFDL\ FMS\ Library\ 2022.04.00-dev\" -DPACKAGE_BUGREPORT=\"gfdl.climate.model.info@noaa.gov\" -DPACKAGE_URL=\"https://www.github.com/NOAA-GFDL/FMS\" -DPACKAGE=\"FMS\" -DVERSION=\"2022.04.00-dev\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_MPI_H=1 -DHAVE_YAML_H=1 -Duse_yaml=1 -DHAVE_NETCDF_H=1 -DHAVE_GETTID=1 -DHAVE_SCHED_GETAFFINITY=1 -DHAVE_MOD_MPI=1 -DHAVE_MPIF_H=1 -DHAVE_MOD_NETCDF=1 -DHAVE_CRAY_POINTER=1 -DHAVE_QUAD_PRECISION=1 -DHAVE_INTERNAL_NML=1 -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../../include -I../../.mods -fopenmp -I/libs/include -g -traceback -FR -real-size 64 -fopenmp -c -o test_mpp_alltoall.o test_mpp_alltoall.F90

/tmp/iforteD1rI4.i90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

compilation aborted for test_mpp_alltoall.F90 (code 1)

Here's the file it was compiling at the time: https://raw.githubusercontent.com/NOAA-GFDL/FMS/main/test_fms/mpp/test_mpp_alltoall.F90

It's mainly testing a wrapper for mpi_alltoall, error doesn't give much information so not sure what could be causing it. I was running this in a container (intel/oneapi-hpckit:latest, using shm for I_MPI_FABRICS).

[Barbara_P_Intel]

Can you please provide the module files? Looks like test_mpp_alltoall.F90 needs

use platform_mod
use mpp_mod

[jimdempseyatthecove]

When pasting source code and/or console window text captures, please use the formatted paste capability for this forum.

This is not intuitive.

Click on the toolbar ... icon, this opens up additional tools.

Click on the </> icon, this opens an additional edit box with a pull-down control.

Click on the Markup pull down, to select the language or "Plane Text". In this case I choose Plan Text:

mpiifort -DPACKAGE_NAME=\"GFDL\ FMS\ Library\" -DPACKAGE_TARNAME=\"FMS\" -DPACKAGE_VERSION=\"2022.04.00-dev\" -DPACKAGE_STRING=\"GFDL\ FMS\ Library\ 2022.04.00-dev\" -DPACKAGE_BUGREPORT=\"gfdl.climate.model.info@noaa.gov\" -DPACKAGE_URL=\"https://www.github.com/NOAA-GFDL/FMS\" -DPACKAGE=\"FMS\" -DVERSION=\"2022.04.00-dev\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_MPI_H=1 -DHAVE_YAML_H=1 -Duse_yaml=1 -DHAVE_NETCDF_H=1 -DHAVE_GETTID=1 -DHAVE_SCHED_GETAFFINITY=1 -DHAVE_MOD_MPI=1 -DHAVE_MPIF_H=1 -DHAVE_MOD_NETCDF=1 -DHAVE_CRAY_POINTER=1 -DHAVE_QUAD_PRECISION=1 -DHAVE_INTERNAL_NML=1 -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../../include -I../../.mods -fopenmp -I/libs/include -g -traceback -FR -real-size 64 -fopenmp -c -o test_mpp_alltoall.o test_mpp_alltoall.F90

 
/tmp/iforteD1rI4.i90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
 
compilation aborted for test_mpp_alltoall.F90 (code 1)

This makes the pasted text more readable.

Segmentation fault during compilation requires a reproducer (which you provided), but also requires full information as to the build environment. In particular: Microsoft Visual Studio version.

Hopefully, Intel will reproduce the error using the Intel compiler version and supplied .f90 file.

Jim Dempsey

[Ron_Green]

I pulled FMS from Github, followed the instructions and did a CMake build.  It succeeded.  Now I did not set ICC or IFORT flags, just used defaults.  NetCDF 4.9.0

I did this:

export FC=mpiifort

export CC=mpiicc

export CXX=mpiicpc

export NetCDF=/cts/tools/library/netcdf4-intel/4.9.0

cmake -DCMAKE_INSTALL_PREFIX=/home/foo/projects/fms-install

mkdir build ; cd build

cmake -DCMAKE_INSTALL_PREFIX=/nfs/pdx/home/rwgreen/projects/fms-install ..

make -j 8 

Did you set compiler flags and various FMS defines for CMake?

...

[ 98%] Built target fms_r4_f

[100%] Linking Fortran static library libfms_r4.a

[100%] Built target fms_r4

[rem0]

@jimdempseyatthecove  @Ron_Green Thank you for looking into this! Sorry for the long delay.

FMS mainly uses GNU autotools as its build system, the cmake build is used by external projects and does not include compilation of the test directory.

To give a bit more background, this error first occurred in a CI environment when I tried to bump our container up to the latest version. Then I ran it myself with docker and got the same result. Haven't had a chance to run with a normal install yet but will update when possible. This was using the Intel provided image on dockerhub (intel/oneapi-hpckit:latest).

Using mpirun in the container seems to necessitate that I_MPI_FABRICS is set to "shm", but I don't know if that affects compilation at all. I'll include the steps to reproduce assuming FMS has been cloned:

export CC=mpiicc FC=mpiifort
export FCFLAGS="-I<netcdf fortran include>"
export CFLAGS="-I<netcdf c include>"
mkdir build; cd build
autoreconf -if ../configure.ac
../configure
make
make check -C test_fms/mpp

[rem0]

Looks like this isn't container-specific, also saw it on our Centos8 machine.

Got this error with oneapi 2023.01,  ifort v2021.8.0

mpiifort -DPACKAGE_NAME=\"GFDL\ FMS\ Library\" -DPACKAGE_TARNAME=\"FMS\" -DPACKAGE_VERSION=\"2022.04.00-dev\" -DPACKAGE_STRING=\"GFDL\ FMS\ Library\ 2022.04.00-dev\" -DPACKAGE_BUGREPORT=\"gfdl.climate.model.info@noaa.gov\" -DPACKAGE_URL=\"https://www.github.com/NOAA-GFDL/FMS\" -DPACKAGE=\"FMS\" -DVERSION=\"2022.04.00-dev\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_MPI_H=1 -DHAVE_NETCDF_H=1 -DHAVE_SCHED_GETAFFINITY=1 -DHAVE_MOD_MPI=1 -DHAVE_MPIF_H=1 -DHAVE_MOD_NETCDF=1 -DHAVE_CRAY_POINTER=1 -DHAVE_QUAD_PRECISION=1 -DHAVE_INTERNAL_NML=1 -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../../../FMS/test_fms/mpp  -I../../../FMS/include -I../../.mods -fopenmp  -I/opt/netcdf/4.9.0/ONEAPI/2023.0/include -I/opt/netcdf/4.9.0/ONEAPI/2023.0/include -g -O0 -traceback  -FR -real-size 64 -fopenmp -c -o test_mpp_alltoall.o ../../../FMS/test_fms/mpp/test_mpp_alltoall.F90
/tmp/ifortY6cf8h.i90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for ../../../FMS/test_fms/mpp/test_mpp_alltoall.F90 (code 1)
make[1]: *** [Makefile:1216: test_mpp_alltoall.o] Error 1

[Ron_Green]

Fortunately I was able to spend a few days on this, maybe 8hrs total, to strip out all the MPI, mpiifort, netCDF fluff and get to the core of the issue.  IFORT is getting an ICE on test_mpp_alltoall.F90.

Today I finally got this cut down to a very compact example.  the error was in mpp.F90, well use of mpp_type and mpp_byte as shown.

module mpp_mod
  type :: mpp_type
   private
     integer, allocatable :: sizes(:)
  end type mpp_type

  type(mpp_type), target :: mpp_byte

end module mpp_mod



program test_mpp_alltoall
  use mpp_mod, only :  mpp_type, mpp_byte !>>..surprise, we need the "only" clause for ICE
  implicit none

  integer :: npes

  contains

        
  subroutine test_mpp_alltoallw_real8(npes)
    implicit none
    integer, intent(in) :: npes

    integer :: ssize(0:npes-1)
    type(mpp_type) :: stype(0:npes-1)

    stype(:) = mpp_byte
  end subroutine test_mpp_alltoallw_real8



  subroutine test_mpp_alltoallw_int4(npes)
    implicit none
    integer, intent(in) :: npes
    type(mpp_type) :: stype(0:npes-1)

    stype(:) = mpp_byte
  end subroutine test_mpp_alltoallw_int4


end program test_mpp_alltoall

and this ICE is only triggered with  ifort -qopenmp (you knew that?).

IFX is not getting the ICE

$ ifort -O0 -c -what -V repro.f90 -qopenmp
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.8.0 Build 20221119_000000
Copyright (C) 1985-2022 Intel Corporation.  All rights reserved.

 Intel(R) Fortran 2021.8.0-1196
repro.f90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for repro.f90 (code 1)


$ ifx -O0 -c -what -V repro.f90 -qopenmp
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2023.0.0 Build 20221201
Copyright (C) 1985-2022 Intel Corporation. All rights reserved.

 Intel(R) Fortran 23.0-1198

again, if you remove -qopenmp the error goes away.

You can switch over to IFX from IFORT in mpiifort very easily.  Change

  mpiifort

to

  mpiifort fc=ifx

That hopefully will allow you to build test_mpp

I'm trying to figure out why IFX is working whereas ifort is not.  I will open up a bug report on iFORT once I test out a few more variables.

[Ron_Green]

I forgot to post the bug ID on this: it's CMPLRLLVM-44390

[Ron_Green]

This bug is fixed in 2023.2.0