topic Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1 in Intel® Fortran Compiler

ifort internal compiler error starting with 2022.3.0 and 2022.3.1

rem0 — Thu, 17 Nov 2022 16:17:46 GMT

Started to get this error after moving from 2022.2 to 2022.3:

/tmp/iforteD1rI4.i90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

compilation aborted for test_mpp_alltoall.F90 (code 1)

Here's the file it was compiling at the time: https://raw.githubusercontent.com/NOAA-GFDL/FMS/main/test_fms/mpp/test_mpp_alltoall.F90

It's mainly testing a wrapper for mpi_alltoall, error doesn't give much information so not sure what could be causing it. I was running this in a container (intel/oneapi-hpckit:latest, using shm for I_MPI_FABRICS).

Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1

Barbara_P_Intel — Thu, 17 Nov 2022 19:27:21 GMT

Can you please provide the module files? Looks like test_mpp_alltoall.F90 needs

use platform_mod
use mpp_mod

Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1

jimdempseyatthecove — Thu, 17 Nov 2022 19:29:49 GMT

When pasting source code and/or console window text captures, please use the formatted paste capability for this forum.

This is not intuitive.

Click on the toolbar ... icon, this opens up additional tools.

Click on the </> icon, this opens an additional edit box with a pull-down control.

Click on the Markup pull down, to select the language or "Plane Text". In this case I choose Plan Text:

mpiifort -DPACKAGE_NAME=\"GFDL\ FMS\ Library\" -DPACKAGE_TARNAME=\"FMS\" -DPACKAGE_VERSION=\"2022.04.00-dev\" -DPACKAGE_STRING=\"GFDL\ FMS\ Library\ 2022.04.00-dev\" -DPACKAGE_BUGREPORT=\"gfdl.climate.model.info@noaa.gov\" -DPACKAGE_URL=\"https://www.github.com/NOAA-GFDL/FMS\" -DPACKAGE=\"FMS\" -DVERSION=\"2022.04.00-dev\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_MPI_H=1 -DHAVE_YAML_H=1 -Duse_yaml=1 -DHAVE_NETCDF_H=1 -DHAVE_GETTID=1 -DHAVE_SCHED_GETAFFINITY=1 -DHAVE_MOD_MPI=1 -DHAVE_MPIF_H=1 -DHAVE_MOD_NETCDF=1 -DHAVE_CRAY_POINTER=1 -DHAVE_QUAD_PRECISION=1 -DHAVE_INTERNAL_NML=1 -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../../include -I../../.mods -fopenmp -I/libs/include -g -traceback -FR -real-size 64 -fopenmp -c -o test_mpp_alltoall.o test_mpp_alltoall.F90 /tmp/iforteD1rI4.i90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error. compilation aborted for test_mpp_alltoall.F90 (code 1)

This makes the pasted text more readable.

Segmentation fault during compilation requires a reproducer (which you provided), but also requires full information as to the build environment. In particular: Microsoft Visual Studio version.

Hopefully, Intel will reproduce the error using the Intel compiler version and supplied .f90 file.

Jim Dempsey

Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1

Ron_Green — Thu, 17 Nov 2022 23:49:55 GMT

I pulled FMS from Github, followed the instructions and did a CMake build. It succeeded. Now I did not set ICC or IFORT flags, just used defaults. NetCDF 4.9.0

I did this:

export FC=mpiifort

export CC=mpiicc

export CXX=mpiicpc

export NetCDF=/cts/tools/library/netcdf4-intel/4.9.0

cmake -DCMAKE_INSTALL_PREFIX=/home/foo/projects/fms-install

mkdir build ; cd build

cmake -DCMAKE_INSTALL_PREFIX=/nfs/pdx/home/rwgreen/projects/fms-install ..

make -j 8

Did you set compiler flags and various FMS defines for CMake?

...

[ 98%] Built target fms_r4_f

[100%] Linking Fortran static library libfms_r4.a

[100%] Built target fms_r4

Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1

rem0 — Wed, 04 Jan 2023 20:31:55 GMT

@jimdempseyatthecove @Ron_Green Thank you for looking into this! Sorry for the long delay.

FMS mainly uses GNU autotools as its build system, the cmake build is used by external projects and does not include compilation of the test directory.

To give a bit more background, this error first occurred in a CI environment when I tried to bump our container up to the latest version. Then I ran it myself with docker and got the same result. Haven't had a chance to run with a normal install yet but will update when possible. This was using the Intel provided image on dockerhub (intel/oneapi-hpckit:latest).

Using mpirun in the container seems to necessitate that I_MPI_FABRICS is set to "shm", but I don't know if that affects compilation at all. I'll include the steps to reproduce assuming FMS has been cloned:

export CC=mpiicc FC=mpiifort export FCFLAGS="-I<netcdf fortran include>" export CFLAGS="-I<netcdf c include>" mkdir build; cd build autoreconf -if ../configure.ac ../configure make make check -C test_fms/mpp

Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1

rem0 — Thu, 26 Jan 2023 16:08:28 GMT

Looks like this isn't container-specific, also saw it on our Centos8 machine.

Got this error with oneapi 2023.01, ifort v2021.8.0

mpiifort -DPACKAGE_NAME=\"GFDL\ FMS\ Library\" -DPACKAGE_TARNAME=\"FMS\" -DPACKAGE_VERSION=\"2022.04.00-dev\" -DPACKAGE_STRING=\"GFDL\ FMS\ Library\ 2022.04.00-dev\" -DPACKAGE_BUGREPORT=\"gfdl.climate.model.info@noaa.gov\" -DPACKAGE_URL=\"https://www.github.com/NOAA-GFDL/FMS\" -DPACKAGE=\"FMS\" -DVERSION=\"2022.04.00-dev\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_MPI_H=1 -DHAVE_NETCDF_H=1 -DHAVE_SCHED_GETAFFINITY=1 -DHAVE_MOD_MPI=1 -DHAVE_MPIF_H=1 -DHAVE_MOD_NETCDF=1 -DHAVE_CRAY_POINTER=1 -DHAVE_QUAD_PRECISION=1 -DHAVE_INTERNAL_NML=1 -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../../../FMS/test_fms/mpp -I../../../FMS/include -I../../.mods -fopenmp -I/opt/netcdf/4.9.0/ONEAPI/2023.0/include -I/opt/netcdf/4.9.0/ONEAPI/2023.0/include -g -O0 -traceback -FR -real-size 64 -fopenmp -c -o test_mpp_alltoall.o ../../../FMS/test_fms/mpp/test_mpp_alltoall.F90 /tmp/ifortY6cf8h.i90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error. compilation aborted for ../../../FMS/test_fms/mpp/test_mpp_alltoall.F90 (code 1) make[1]: *** [Makefile:1216: test_mpp_alltoall.o] Error 1

Re: ifort internal compiler error starting with 2022.3.0 and 2022.3.1

Ron_Green — Fri, 27 Jan 2023 21:46:52 GMT

Fortunately I was able to spend a few days on this, maybe 8hrs total, to strip out all the MPI, mpiifort, netCDF fluff and get to the core of the issue. IFORT is getting an ICE on test_mpp_alltoall.F90.

Today I finally got this cut down to a very compact example. the error was in mpp.F90, well use of mpp_type and mpp_byte as shown.

module mpp_mod type :: mpp_type private integer, allocatable :: sizes(:) end type mpp_type type(mpp_type), target :: mpp_byte end module mpp_mod program test_mpp_alltoall use mpp_mod, only : mpp_type, mpp_byte !>>..surprise, we need the "only" clause for ICE implicit none integer :: npes contains subroutine test_mpp_alltoallw_real8(npes) implicit none integer, intent(in) :: npes integer :: ssize(0:npes-1) type(mpp_type) :: stype(0:npes-1) stype(:) = mpp_byte end subroutine test_mpp_alltoallw_real8 subroutine test_mpp_alltoallw_int4(npes) implicit none integer, intent(in) :: npes type(mpp_type) :: stype(0:npes-1) stype(:) = mpp_byte end subroutine test_mpp_alltoallw_int4 end program test_mpp_alltoall

and this ICE is only triggered with ifort -qopenmp (you knew that?).

IFX is not getting the ICE

$ ifort -O0 -c -what -V repro.f90 -qopenmp Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.8.0 Build 20221119_000000 Copyright (C) 1985-2022 Intel Corporation. All rights reserved. Intel(R) Fortran 2021.8.0-1196 repro.f90: catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error. compilation aborted for repro.f90 (code 1) $ ifx -O0 -c -what -V repro.f90 -qopenmp Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2023.0.0 Build 20221201 Copyright (C) 1985-2022 Intel Corporation. All rights reserved. Intel(R) Fortran 23.0-1198

again, if you remove -qopenmp the error goes away.

You can switch over to IFX from IFORT in mpiifort very easily. Change

mpiifort

mpiifort fc=ifx

That hopefully will allow you to build test_mpp

I'm trying to figure out why IFX is working whereas ifort is not. I will open up a bug report on iFORT once I test out a few more variables.

Re:ifort internal compiler error starting with 2022.3.0 and 2022.3.1

Ron_Green — Wed, 01 Feb 2023 23:19:29 GMT

I forgot to post the bug ID on this: it's CMPLRLLVM-44390

Re:ifort internal compiler error starting with 2022.3.0 and 2022.3.1

Ron_Green — Fri, 21 Jul 2023 23:07:24 GMT

This bug is fixed in 2023.2.0