IFORT and IFX compilations on linux in debug mode stop with **Internal compiler error...

fridman__sergey · ‎08-13-2024

We get an "Internal compiler error" when trying to build our code on Linux with debug information enabled. The error occurs (at some point within the build) when compiling with either

ifort (IFORT) 2021.10.0 20230609

or

ifx (IFX) 2023.2.0 20230721

(The same code compiled at equivalent settings on Windows with IFORT or IFX does not trigger any errors)

For example, here are some snippets from typical linux build logs:

Compile using
ifort (IFORT) 2021.10.0 20230609

/usr/bin/cmake -E cmake_copy_f90_mod tikhonov_internal_staff.mod CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/tikhonov_internal_staff.mod.stamp Intel

[ 11%] Building Fortran object CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90.o

/opt/intel/oneapi/compiler/2023.2.1/linux/bin/intel64/ifort -I/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../ionomodels/PIM/PIM/source -I/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../WSBITikhFltr/GPS_Data_makeUp -I/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../tleTrajectoryPropagator/SGP4/for -I/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../utils/saoXml -m64 -fpp -warn noerrors,nointerfaces -O0 -g -debug full -traceback -check all,noarg_temp_created -assume realloc_lhs -ftrapuv -init=snan,arrays -c /var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90 -o CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90.o

/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90(674 warning #5462: Global name too long, shortened from: unknown_reference_points_mod_mp_ESTABLISH_KERNEL_AND_COV_HF_FOR_HIDDEN_PARAMETERS to: known_reference_points_mod_mp_ESTABLISH_KERNEL_AND_COV_HF_FOR_HIDDEN_PARAMETERS

subroutine establish_kernel_and_cov_HF_for_hidden_parameters(reference_points_set,RT_of_Data_summary)

-------------^

/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90(870 warning #5462: Global name too long, shortened from: unknown_reference_points_mod_mp_PRESERVE_URP_STATE_INTO_URP_LIST_BEFORE_TIKHUPDATE to: nown_reference_points_mod_mp_PRESERVE_URP_STATE_INTO_URP_LIST_BEFORE_TIKHUPDATE

subroutine preserve_URP_state_into_URP_List_before_TikhUpdate

---------------^

/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90(851 catastrophic error: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

compilation aborted for /var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90 (code 1)

make[2]: *** [CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/build.make:3628: CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90.o] Error 1

make[2]: Leaving directory '/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build_out'

make[1]: *** [CMakeFiles/Makefile2:86: CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/all] Error 2

make[1]: Leaving directory '/var/lib/buildbot/gpsii/gpsii_es_ifort_alma8_serial_debug/build_out'

make: *** [Makefile:94: all] Error 2

program finished with exit code 2

Compile using
ifx (IFX) 2023.2.0 20230721

/usr/bin/cmake -E cmake_copy_f90_mod magnetoionic_ray_tracing_module.mod CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/magnetoionic_ray_tracing_module.mod.stamp IntelLLVM

/usr/bin/cmake -E cmake_copy_f90_mod rt_output_vecor_components_assignment.mod CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/rt_output_vecor_components_assignment.mod.stamp IntelLLVM

[ 8%] Building Fortran object CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/combine10parQVIsSpr2hop/Hamiltonian_RayTracing_module.f90.o

/opt/intel/oneapi/compiler/2023.2.1/linux/bin/ifx -I/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../ionomodels/PIM/PIM/source -I/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../WSBITikhFltr/GPS_Data_makeUp -I/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../tleTrajectoryPropagator/SGP4/for -I/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../utils/saoXml -m64 -fpp -warn noerrors,nointerfaces -O0 -g -debug full -traceback -check all,noarg_temp_created -assume realloc_lhs -ftrapuv -init=snan,arrays -c /var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/combine10parQVIsSpr2hop/Hamiltonian_RayTracing_module.f90 -o CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/combine10parQVIsSpr2hop/Hamiltonian_RayTracing_module.f90.o

#0 0x00000000021c51e2

#1 0x0000000002228e97

#2 0x0000000002228fc0

#3 0x00007f04c984b5b0

#4 0x0000000002e01eb8

#5 0x00000000031eb4c3

#6 0x00000000031ebf18

#7 0x000000000311529d

#8 0x0000000002555100

#9 0x000000000216676b

#10 0x0000000002165047

#11 0x0000000002111a78

#12 0x00000000022e13ce

#13 0x00007f04c98377e5 __libc_start_main + 229

#14 0x0000000001f51aa9

/tmp/ifx0913271141uUEwL9/ifxEZNq88.i90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

compilation aborted for /var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/combine10parQVIsSpr2hop/Hamiltonian_RayTracing_module.f90 (code 3)

make[2]: *** [CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/build.make:274: CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build/combine10parQVIsSpr2hop/Hamiltonian_RayTracing_module.f90.o] Error 3

make[1]: *** [CMakeFiles/Makefile2:86: CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/all] Error 2

make: *** [Makefile:94: all] Error 2

make[2]: Leaving directory '/var/lib/buildbot/gpsii/gpsii_es_ifx_alma8_serial_debug/build_out'

make[1]: Leaving directory '/var/lib/buildbot

Devorah_H_Intel · ‎08-14-2024

Check to see if the issue exists in the 2024.2.1 version of the compiler. To reproduce it, we would need source codes.

Greg_B · ‎08-15-2024

(I work with Sergey in the same office.)

We confirm that yes, the issue persists in the 2024.2.1 version of the compiler. Here are the final lines of a log from building with Fortran compiler IntelLLVM 2024.2.1:

...

...
[ 11%] Building Fortran object CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90.o
/opt/intel/oneapi/compiler/2024.2/bin/ifx  -I/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../ionomodels/PIM/PIM/source -I/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../WSBITikhFltr/GPS_Data_makeUp -I/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../tleTrajectoryPropagator/SGP4/for -I/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/Projects/cmake/GPSII_WSBI_RefPnts_MGrids_URP_Es/../../../utils/saoXml -fiopenmp  -m64 -fpp -warn noerrors,nointerfaces -O0 -g -debug full -traceback -check all,noarg_temp_created -assume realloc_lhs -ftrapuv -init=snan,arrays -c /nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90 -o CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90.o
          #0 0x000000000312d792
          #1 0x0000000003191e57
          #2 0x000000000317d936
          #3 0x000000000307fa0c
          #4 0x000000000307f74d
          #5 0x0000000003168cdc
          #6 0x0000000003169d2f
          #7 0x000000000323b5a9
          #8 0x000000000323e282
          #9 0x00000000032340a5
         #10 0x0000000003234d74
         #11 0x00000000032327f7
         #12 0x00000000031694ab
         #13 0x0000000003169861
         #14 0x0000000003168cb2
         #15 0x0000000003169d2f
         #16 0x0000000003080cbd
         #17 0x00000000031d65ad
         #18 0x00000000031e0f14
         #19 0x00000000031e1495
         #20 0x00000000031de710
         #21 0x00000000031e0f14
         #22 0x00000000031e1495
         #23 0x00000000031665d7
         #24 0x0000000003166289
         #25 0x000000000316814e
         #26 0x00000000031e0f14
         #27 0x00000000031e1495
         #28 0x00000000031e3ed0
         #29 0x00000000031e0f14
         #30 0x00000000031e1495
         #31 0x00000000031decd2
         #32 0x00000000031e0f14
         #33 0x00000000031de38a
         #34 0x00000000031e0f14
         #35 0x00000000030787c6
         #36 0x0000000003078175
         #37 0x000000000324df17
         #38 0x0000152bf51aa7e5 __libc_start_main + 229
         #39 0x0000000002eb06ea

/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90(851): error #5623: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
        URP_list(i)%LatLonAlt(1:2)=UpdatedHiddenParameters(ipars1+[0,1])+URP_List_start_of_tstep(i)%LatLonAlt(1:2)
-----------------------------------------------------------------^
compilation aborted for /nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90 (code 3)
make[2]: *** [CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/build.make:3628: CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/WSBITikhFltr/with_reference_points/Unknown_Reference_Points_mod.f90.o] Error 3
make[2]: Leaving directory '/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/build_out'
make[1]: *** [CMakeFiles/Makefile2:86: CMakeFiles/GPSII_WSBI_RefPnts_MGrids_URP_Es.dir/all] Error 2
make[1]: Leaving directory '/nwra/monterey/monterey_users/greg/gpsii_code/gpsii_es_src_prj/build_out'
make: *** [Makefile:94: all] Error 2

TobiasK · ‎08-16-2024

@Greg_B @fridman__sergey as Devorah already mentioned, we need something to work with. If you could share a minimal reproducer, we are happy to debug and implement a fix, but with this information so far it's hard to do something meaningful.

Greg_B · ‎08-16-2024

We'll try to come up with a reproducer, but so far, we haven't seen the problem with Fortran code that we could upload.

Is there a debug build of ifx and/or ifort available that could provide more information (or maybe a traceback) about the error or the state of the compilers when the error occurs?

Or is there a compiler option that would generate more informative diagnostics?