Internal compiler error: segmentation violation signal raised (WRF 4.5.1)

[kis331ema]

Hello

I'm trying to compile WRF 4.5.1 with Intel Compilers 2024.0.2 and Intel MPI 2021.10.0 but if I add the -O3 optimization, I get this error:

mpiifx -o module_mp_nssl_2mom.o -c -O3 -axCORE-AVX512 -ip -fp-model precise -w -ftz -align all -fno-alias -FR -convert big_endian -qopenmp -I../dyn_em -I/gxfs/scratch/process/kserradell/WRFV4.5.1/external/esmf_time_f90 -I/gxfs/scratch/process/kserradell/WRFV4.5.1/main -I/gxfs/scratch/process/kserradell/WRFV4.5.1/external/io_netcdf -I/gxfs/scratch/process/kserradell/WRFV4.5.1/external/io_int -I/gxfs/scratch/process/kserradell/WRFV4.5.1/frame -I/gxfs/scratch/process/kserradell/WRFV4.5.1/share -I/gxfs/scratch/process/kserradell/WRFV4.5.1/phys -I/gxfs/scratch/process/kserradell/WRFV4.5.1/wrftladj -I/gxfs/scratch/process/kserradell/WRFV4.5.1/chem -I/gxfs/scratch/process/kserradell/WRFV4.5.1/inc -I/gxfs/apps/easybuild/software/netCDF-merged/4.9.0-4.6.0-iimpi-2022b/include -real-size `expr 8 \* 4` -i4 module_mp_nssl_2mom.f90
#0 0x000000000232d4ea
#1 0x0000000002394d07
#2 0x0000000002394e30
#3 0x000014d01dafcb20
#4 0x00000000042360a5
#5 0x00000000042343f4
#6 0x0000000004232f25
#7 0x0000000004232879
#8 0x00000000032a58ef
#9 0x00000000032a576d
#10 0x00000000026d1053
#11 0x00000000022e190d
#12 0x00000000026d7b02
#13 0x00000000022e167d
#14 0x00000000026cfc53
#15 0x00000000022c9c9b
#16 0x00000000022c7c67
#17 0x0000000002270263
#18 0x0000000002452dbe
#19 0x000014d01dae8cf3 __libc_start_main + 243
#20 0x00000000020ab129

module_mp_nssl_2mom.f90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
compilation aborted for module_mp_nssl_2mom.f90 (code 3)
make[3]: [../configure.wrf:403: module_mp_nssl_2mom.o] Error 3 (ignored)

This error does not appear if I remove -O3.

Could you please take a look and correct if possible?

Thanks,

KiM

[Barbara_P_Intel]

WRF is a HUGE application! Can you please post module_mp_nssl_2mom.f90 and all the files it needs to reproduce the issue?

Does the WRF build environment use -O3 as the default? I'm curious.

[kis331ema]

Hello Barbara

Thanks for your answer.

Yes, WRF is quite big I'm attaching the file but the file alone won't be very useful, I think. Unfortunately, you should:

1. Download the source code 
2. Untar
3. ./configure
4. Select 16 
5. ./compile em_real

And yes, WRF enables -O3 by default.

Thanks for your support!

KiM

[mecej4]

It is certainly true that WRF is a huge software package with lots of dependencies and interdependencies, but the particular source file that @kis331ema posted can be compiled by itself, since it has no USE or INCLUDE statements in it. I tried compiling the file using /O3 on Windows using IFX Version 2024.0.0 Build 20231017, and the compiler said,

xfortcom: Fatal: There has been an internal compiler error (C0000005).
compilation aborted for module_mp_nssl_2mom.f90 (code 1)

With /O2, the compilation of the  20 kLOC file succeeded, and took 28 seconds (on an AMD 4800U).

[Ron_Green · Accepted solution]

I can confirm the ICE with -O3 and it needs a -x or -ax option to crash.  My testing was on Linux.  -O3 by itself does not crash, but we do want some processor targeting with -x, like -xhost that I use.

But the preview build of 2024.1 does not show the issue.  This compiler will be available late March or early April.  My guess is early April.   Below are 3 compilations.  First one is 2024.0 and it does crash, as you know.  Second one is the 2024.1 early build.  The third is our main branch code, just to confirm we won't see this crash in 2024.2.  So it appears fixed in that 2024.1 release that is coming soon. 

ifx -c -O3 -xhost  module_mp_nssl_2mom.f90 -V
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.0.0 Build 20231017
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

 Intel(R) Fortran 24.0-1238.2
 #0 0x0000000002394df7 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x2394df7)
 #1 0x0000000002394f20 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x2394f20)
 #2 0x00007f4dbb8e6b70 __restore_rt (/lib64/libc.so.6+0x3db70)
 #3 0x0000000004236205 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x4236205)
 #4 0x0000000004234554 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x4234554)
 #5 0x0000000004233085 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x4233085)
 #6 0x00000000042329d9 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x42329d9)
 #7 0x00000000032a587f (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x32a587f)
 #8 0x00000000032a56fd (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x32a56fd)
 #9 0x00000000026d10e3 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x26d10e3)
#10 0x00000000022e19fd (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x22e19fd)
#11 0x00000000026d7b82 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x26d7b82)
#12 0x00000000022e176d (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x22e176d)
#13 0x00000000026cfce3 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x26cfce3)
#14 0x00000000022c9d9a (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x22c9d9a)
#15 0x00000000022c7d6d (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x22c7d6d)
#16 0x0000000002270263 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x2270263)
#17 0x0000000002452e7e (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x2452e7e)
#18 0x00007f4dbb8d0b4a __libc_start_call_main (/lib64/libc.so.6+0x27b4a)
#19 0x00007f4dbb8d0c0b __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x27c0b)
#20 0x00000000020ab129 (/nfs/pdx/disks/cts2/tools/oneapi/2024.0.0/compiler/2024.0/bin/compiler/xfortcom+0x20ab129)

module_mp_nssl_2mom.f90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for module_mp_nssl_2mom.f90 (code 3)
rwgreen@orcsle153:~/quad/friday$ 
rwgreen@orcsle153:~/quad/friday$ module purge ; module load oneapi/2024.1.0-preview 
rwgreen@orcsle153:~/quad/friday$ 
rwgreen@orcsle153:~/quad/friday$ ifx -c -O3 -xhost  module_mp_nssl_2mom.f90 -V
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.1.0 Build 20240125
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

 Intel(R) Fortran 24.0-1472.3
rwgreen@orcsle153:~/quad/friday$ module purge ; module load oneapi/nightly
rwgreen@orcsle153:~/quad/friday$ 
rwgreen@orcsle153:~/quad/friday$ ifx -c -O3 -xhost  module_mp_nssl_2mom.f90 -V
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version dev.x.0 Mainline Build 20240229
Copyright (C) 1985-2024 Intel Corporation. All rights reserved.

 Intel(R) Fortran 24.0-1594

[Barbara_P_Intel]

@kis331ema, the Intel WRF knowledge expert cannot reproduce the ICE using the standard WRF configure.wrf file. Please post your configure.wrf file.

Also, share the output from

ifx --version

gcc --version

I realize that Ron reproduced the ICE and that is a problem, but he didn't use the compiler options in configure.wrf.

Thank you!

[kis331ema]

Hello

I'm confirming @Ron_Green findings. As @mecej4 ssuggests, the file can be compiled independently. On our HPC  Linux cluster:

[process@mmlogin01 ~ ]$ ifx -c -O3 -xhost module_mp_nssl_2mom.f90 -V
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.0.2 Build 20231213
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

Intel(R) Fortran 24.0-1238.2
#0 0x000000000232d4ea
#1 0x0000000002394d07
#2 0x0000000002394e30
#3 0x000014d1bb492b20
#4 0x00000000042360a5
#5 0x00000000042343f4
#6 0x0000000004232f25
#7 0x0000000004232879
#8 0x00000000032a58ef
#9 0x00000000032a576d
#10 0x00000000026d1053
#11 0x00000000022e190d
#12 0x00000000026d7b02
#13 0x00000000022e167d
#14 0x00000000026cfc53
#15 0x00000000022c9c9b
#16 0x00000000022c7c67
#17 0x0000000002270263
#18 0x0000000002452dbe
#19 0x000014d1bb47ecf3 __libc_start_main + 243
#20 0x00000000020ab129

module_mp_nssl_2mom.f90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
compilation aborted for module_mp_nssl_2mom.f90 (code 3)

[process@mmlogin01 ~ ]$ ifx -c -O3 module_mp_nssl_2mom.f90 -V
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.0.2 Build 20231213
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

Intel(R) Fortran 24.0-1238.2

Using flags from WRF will also crash

[process@mmlogin01 ~ ]$ ifx -o module_mp_nssl_2mom.o -c -O3 -axCORE-AVX512 -ip -fp-model precise -w -ftz -align all -fno-alias -FR -convert big_endian -qopenmp -I../dyn_em -I/gxfs/scratch/process/kserradell/WRFV4.5.1/external/esmf_time_f90 -I/gxfs/scratch/process/kserradell/WRFV4.5.1/main -I/gxfs/scratch/process/kserradell/WRFV4.5.1/external/io_netcdf -I/gxfs/scratch/process/kserradell/WRFV4.5.1/external/io_int -I/gxfs/scratch/process/kserradell/WRFV4.5.1/frame -I/gxfs/scratch/process/kserradell/WRFV4.5.1/share -I/gxfs/scratch/process/kserradell/WRFV4.5.1/phys -I/gxfs/scratch/process/kserradell/WRFV4.5.1/wrftladj -I/gxfs/scratch/process/kserradell/WRFV4.5.1/chem -I/gxfs/scratch/process/kserradell/WRFV4.5.1/inc -I/gxfs/apps/easybuild/software/netCDF-merged/4.9.0-4.6.0-iimpi-2022b/include -real-size `expr 8 \* 4` -i4 module_mp_nssl_2mom.f90
#0 0x000000000232d4ea
#1 0x0000000002394d07
#2 0x0000000002394e30
#3 0x0000151a2831db20
#4 0x00000000042360a5
#5 0x00000000042343f4
#6 0x0000000004232f25
#7 0x0000000004232879
#8 0x00000000032a58ef
#9 0x00000000032a576d
#10 0x00000000026d1053
#11 0x00000000022e190d
#12 0x00000000026d7b02
#13 0x00000000022e167d
#14 0x00000000026cfc53
#15 0x00000000022c9c9b
#16 0x00000000022c7c67
#17 0x0000000002270263
#18 0x0000000002452dbe
#19 0x0000151a28309cf3 __libc_start_main + 243
#20 0x00000000020ab129

module_mp_nssl_2mom.f90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
compilation aborted for module_mp_nssl_2mom.f90 (code 3)

I'm also sharing the configure.wrf file and the requested version information.

[process@mmlogin01 ~ ]$ ifx --version
ifx (IFX) 2024.0.2 20231213
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

[process@mmlogin01 ~ ]$ gcc --version
gcc (GCC) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks!

KiM

[Barbara_P_Intel]

Thank you!