intel-oneapi-compilers@2025.1.0 crashes

[aathan]

How do I proceed? I could tarball "the world" to provide Intel with the necessary context to reproduce, but that'd be a big file...

$ spack compilers
==> Available compilers
-- gcc rhel9-x86_64 ---------------------------------------------
gcc@11.4.1

-- intel-oneapi-compilers rhel9-x86_64 --------------------------
intel-oneapi-compilers@2025.1.0

$ /mnt/beegfs/xx/openmpi/bin/mpifort --show
ifx -I/mnt/beegfs/xx/openmpi/include -I/mnt/beegfs/xx/openmpi/lib -L/mnt/beegfs/xx/openmpi/lib -Wl,-rpath -Wl,/mnt/beegfs/xx/openmpi/lib -Wl,--enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

/mnt/beegfs/xx/openmpi/bin/mpifort -DINTERNAL_FILE_NML -g -Duse_libMPI -Duse_netCDF -Duse_netCDF -DHAVE_SCHED_GETAFFINITY -DHAVE_GETTID -fpp -Wp,-w -I/home/xx/libs/include -I/home/xx/libs/include -I/home/xx/libs/include -fno-alias -auto -ftz -assume byterecl -i4 -r8 -nowarn -traceback -g -O0 -qopenmp -c -I/mnt/beegfs/xx/cm5/intel-CM5/src/FMS/include -I/mnt/beegfs/xx/cm5/intel-CM5/src/FMS/mpp/include /mnt/beegfs/xx/cm5/intel-CM5/src/FMS/mpp/mpp.F90
#0 0x0000000003310581
#1 0x0000000003375357
#2 0x0000000003375485
#3 0x000015555503e6f0
#4 0x000000000330d2b3
#5 0x0000000003303a99
#6 0x0000000003301727
#7 0x000000000330091d
#8 0x00000000033c2f18
#9 0x00000000033c3505
#10 0x00000000033c6054
#11 0x00000000033c2f18
#12 0x00000000033c3505
#13 0x00000000033c0c96
#14 0x00000000033c2f18
#15 0x00000000033c034e
#16 0x00000000033c2f18
#17 0x000000000325628c
#18 0x0000000003255c44
#19 0x0000000003433b58
#20 0x0000155555029590
#21 0x0000155555029640 __libc_start_main + 128
#22 0x000000000308f0ee

[JohnNichols]

A better explanation. 

[aathan]

John, hope you're having a good weekend!

I'd be happy to give you more information. I first wanted to see if this forum is even the right and best place to engage with Intel about the issue. I understand it can be frustrating to deal with incomplete problem descriptions.

I suppose I did leave out the precise version as reported by the compiler executable:

$ mpifort --version

ifx (IFX) 2025.1.0 20250317
Copyright (C) 1985-2025 Intel Corporation. All rights reserved.

$ ifx --version
ifx (IFX) 2025.1.0 20250317
Copyright (C) 1985-2025 Intel Corporation. All rights reserved.

Beyond that, would you tell me what you feel would be helpful that can be provided in short-form in a text forum? My original messages shows the high level version of the compiler, and its command line. I know that's limited information, but I'm assuming Intel would want to reproduce the issue on their machines or at least be able to decode the stack trace perhaps with a corefile available? Please instruct me, as I did not immediately find a cookbook for preparing a useful problem report.

The 2023 version of the compiler compiles this file without crashing. This is the only file that crashes while compiling the CM5 NOAA benchmarking codes (https://code.ornl.gov/noaa-benchmarks). I'm building those against hand-built OpenMPI and UCX libraries using the same compiler suite on an Intel Xeon host.

There's a large number of additional facts I could list, but would prefer some guidance before writing a tome.

I imagine you'll ask me to create a minimal reproduction of the crash, but that'd be challenging given resource constraints. TBH I don't even know where to start, except perhaps a bisection of the source code, which would (again) be time consuming and inconclusive if other errors are introduced. I am not a fortran coder.

I'll state that I have decades of experience coding in other languages, so you can level set any response.

A.

PS: I chuckled that the forum put a "Translate" link under your three word response, as if snark is a foreign language. lol

[JohnNichols]

A simple sample is the best.  But you are working in an area where there are limited folk with the skills on this forum, probably Jim is the best.  If he did not respond then it is beyond even his skill set.  

Which NOAA models is a good place to start?

[aathan]

CM5.

I don't know how sensitive the bug is to particular ways the dependencies are built you may be able to get away with installing all of the below with spack or by using pre-builds but I found spack fails to correctly build a lot of these without interventions. I know this is a lot. That's why I didn't spew all of this in my initial message.

Ultimately, if someone from the Intel compiler group wants a giant tarball, it might be possible to arrange--but maybe the stack I already included in my first message will be enough... Let me know.

The rough recipe is to install the intel compiler, yaml, HDF5, netcdf-c, netcdf-fortran and xpmem using spack, pull down openmpi and ucx from repos, checkout v5.0.7rc2 openmpi and HEAD for ucx, then:

configure those similar to below. Forgive me, I have limited time. Grabbing these from bash_history, you can figure out what's what by asking ./configure --help and matching them with each library mentioned above. 

LDFLAGS="-L$(spack location -i xpmem)/lib" CFLAGS="-O3 -march=skylake-avx512 -I$(spack location -i xpmem)/include" FCFLAGS="-march=skylake-avx512 -O3" CC=icx FC=ifx CXX=icpx ../configure --enable-cma --enable-mt --enable-devel-headers --with-mlx5 --with-avx --with-rc --with-ud --with-dc --with-ib-hw-tm --with-dm --with-devx --with-efa --with-verbs --with-xpmem --prefix=/mnt/beegfs/xx/ucx

CXXFLAGS="-pthread -march=skylake-avx512 -O3" CFLAGS="-pthread -O3 -march=skylake-avx512" FCFLAGS="-pthread -fPIC -march=skylake-avx512 -O3" CC=icx FC=ifx CXX=icpx ../configure --prefix /mnt/beegfs/xx/openmpi --enable-branch-probabilities --with-libevent=internal --with-hwloc --with-pmix --with-curl --with-slurm --with-ucx=/mnt/beegfs/xx/ucx --with-tests-examples --with-xpmem=$(spack location -i xpmem) 

CFLAGS="-O3 -march=skylake-avx512" FCFLAGS="-march=skylake-avx512 -O3" CC=icx FC=ifx ./configure --enable-parallel --enable-fortran --enable-fortran2003 --prefix /home/xx/libs

H5DIR=/home/xx/libs CC=mpiicc CFLAGS=-I/home/xx/libs/include/ LDFLAGS=-L/home/xx/libs/lib LIBS=-ldl ./configure --prefix /home/xx/libs --disable-shared --enable-parallel-tests

NCDIR=/home/xx/libs NFDIR=/home/xx/libs CC=mpiicc FC=ifort CPPFLAGS=-I$(nc-config --includedir) LDFLAGS="-L/home/xx/libs/lib" LIBS="$(nc-config --static --libs)" ./configure --prefix=/home/xx/libs --disable-shared

and then attempt to compile CM5 by modifying the exec/intel-classic.mk file having made modifications such as below. The build instructions for CM5 are in the package you can get from NOAA.

FFLAGS_DEBUG = -g -O0 -check -check noarg_temp_created -check nopointer -warn -warn noerrors -fpe0 -ftrapuv $(ISA_DEBUG)OPENMP = yes # If non-blank, compile with openmp enabled
INCLUDES := $(shell pkg-config --cflags yaml-0.1)

LIBS := $(shell pkg-config --libs yaml-0.1) -L /cm/shared/apps/cm-pmix4/4.1.3/lib -L /home/xx/libs/lib -Wl,-Bstatic -lnetcdff -lnetcdf -lhdf5_fortran -lhdf5_hl_fortran -lhdf5_hl -lhdf5 -Wl,-Bdynamic -lm -lcurl -lz -lxml2 -lpmix