ICX says no. :(

[dalel487]

Since the classic compiler's been dropped with 2024.0.0, I've been trying to compile my Lattice QCD simulation with mpiicx instead. But it keeps failing.

There's an open git issue here:

https://github.com/dalel487/su2hmc/issues/27

if it helps.

Compilation works perfectly with mpiicc, gcc and AMD's AOCC compiler (with MPICH or OpenMPI). I've also manually sourced the classic compiler using

source /opt/intel/oneapi/compiler/2023.2.2/env/vars.sh

Useful details:

1. I'm using Ubuntu Jammy (Linux version 6.2.0-36-generic (buildd@lcy02-amd64-050) (x86_64-linux-gnu-gcc-11 (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #37~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 9 15:34:04 UTC 2)
2. Build/target machine is a W-2245 CPU
3. In the file 'include/sizes.h' there are variables 'npx' and 'npt'. If these are defined to be one then the compiler does not compile the MPI statements. They should be one in the git commit.
4. To build, go to the `production_c`directory and type

make -B -f Makefile_OneAPI release​

`debug` disables optimisations and `profiler` builds the same as `release` but with line information included.

First up. Building with the `-ipo` option gives the following output.

"/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o su2hmc_OneAPI /lib/x86_64-linux-gnu/Scrt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/12/crtbeginS.o -L/opt/intel/oneapi/mkl/2024.0/lib/intel64 -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/opt/intel/oneapi/mkl/2024.0/lib -L/opt/intel/oneapi/compiler/2024.0/lib/clang/17/lib/x86_64-unknown-linux-gnu -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/usr/lib/gcc/x86_64-linux-gnu/12 -L/usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/lib/../lib64 -L/usr/lib/gcc/x86_64-linux-gnu/12/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/lib -L/usr/lib/gcc/x86_64-linux-gnu/12/../../.. -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../opt/compiler/lib -L/lib -L/usr/lib -plugin /opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib/icx-lto.so -plugin-opt=mcpu=skylake -plugin-opt=O3 -plugin-opt=-vector-library=SVML -plugin-opt=fintel-libirc-allowed -plugin-opt=fintel-advanced-optim -plugin-opt=-disable-hir-generate-mkl-call -plugin-opt=-enable-npm-multiversioning -plugin-opt=-loopopt -plugin-opt=-intel-abi-compatible=true -plugin-opt=-x86-enable-unaligned-vector-move=true -L/opt/intel/oneapi/tbb/2021.11/env/../lib/intel64/gcc4.8 -L/opt/intel/oneapi/mpi/2021.11/lib -L/opt/intel/oneapi/mkl/2024.0/lib/ -L/opt/intel/oneapi/ippcp/2021.9/lib/ -L/opt/intel/oneapi/ipp/2021.10/lib -L/opt/intel/oneapi/dpl/2022.3/lib -L/opt/intel/oneapi/dnnl/2024.0/lib -L/opt/intel/oneapi/dal/2024.0/lib -L/opt/intel/oneapi/compiler/2024.0/lib -L/opt/intel/oneapi/ccl/2021.11/lib/ -L/opt/intel/oneapi/compiler/2023.2.2/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2023.2.2/linux/lib -L/opt/AMD/aocl/aocl-linux-aocc-4.1.0/aocc/lib -L/opt/AMD/aocc-compiler-4.1.0/lib -L/opt/AMD/aocc-compiler-4.1.0/lib32 -L/usr/lib/x86_64-linux-gnu -L/usr/lib64 -L/usr/lib32 -L/usr/lib -L. coord.o random.o matrices.o congrad.o bosonic.o fermionic.o force.o par_mpi.o su2hmc.o -limf -lgsl main.o --start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core --end-group -Bstatic -lsvml -Bdynamic -Bstatic -lirng -Bdynamic -Bstatic -limf -Bdynamic -lm -lgcc --as-needed -lgcc_s --no-as-needed -Bstatic -lirc -Bdynamic -ldl -liomp5 -L/opt/intel/oneapi/compiler/2024.0/bin/lib -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed -Bstatic -lirc_s -Bdynamic /usr/lib/gcc/x86_64-linux-gnu/12/crtendS.o /lib/x86_64-linux-gnu/crtn.o
icx: error: unable to execute command: Segmentation fault (core dumped)
icx: error: linker command failed due to signal (use -v to see invocation)
make: *** [Makefile_OneAPI:62: su2hmc_OneAPI] Error 1

However, when I tried to compile without `-flto` or `-ipo` I got

"/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang" -cc1 -triple x86_64-unknown-linux-gnu -S -save-temps=cwd -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name fermionic.c -mrelocation-model static -fveclib=SVML -mframe-pointer=none -menable-no-infs -menable-no-nans -fapprox-func -funsafe-math-optimizations -fno-signed-zeros -mreassociate -freciprocal-math -fdenormal-fp-math=preserve-sign,preserve-sign -ffp-contract=fast -fno-rounding-math -ffast-math -ffinite-math-only -mconstructor-aliases -funwind-tables=2 -target-cpu skylake -mllvm -x86-enable-unaligned-vector-move=true -debugger-tuning=gdb -v -fcoverage-compilation-dir=/home/postgrad/dalel487/Code/su2/hybrid_code/production_c -resource-dir /opt/intel/oneapi/compiler/2023.2.2/linux/lib/clang/17 -O3 -std=gnu11 -fdebug-compilation-dir=/home/postgrad/dalel487/Code/su2/hybrid_code/production_c -ferror-limit 19 -fheinous-gnu-extensions -fopenmp-late-outline -fopenmp-threadprivate-legacy -fopenmp -fgnuc-version=4.2.1 -finline-functions -mllvm -enable-gvn-hoist -fcolor-diagnostics -vectorize-loops -vectorize-slp -mprefer-vector-width=512 -mllvm -paropt=31 -fopenmp-typed-clauses -ax=skylake-avx512,cascadelake,tigerlake,skylake,alderlake,raptorlake -D__GCC_HAVE_DWARF2_CFI_ASM=1 -fintel-compatibility -fintel-libirc-allowed -fintel-advanced-optim -mllvm -disable-hir-generate-mkl-call -mllvm -loopopt -floopopt-pipeline=full -mllvm -intel-abi-compatible=true -o fermionic.s -x cpp-output fermionic.i
clang -cc1 version 17.0.0 based upon LLVM 17.0.0git default target x86_64-unknown-linux-gnu
#include "..." search starts here:
End of search list.
PLEASE append the compiler options "-save-temps -v", rebuild the application to to get the full command which is failing and submit a bug report to https://software.intel.com/en-us/support/priority-support which includes the failing command, input files for the command and the crash backtrace (if any).
Stack dump:
0. Program arguments: /opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang -cc1 -triple x86_64-unknown-linux-gnu -S -save-temps=cwd -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name fermionic.c -mrelocation-model static -fveclib=SVML -mframe-pointer=none -menable-no-infs -menable-no-nans -fapprox-func -funsafe-math-optimizations -fno-signed-zeros -mreassociate -freciprocal-math -fdenormal-fp-math=preserve-sign,preserve-sign -ffp-contract=fast -fno-rounding-math -ffast-math -ffinite-math-only -mconstructor-aliases -funwind-tables=2 -target-cpu skylake -mllvm -x86-enable-unaligned-vector-move=true -debugger-tuning=gdb -v -fcoverage-compilation-dir=/home/postgrad/dalel487/Code/su2/hybrid_code/production_c -resource-dir /opt/intel/oneapi/compiler/2023.2.2/linux/lib/clang/17 -O3 -std=gnu11 -fdebug-compilation-dir=/home/postgrad/dalel487/Code/su2/hybrid_code/production_c -ferror-limit 19 -fheinous-gnu-extensions -fopenmp-late-outline -fopenmp-threadprivate-legacy -fopenmp -fgnuc-version=4.2.1 -finline-functions -mllvm -enable-gvn-hoist -fcolor-diagnostics -vectorize-loops -vectorize-slp -mprefer-vector-width=512 -mllvm -paropt=31 -fopenmp-typed-clauses -ax=skylake-avx512,cascadelake,tigerlake,skylake,alderlake,raptorlake -D__GCC_HAVE_DWARF2_CFI_ASM=1 -fintel-compatibility -fintel-libirc-allowed -fintel-advanced-optim -mllvm -disable-hir-generate-mkl-call -mllvm -loopopt -floopopt-pipeline=full -mllvm -intel-abi-compatible=true -o fermionic.s -x cpp-output fermionic.i
1. <eof> parser at end of file
2. Optimizer
#0 0x000055d6d6336303 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x4f85303)
#1 0x000055d6d6334780 llvm::sys::RunSignalHandlers() (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x4f83780)
#2 0x000055d6d633694f SignalHandler(int) Signals.cpp:0:0
#3 0x00007f15d5e42520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x42520)
#4 0x000055d6d665c507 llvm::loopopt::RegDDRef::getTempBaseValue() const (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x52ab507)
#5 0x000055d6d66acb93 llvm::loopopt::DDTest::depends(llvm::loopopt::DDRef const*, llvm::loopopt::DDRef const*, llvm::loopopt::DirectionVector const&, bool, bool) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x52fbb93)
#6 0x000055d6d66b00e3 llvm::loopopt::DDTest::findDependencies(llvm::loopopt::DDRef*, llvm::loopopt::DDRef*, llvm::loopopt::DirectionVector const&, llvm::loopopt::DirectionVectorInfo&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x52ff0e3)
#7 0x000055d6d669d3bb llvm::loopopt::HIRDDAnalysis::buildGraph(llvm::loopopt::HIRGraph<llvm::loopopt::DDRef, llvm::loopopt::DDEdge>&, llvm::loopopt::HLNode const*) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x52ec3bb)
#8 0x000055d6d669c8e0 llvm::loopopt::HIRDDAnalysis::getGraphImpl(llvm::loopopt::HLRegion const*, llvm::loopopt::HLNode const*) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x52eb8e0)
#9 0x000055d6d6766642 llvm::loopopt::scalarreplarray::MemRefGroup::isLegal() const (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x53b5642)
#10 0x000055d6d6767719 llvm::loopopt::scalarreplarray::HIRScalarReplArray::doAnalysis(llvm::loopopt::HLLoop*) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x53b6719)
#11 0x000055d6d676750b llvm::loopopt::scalarreplarray::HIRScalarReplArray::run() (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x53b650b)
#12 0x000055d6d6768282 llvm::loopopt::HIRScalarReplArrayPass::runImpl(llvm::Function&, llvm::AnalysisManager<llvm::Function>&, llvm::loopopt::HIRFramework&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x53b7282)
#13 0x000055d6d769fdb0 llvm::detail::PassModel<llvm::Function, llvm::loopopt::HIRScalarReplArrayPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) PassBuilder.cpp:0:0
#14 0x000055d6d5e84340 llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x4ad3340)
#15 0x000055d6d53c843d llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) NVPTXTargetMachine.cpp:0:0
#16 0x000055d6d5e8b0e9 llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x4ada0e9)
#17 0x000055d6d53c81ad llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) NVPTXTargetMachine.cpp:0:0
#18 0x000055d6d5e83260 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x4ad2260)
#19 0x000055d6d6d5dda6 (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>&, std::__1::unique_ptr<llvm::ToolOutputFile, std::__1::default_delete<llvm::ToolOutputFile>>&) BackendUtil.cpp:0:0
#20 0x000055d6d6d582d6 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x59a72d6)
#21 0x000055d6d70e4dac clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) CodeGenAction.cpp:0:0
#22 0x000055d6d824790b clang::ParseAST(clang::Sema&, bool, bool) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x6e9690b)
#23 0x000055d6d70e39aa clang::CodeGenAction::ExecuteAction() (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x5d329aa)
#24 0x000055d6d7053faa clang::FrontendAction::Execute() (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x5ca2faa)
#25 0x000055d6d6fe08f5 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x5c2f8f5)
#26 0x000055d6d70e0561 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x5d2f561)
#27 0x000055d6d509cc5d cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x3cebc5d)
#28 0x000055d6d5098bb0 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0
#29 0x000055d6d5098542 clang_main(int, char**, llvm::ToolContext const&) (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x3ce7542)
#30 0x000055d6d50a5ace main (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x3cf4ace)
#31 0x00007f15d5e29d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#32 0x00007f15d5e29e40 call_init ./csu/../csu/libc-start.c:128:20
#33 0x00007f15d5e29e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#34 0x000055d6d5095da9 _start (/opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/clang+0x3ce4da9)
icx: error: unable to execute command: Segmentation fault (core dumped)
icx: error: clang frontend command failed due to signal (use -v to see invocation)
Intel(R) oneAPI DPC++/C++ Compiler 2023.2.2 (2023.2.2.20230908)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm
Configuration file: /opt/intel/oneapi/compiler/2023.2.2/linux/bin-llvm/../bin/icx.cfg
icx: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
icx: note: diagnostic msg: /tmp/icx-e01265/fermionic-40cf65.c
icx: note: diagnostic msg: /tmp/icx-e01265/fermionic-40cf65.sh
icx: note: diagnostic msg:

********************
make: *** [Makefile_OneAPI:65: fermionic.o] Error 1

Annoyingly those bug report files aren't in the tmp directory when I got to check, probably because it's a temp directory.

Thanks

[AishwaryaCV_Intel]

Hi,

Thank you for posting in Intel Communities.

We were able to reproduce the issue and we are working on it internally, we will get back to you soon.

Thanks and regards,

Aishwarya

[dalel487]

Thanks!

[VeenaJ_Intel]

Hi,

The code can be successfully compiled with mpiicx by implementing the following changes in the Makefile_OneApi:

CFLAGS= $(IFLAGS) -qmkl -ipo -qopt-zmm-usage=high -MD\

-axSKYLAKE-AVX512\

-DMKL_DIRECT_CALL -D_INTEL_MKL_ -D_RANLUX_\

-qopenmp -qopenmp-simd -DSA3AT -MMD -MP

Kindly consider applying the suggested modifications and testing the compilation. We would appreciate it if you could share your feedback or comments after implementing these changes.

Regards,

Veena

[VeenaJ_Intel]

Hi,

Since we didn't hear back from you, we assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards,

Veena