topic Re: HPL compiled with OneAPI HPC toolkit failed to converge in Intel® oneAPI Math Kernel Library

HPL compiled with OneAPI HPC toolkit failed to converge

PingLin — Mon, 14 Apr 2025 15:12:46 GMT

I have tried to compile HPL using the new 2025 intel OneAPI HPC toolkit to run benchmark test on RedHat Linux Enterprise 8 and 9. I downloaded the Netlib HPL source code from http://www.netlib.org/benchmark/hpl/ .

And following the steps suggested by the manual:
1. Download and extract the source code.
2. Copy the makefile:
$> cp setup/Make.Linux_Intel64 .
3. Edit Make.Linux_Intel64 as appropriate
4. Build the HPL binary:
$> make arch=Linux_Intel64
5. Check that the built binary is available in the bin/Linux_Intel64 directory.

I have played with different changes on Make.Linux_Intel64 file. While they all compiled without complains. During the test run, all 864 tests completed and failed residual checks.

Here is one of the changes I made to the Make.Linux_Intel64 file (parts that changed shown).

# - Message Passing library (MPI) --------------------------------------
MPdir = /opt/intel/oneapi/mpi/2021.15/
MPinc = -I$(MPdir)/include
MPlib = $(MPdir)/lib/libmpi.a

# - Linear Algebra library (BLAS or VSIPL) -----------------------------
LAdir = $(MKLROOT)
ifndef LAinc
LAinc = $(LAdir)/include
endif
ifndef LAlib
LAlib = -L$(LAdir)/lib \
        -Wl,--start-group \
        $(LAdir)/lib/libmkl_intel_lp64.a \
        $(LAdir)/lib/libmkl_intel_thread.a \
        $(LAdir)/lib/libmkl_core.a \
        -Wl,--end-group -lpthread -ldl
endif
# - Compilers / linkers - Optimization flags ---------------------------
CC = mpiicx
CCNOOPT = $(HPL_DEFS)
OMP_DEFS = -qopenmp
CCFLAGS = $(HPL_DEFS) -O3 -w -ansi-alias -z noexecstack -z relro -z now -Wall

I have changes the settings and none is working.

My question is, what is the appropriate change for the Make.Linux_Intel64 file?

The Intel® Distribution for LINPACK Benchmark may contain additional optimizations compared to the reference Netlib HPL implementation. Is there anyway to find out the additional optimizations?

The output from the failed test is attached.

Re: HPL compiled with OneAPI HPC toolkit failed to converge

PingLin — Tue, 15 Apr 2025 13:11:00 GMT

Additional notes: compilation works with GCC (mpicc) and linked with MKL. And it appears to be the way many compilation is done. And my question is why OneAPI C compiler doesn't work?

Re: HPL compiled with OneAPI HPC toolkit failed to converge

PingLin — Sun, 27 Apr 2025 23:49:25 GMT

As it turns out, OneAPI C compiler includes certain level of optimization even if not specify -O0, which cause the subroutine HPL_dlamch.c to fail. On line 274 of the file, `{ a *= HPL_rtwo; c = HPL_dlamc3( a, one ); c = HPL_dlamc3( c, -a ); } ` should be executed without optimization, so it can use limited precision of the float data type to determine the value of `a` where `float( a + 1.0 ) == a`. With optimization it returns `a = inf`. That results TEST->epsil = 0 instead of a value of about ~ 1e-16.

Re: HPL compiled with OneAPI HPC toolkit failed to converge

Kazushige_G_Intel — Mon, 28 Apr 2025 16:23:36 GMT

Can you add -O0 option into CCNOOPT?

CCNOOPT = $(HPL_DEFS) -O0

Re: HPL compiled with OneAPI HPC toolkit failed to converge

Fengrui — Mon, 28 Apr 2025 20:03:05 GMT

As it turns out, OneAPI C compiler includes certain level of optimization even if not specify -O0, ...

> If -O[n] is not specify, the default is -O2.

Re: HPL compiled with OneAPI HPC toolkit failed to converge

PingLin — Wed, 30 Apr 2025 12:20:24 GMT

Changing intel default with `-fp-model=precise` while leaving -O2 option on also enable HPL to be compiled correctly and passed the tests. But the combination of both is too aggressive.