topic Re: Error mkl_avx2.2.dll when calculating large array in Intel® oneAPI Math Kernel Library

Error mkl_avx2.2.dll when calculating large array

nvh10 — Mon, 22 Aug 2022 03:12:47 GMT

I need to calculate the operation A*P*A' by mkl function mkl_sparse_d_syprd.

In the example below, A is the identity matrix and P is the matrix with all elements are one. In my code, for the small array with nstate=100 or 1000, the code can run well. However if the values of nstate=50000 or larger, the following error appears: "Exception thrown at 0x00007FFDFB353C57 (mkl_avx2.2.dll) in Console22.exe: 0xC0000005: Access violation reading location 0x0000025986EEA108."

Please help me solve this problem. Thank you very much!

! A=Identity matrix !P=matrix with all elements are one program test_spblas use mkl_spblas implicit none integer, parameter :: nstate = 50000 double precision,allocatable, dimension (:,:):: P,APAT integer, allocatable, dimension (:):: c_A,pB_A,pE_A double precision, allocatable, dimension (:):: v_A integer stat,nnz_A,i type(sparse_matrix_t) :: A_s nnz_A=nstate allocate(v_A(nnz_A),c_A(nnz_A),pB_A(nstate),pE_A(nstate)) allocate(P(nstate,nstate),APAT(nstate,nstate)) do i=1,nstate pB_A(i)=i pE_A(i)=i+1 c_A(i)=i enddo v_A=1d0 P=1d0 APAT=0d0 stat = mkl_sparse_d_create_csr(a_s,sparse_index_base_one,nstate,nstate,pb_a,pe_a,c_a,v_a) stat = mkl_sparse_d_syprd (sparse_operation_non_transpose,a_s, p, sparse_layout_column_major, nstate, 1d0, 0d0, apat,spaRSE_LAYOUT_COLUMN_MAJOR, nstate) end program test_spblas

Re: Error mkl_avx2.2.dll when calculating large array

VidyalathaB_Intel — Mon, 22 Aug 2022 12:25:06 GMT

Hi,

Thanks for reaching out to us.

Could you please try running the code from Intel oneAPI command prompt and see if it is working there?

Please do let us know the MKL version with which you are working.

Regards,

Vidya.

Re: Error mkl_avx2.2.dll when calculating large array

nvh10 — Tue, 23 Aug 2022 00:57:24 GMT

This is my mkl version:

"Intel(R) oneAPI Math Kernel Library Version 2022.1-Product Build 20220311 for Intel(R) 64 architecture applications"

Re: Error mkl_avx2.2.dll when calculating large array

nvh10 — Tue, 23 Aug 2022 04:30:18 GMT

This is what I got from oneAPI command prompt :

ifort test_spblas.f90 /Qiopenmp /Qopenmp-targets:spir64 /module:"D:\Fortran\oneAPI\mkl\2021.2.0\include\intel64\ilp64" /DMKL_ILP64 /4I8 -I"D:\Fortran\oneAPI\mkl\2021.2.0\include" /MD /fpp

ifort: command line warning #10148: option '/Qiopenmp' not supported
ifort: command line warning #10148: option '/Qopenmp-targets:spir64' not supported
test.f90(30): error #6633: The type of the actual argument differs from the type of the dummy argument. [PB_A]
stat = mkl_sparse_d_create_csr(a_s,sparse_index_base_one,nstate,nstate,pb_a,pe_a,c_a,v_a)
---------------------------------------------------------------------------^
test.f90(30): error #6633: The type of the actual argument differs from the type of the dummy argument. [PE_A]
stat = mkl_sparse_d_create_csr(a_s,sparse_index_base_one,nstate,nstate,pb_a,pe_a,c_a,v_a)
--------------------------------------------------------------------------------^
test.f90(30): error #6633: The type of the actual argument differs from the type of the dummy argument. [C_A]
stat = mkl_sparse_d_create_csr(a_s,sparse_index_base_one,nstate,nstate,pb_a,pe_a,c_a,v_a)
-------------------------------------------------------------------------------------^

Re: Error mkl_avx2.2.dll when calculating large array

nvh10 — Tue, 23 Aug 2022 06:09:41 GMT

This is my last try. I used this command:

ifort /DMKL_DIRECT_CALL /fpp test.f90 mkl_intel_lp64.lib mkl_core.lib mkl_intel_thread.lib /Qopenmp -I"D:\Fortran\oneAPI\mkl\2021.2.0\include"/include

For nstate=10000. It's OK.

But for nstate=50000. It's said:

forrtl: severe (157): Program Exception - access violation Image PC Routine Line Source test.exe 00007FF7361CC128 Unknown Unknown Unknown test.exe 00007FF736169AE2 Unknown Unknown Unknown libiomp5md.dll 00007FFAE04B65D3 Unknown Unknown Unknown libiomp5md.dll 00007FFAE0409877 Unknown Unknown Unknown libiomp5md.dll 00007FFAE040B54C Unknown Unknown Unknown libiomp5md.dll 00007FFAE03C4CE1 Unknown Unknown Unknown test.exe 00007FF736169477 Unknown Unknown Unknown test.exe 00007FF736168012 Unknown Unknown Unknown test.exe 00007FF736151818 Unknown Unknown Unknown test.exe 00007FF7361D21BE Unknown Unknown Unknown test.exe 00007FF7361D2584 Unknown Unknown Unknown KERNEL32.DLL 00007FFB50CB7034 Unknown Unknown Unknown ntdll.dll 00007FFB51C62651 Unknown Unknown Unknown

Re:Error mkl_avx2.2.dll when calculating large array

VidyalathaB_Intel — Mon, 29 Aug 2022 07:15:23 GMT

Hi,

Thanks for sharing the details.

The issue is reproducible from our end as well.

We are working on this issue, we will get back to you soon.

Regards,

Vidya.

Re: Re:Error mkl_avx2.2.dll when calculating large array

nvh10 — Sat, 03 Sep 2022 02:13:55 GMT

Thank you very much for helping me!

Re: Error mkl_avx2.2.dll when calculating large array

Spencer_P_Intel — Fri, 17 Feb 2023 23:56:33 GMT

Hi nvh10,

It looks like there are a few things going on here, but the main one that is causing the overflow for nstate=50000 and not for nstate=10000 is the use of lp64. This means that pointers are 64 bit addresses, but integers are only 32 bit integers. Additionally, the internal implementation is actually in C language, so any 2D fortran pointers are actually collapsed and treated like 1D C arrays (you can see this in the module file with regards to the pointer DIMENSION(*)) . This normally works out, but may be to our disadvantage here in this case when dealing with integers and offsets...

It turns out that the range of int32 is [-2147483648, 2147483647] and the smallest int32 N such that N*N is in this range is N=46340. So for 50000, if we are doing something like C[ row * ldc + col] and row, ldc and col are 32 bit integers, then it is possible (likely) they could overflow and end up negative, then be upcast (still negative) in some way in the address offset computation which results in a seg fault. There are things we can do internally to make sure these addresses are computed using 64 bit integers which are compatible with the 64 bit addresses, and we will do this more carefully in the product, but otherwise, we need to be careful about this.

We are still looking into some other aspects of the ilp64 solution where it appears that the ldb and ldc are sometimes incorrect when they get to our internal kernels. Will update on that once more is understood.

Hope this helps a bit so far

Spencer

Re: Error mkl_avx2.2.dll when calculating large array

Spencer_P_Intel — Wed, 22 Feb 2023 19:01:16 GMT

Ok, here is the rest of the details. It turns out that there was an additional issue in the mkl_spblas.f90 module file for mkl_sparse_x_syprd which prevented the ilp64 version from working properly. We use ISO_C_BINDING 's to map this from the Fortran API you are calling to a C function implemented internally. In the case of mkl_sparse_x_syprd, there are two input arguments: ldb and ldc which where being incorrectly mapped.

If you change

INTEGER(C_INT) , INTENT(IN) :: ldb
and
INTEGER(C_INT) , INTENT(IN) :: ldc

to

INTEGER, INTENT(IN) :: ldb
and
INTEGER, INTENT(IN) :: ldc

then everything will work as desired. The C_INT kind always maps to a 4 byte integer, but ldb and ldc should be a 4 or 8 byte integer depending on use of the compiler option ( -i8 on linux/mac ) or ( /4I8 on Windows) to make integer 8 bytes in Fortran.

These are changes you can make yourself to the module file if this is necessary for another project immediately and will be fixed in the next oneMKL release (likely oneMKL 2023.1). Thank you for sharing this issue, so we could fix it

Re:Error mkl_avx2.2.dll when calculating large array

Ruqiu_C_Intel — Wed, 01 Mar 2023 01:42:58 GMT

Thank you again to raise the issue. The fixed will be available in oneMKL next release.