dcopy problem with very large arrays

[Mark_Muller]

I have run into a problem when using dcopy with very large arrays.  Specifically, I am using dcopy to initialize a very large array to zero before doing other operations, and the dcopy is not setting all of the array to zero.  I think it is having a problem when the input n (ie, the length of the array) is greater than 2^31.  Attached is a simple piece of c++ that demonstrates the issue when run.  When compiled and run on my system, it gives the following output:

testing use of dcopy on very large arrays
trying length = 2147483647
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
trying length = 2147483649
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
problem with dbl_array
dbl_array[0] = 1

A few notes / some more information:

1)  I have found the same problem with dscal and daxpy when n exceed 2^31.

2)  I am using/linking with the ILP64 version of MKL.  I switched to the ILP64 version of MKL so that I could work with matrices that have more than 2^31 entries.  My compile and link command looks like:

   icpc dcopy_test.cc -DMKL_ILP64 -mkl

3)  I am compiling/running this on a scientific linux 6.2 (ie, a redhat like) system.  The system is dual xeon e5-2667 with 128 GB memory, so it isn't running out of memory space.

4)  I am compiling with icpc, whose version is:

Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.1.4.319 Build 20120410

5)  I am using the MKL libraries from composer_xe_2011_sp1.10.319.  Specifially, an ldd on the simple executable gives:

    linux-vdso.so.1 =>  (0x00007fffdbfff000)
    libmkl_intel_lp64.so => /opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64/libmkl_intel_lp64.so (0x00007f65bcf3f000)
    libmkl_intel_thread.so => /opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64/libmkl_intel_thread.so (0x00007f65bbebf000)
    libmkl_core.so => /opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64/libmkl_core.so (0x00007f65bae49000)
    libiomp5.so => /opt/intel/composer_xe_2011_sp1.10.319/compiler/lib/intel64/libiomp5.so (0x00007f65bab57000)
    libm.so.6 => /lib64/libm.so.6 (0x0000003349800000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003354400000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003354000000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003349400000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003348c00000)
    libdl.so.2 => /lib64/libdl.so.2 (0x0000003349000000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003348800000)

Thanks in advance for any insight.

Mark Muller

markmuller@gmail.com

[Zhang_Z_Intel]

Hi, Thanks for reporting the problem and providing a reproducer. I'm investigating this and will report back here as soon as I find anything.

[Vamsi_S_Intel]

Hi Mark,

The compiler option "-mkl" does not support MKL ILP64 interface. Quoting from the MKL user guide (http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_userguide_lnx/index.htm#GUID-4CD563F6-40A0-41E7-89C9-9F30B80DA693.htm )
======
On Intel® 64 architecture systems, for each variant of the -mkl option, the compiler links your application using the LP64 interface.
If you specify any variant of the -mkl compiler option, the compiler automatically includes the Intel MKL libraries. In cases not covered by the option, use the Link-line Advisor or see Linking in Detail.
========
When a program is compiled/linked with "-mkl", the compiler chooses to pick the MKL LP64 interface library. This can be confirmed by looking at the output of ldd,
libmkl_intel_lp64.so is the MKL LP64 interface library and we can see that it is being used. As the test program uses arrays of sizes greater than 2^31, but links with LP64 interface library it causes errors.

If I explicitly link the test program with MKL and run (see below),  then I do not get any errors.

[linux-host]$icpc  -openmp -DMKL_ILP64 -I$MKLROOT/include dcopy-test.cpp  -L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -lpthread -lm -o mklExplicitLinked.out
[linux-host]$ ./mklExplicitLinked.out

testing use of dcopy on very large arrays
trying length = 2147483647
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
trying length = 2147483649
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
everything worked OK

For more options on how-to explicitly link with MKL, please see the handy webtool "MKL Link Line Advisor": http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor

[Zhang_Z_Intel]

Thanks to Vamsi for your excellent explanation!

[SergeyKostrov]

Take a look at mkl_types.h header file and you will see the following declaration: ... /* MKL LP64 integer types */ #ifndef MKL_INT #define MKL_INT int #endif #ifndef MKL_UINT #define MKL_UINT unsigned int #endif ... On a 64-bit platform int is a 32-bit integer type ( the same is applicable for long ).