- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have run into a problem when using dcopy with very large arrays. Specifically, I am using dcopy to initialize a very large array to zero before doing other operations, and the dcopy is not setting all of the array to zero. I think it is having a problem when the input n (ie, the length of the array) is greater than 2^31. Attached is a simple piece of c++ that demonstrates the issue when run. When compiled and run on my system, it gives the following output:
testing use of dcopy on very large arrays
trying length = 2147483647
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
trying length = 2147483649
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
problem with dbl_array
dbl_array[0] = 1
A few notes / some more information:
1) I have found the same problem with dscal and daxpy when n exceed 2^31.
2) I am using/linking with the ILP64 version of MKL. I switched to the ILP64 version of MKL so that I could work with matrices that have more than 2^31 entries. My compile and link command looks like:
icpc dcopy_test.cc -DMKL_ILP64 -mkl
3) I am compiling/running this on a scientific linux 6.2 (ie, a redhat like) system. The system is dual xeon e5-2667 with 128 GB memory, so it isn't running out of memory space.
4) I am compiling with icpc, whose version is:
Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.1.4.319 Build 20120410
5) I am using the MKL libraries from composer_xe_2011_sp1.10.319. Specifially, an ldd on the simple executable gives:
linux-vdso.so.1 => (0x00007fffdbfff000)
libmkl_intel_lp64.so => /opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64/libmkl_intel_lp64.so (0x00007f65bcf3f000)
libmkl_intel_thread.so => /opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64/libmkl_intel_thread.so (0x00007f65bbebf000)
libmkl_core.so => /opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64/libmkl_core.so (0x00007f65bae49000)
libiomp5.so => /opt/intel/composer_xe_2011_sp1.10.319/compiler/lib/intel64/libiomp5.so (0x00007f65bab57000)
libm.so.6 => /lib64/libm.so.6 (0x0000003349800000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003354400000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003354000000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003349400000)
libc.so.6 => /lib64/libc.so.6 (0x0000003348c00000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003349000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003348800000)
Thanks in advance for any insight.
Mark Muller
markmuller@gmail.com
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, Thanks for reporting the problem and providing a reproducer. I'm investigating this and will report back here as soon as I find anything.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mark,
The compiler option "-mkl" does not support MKL ILP64 interface. Quoting from the MKL user guide (http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_userguide_lnx/index.htm#GUID-4CD563F6-40A0-41E7-89C9-9F30B80DA693.htm )
======
On Intel® 64 architecture systems, for each variant of the -mkl option, the compiler links your application using the LP64 interface.
If you specify any variant of the -mkl compiler option, the compiler automatically includes the Intel MKL libraries. In cases not covered by the option, use the Link-line Advisor or see Linking in Detail.
========
When a program is compiled/linked with "-mkl", the compiler chooses to pick the MKL LP64 interface library. This can be confirmed by looking at the output of ldd,
libmkl_intel_lp64.so is the MKL LP64 interface library and we can see that it is being used. As the test program uses arrays of sizes greater than 2^31, but links with LP64 interface library it causes errors.
If I explicitly link the test program with MKL and run (see below), then I do not get any errors.
[linux-host]$icpc -openmp -DMKL_ILP64 -I$MKLROOT/include dcopy-test.cpp -L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -lpthread -lm -o mklExplicitLinked.out
[linux-host]$ ./mklExplicitLinked.out
testing use of dcopy on very large arrays
trying length = 2147483647
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
trying length = 2147483649
manually setting dbl_array to 1
using dcopy to set dbl_array to 0
everything worked OK
For more options on how-to explicitly link with MKL, please see the handy webtool "MKL Link Line Advisor": http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks to Vamsi for your excellent explanation!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page