Issue with numpy.copy and large arrays?

Nick_C_ · ‎05-23-2017

Looks like there might be an integer overflow problem with indexing in the internals of the copy operation. By way of comparison, this test works on the same machine using gcc compiled Python 2.7.12 and Numpy 1.11.2. Thanks for your help in resolving this matter.

>>> import sys
>>> print sys.version
2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
>>> import numpy as N
>>> print N.version.version
1.11.3
>>> a = N.empty([715827882, 3])
>>> a.fill(1)
>>> b = N.copy(a)
>>> a[-1]
array([ 1.,  1.,  1.])
>>> b[-1]
array([ 1.,  1.,  1.])
>>> a = N.empty([715827883, 3])
>>> a.fill(1) 
>>> b = N.copy(a)
>>> a[-1]
array([ 1.,  1.,  1.])
>>> b[-1]
array([ 0.,  0.,  0.])
>>> print a.size, 2**31
2147483649 2147483648
>>>

#System info
$ uname -a

Linux worker1038 3.10.0-327.3.1.el7.x86_64 #1 SMP Wed Dec 9 14:09:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ cpuinfo 
Intel(R) processor family information utility, Version 2017 Update 1 Build 20161016 (id: 16418)
Copyright (C) 2005-2016 Intel Corporation.  All rights reserved.

=====  Processor composition  =====
Processor name    : Intel(R) Xeon(R)  E5-2680 v4 
Packages(sockets) : 2
Cores             : 28
Processors(CPUs)  : 28
Cores per package : 14
Threads per core  : 1

=====  Processor identification  =====
Processor       Thread Id.      Core Id.        Package Id.
0               0               0               0   
1               0               0               1   
2               0               1               0   
3               0               1               1   
4               0               2               0   
5               0               2               1   
6               0               3               0   
7               0               3               1   
8               0               4               0   
9               0               4               1   
10              0               5               0   
11              0               5               1   
12              0               6               0   
13              0               6               1   
14              0               8               0   
15              0               8               1   
16              0               9               0   
17              0               9               1   
18              0               10              0   
19              0               10              1   
20              0               11              0   
21              0               11              1   
22              0               12              0   
23              0               12              1   
24              0               13              0   
25              0               13              1   
26              0               14              0   
27              0               14              1   
=====  Placement on packages  =====
Package Id.     Core Id.        Processors
0               0,1,2,3,4,5,6,8,9,10,11,12,13,14                0,2,4,6,8,10,12,14,16,18,20,22,24,26
1               0,1,2,3,4,5,6,8,9,10,11,12,13,14                1,3,5,7,9,11,13,15,17,19,21,23,25,27

=====  Cache sharing  =====
Cache   Size            Processors
L1      32  KB          no sharing
L2      256 KB          no sharing
L3      35  MB          (0,2,4,6,8,10,12,14,16,18,20,22,24,26)(1,3,5,7,9,11,13,15,17,19,21,23,25,27)

$ cat /proc/meminfo 
MemTotal:       528281668 kB
MemFree:        517686744 kB
MemAvailable:   519347844 kB
Buffers:             876 kB
Cached:          2326032 kB
SwapCached:         2516 kB
Active:          5285028 kB
Inactive:         875324 kB
Active(anon):    4040624 kB
Inactive(anon):   161560 kB
Active(file):    1244404 kB
Inactive(file):   713764 kB
Unevictable:       24964 kB
Mlocked:               0 kB
SwapTotal:      12582908 kB
SwapFree:       12389796 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:       3856420 kB
Mapped:           127636 kB
Shmem:            343760 kB
Slab:             934096 kB
SReclaimable:     685912 kB
SUnreclaim:       248184 kB
KernelStack:       35680 kB
PageTables:        30924 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    276723740 kB
Committed_AS:    7832192 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     1209020 kB
VmallocChunk:   34090090096 kB
HardwareCorrupted:     0 kB
AnonHugePages:   3117056 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      248664 kB
DirectMap2M:     4898816 kB
DirectMap1G:    533725184 kB

Oleksandr_P_Intel · ‎05-23-2017

Hi,

Thank you for providing a detailed report and for reporting the problem. We have reproduced the unexpected behavior, and will work to identify and fix the problem:

>>> import numpy as np
>>> a = np.ones([715827883, 3], np.double)
>>> b = np.copy(a)
>>> b[-1]
array([ 0.,  0.,  0.])
>>> a[-1]
array([ 1.,  1.,  1.])
>>> a[0]
array([ 1.,  1.,  1.])
>>> b[0]
array([ 0.,  0.,  0.])

Oleksandr

Oleksandr_P_Intel · ‎05-23-2017

Dear Nick,

I am happy to report that the problem was identified and fixed. I will report back when the fix becomes available to you with instructions on how to install it.

Sorry for the inconvenience,
Oleksandr

Oleksandr_P_Intel · ‎06-01-2017

Dear Nick,

I am glad to let you know that the fix has become available on intel channel on Anaconda cloud. You can update by executing

conda install -c intel numpy=1.11.3

After upgrading:

(idp_2017u3) [13:31:17 vmlin scratch]$ python
Python 3.5.3 |Intel Corporation| (default, Apr 27 2017, 18:08:47)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import numpy as np
>>> a = np.ones([715827883, 3], np.double)
>>> b = np.copy(a)
>>> np.allclose(a[-1], b[-1])
True

Thank you,
Oleksandr

Todd_T_Intel · ‎06-01-2017

Miniconda* or Anaconda* users:

For those who are using our packages from an environment of a Miniconda* or Anaconda* installation using the intelpython[2|3]_core metapackages, you can also get the updated numpy by updating the metapackage:

(ipy3core): /localdisk/work$ conda update intelpython3_core -c intel

Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /localdisk/work/miniconda3/envs/ipy3core:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    openmp-2017.0.3            |          intel_8         615 KB  intel
    numpy-1.11.3               |     py35_intel_8         5.1 MB  intel
    intelpython3_core-2017.0.3 |                2           1 KB  intel
    ------------------------------------------------------------
                                           Total:         5.7 MB

The following packages will be UPDATED:

    intelpython3_core: 2017.0.3-1          intel --> 2017.0.3-2          intel
    numpy:             1.11.3-py35_intel_6 intel [intel] --> 1.11.3-py35_intel_8 intel [intel]
    openmp:            2017.0.3-intel_5    intel --> 2017.0.3-intel_8    intel

Proceed (/n)?

Nick_C_ · ‎07-05-2017

I appreciated the effort made, but I think a more thorough review of the issue (potential overflow in the index space) is in order. With the update N.copy works, but other trivial opeartions still fail. Here is an example:

>>> import sys
>>> import numpy as N
>>> print sys.version
2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
>>> print N.version.version
1.11.3
>>> a = N.ones([715827882, 3])
>>> b = N.copy(a)
>>> c = a + b
>>> c[-1]
array([ 2.,  2.,  2.])
>>> a = N.ones([715827883, 3])
>>> b = N.copy(a)
>>> c = a + b

Intel MKL ERROR: Parameter 1 was incorrect on entry to vdAdd.
>>> c[-1]
array([ 0.,  0.,  0.])
# A quick check that b looks like a copy of a:
>>> a[-1]
array([ 1.,  1.,  1.])
>>> b[-1]
array([ 1.,  1.,  1.])
>>>

Thanks.

Yours,

-Nick