- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looks like there might be an integer overflow problem with indexing in the internals of the copy operation. By way of comparison, this test works on the same machine using gcc compiled Python 2.7.12 and Numpy 1.11.2. Thanks for your help in resolving this matter.
>>> import sys >>> print sys.version 2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] >>> import numpy as N >>> print N.version.version 1.11.3 >>> a = N.empty([715827882, 3]) >>> a.fill(1) >>> b = N.copy(a) >>> a[-1] array([ 1., 1., 1.]) >>> b[-1] array([ 1., 1., 1.]) >>> a = N.empty([715827883, 3]) >>> a.fill(1) >>> b = N.copy(a) >>> a[-1] array([ 1., 1., 1.]) >>> b[-1] array([ 0., 0., 0.]) >>> print a.size, 2**31 2147483649 2147483648 >>>
#System info $ uname -a Linux worker1038 3.10.0-327.3.1.el7.x86_64 #1 SMP Wed Dec 9 14:09:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux $ cpuinfo Intel(R) processor family information utility, Version 2017 Update 1 Build 20161016 (id: 16418) Copyright (C) 2005-2016 Intel Corporation. All rights reserved. ===== Processor composition ===== Processor name : Intel(R) Xeon(R) E5-2680 v4 Packages(sockets) : 2 Cores : 28 Processors(CPUs) : 28 Cores per package : 14 Threads per core : 1 ===== Processor identification ===== Processor Thread Id. Core Id. Package Id. 0 0 0 0 1 0 0 1 2 0 1 0 3 0 1 1 4 0 2 0 5 0 2 1 6 0 3 0 7 0 3 1 8 0 4 0 9 0 4 1 10 0 5 0 11 0 5 1 12 0 6 0 13 0 6 1 14 0 8 0 15 0 8 1 16 0 9 0 17 0 9 1 18 0 10 0 19 0 10 1 20 0 11 0 21 0 11 1 22 0 12 0 23 0 12 1 24 0 13 0 25 0 13 1 26 0 14 0 27 0 14 1 ===== Placement on packages ===== Package Id. Core Id. Processors 0 0,1,2,3,4,5,6,8,9,10,11,12,13,14 0,2,4,6,8,10,12,14,16,18,20,22,24,26 1 0,1,2,3,4,5,6,8,9,10,11,12,13,14 1,3,5,7,9,11,13,15,17,19,21,23,25,27 ===== Cache sharing ===== Cache Size Processors L1 32 KB no sharing L2 256 KB no sharing L3 35 MB (0,2,4,6,8,10,12,14,16,18,20,22,24,26)(1,3,5,7,9,11,13,15,17,19,21,23,25,27) $ cat /proc/meminfo MemTotal: 528281668 kB MemFree: 517686744 kB MemAvailable: 519347844 kB Buffers: 876 kB Cached: 2326032 kB SwapCached: 2516 kB Active: 5285028 kB Inactive: 875324 kB Active(anon): 4040624 kB Inactive(anon): 161560 kB Active(file): 1244404 kB Inactive(file): 713764 kB Unevictable: 24964 kB Mlocked: 0 kB SwapTotal: 12582908 kB SwapFree: 12389796 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 3856420 kB Mapped: 127636 kB Shmem: 343760 kB Slab: 934096 kB SReclaimable: 685912 kB SUnreclaim: 248184 kB KernelStack: 35680 kB PageTables: 30924 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 276723740 kB Committed_AS: 7832192 kB VmallocTotal: 34359738367 kB VmallocUsed: 1209020 kB VmallocChunk: 34090090096 kB HardwareCorrupted: 0 kB AnonHugePages: 3117056 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 248664 kB DirectMap2M: 4898816 kB DirectMap1G: 533725184 kB
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Thank you for providing a detailed report and for reporting the problem. We have reproduced the unexpected behavior, and will work to identify and fix the problem:
>>> import numpy as np >>> a = np.ones([715827883, 3], np.double) >>> b = np.copy(a) >>> b[-1] array([ 0., 0., 0.]) >>> a[-1] array([ 1., 1., 1.]) >>> a[0] array([ 1., 1., 1.]) >>> b[0] array([ 0., 0., 0.])
Oleksandr
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Nick,
I am happy to report that the problem was identified and fixed. I will report back when the fix becomes available to you with instructions on how to install it.
Sorry for the inconvenience,
Oleksandr
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Nick,
I am glad to let you know that the fix has become available on intel channel on Anaconda cloud. You can update by executing
conda install -c intel numpy=1.11.3
After upgrading:
(idp_2017u3) [13:31:17 vmlin scratch]$ python Python 3.5.3 |Intel Corporation| (default, Apr 27 2017, 18:08:47) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux Type "help", "copyright", "credits" or "license" for more information. Intel(R) Distribution for Python is brought to you by Intel Corporation. Please check out: https://software.intel.com/en-us/python-distribution >>> import numpy as np >>> a = np.ones([715827883, 3], np.double) >>> b = np.copy(a) >>> np.allclose(a[-1], b[-1]) True
Thank you,
Oleksandr
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Miniconda* or Anaconda* users:
For those who are using our packages from an environment of a Miniconda* or Anaconda* installation using the intelpython[2|3]_core metapackages, you can also get the updated numpy by updating the metapackage:
(ipy3core): /localdisk/work$ conda update intelpython3_core -c intel Fetching package metadata ......... Solving package specifications: .......... Package plan for installation in environment /localdisk/work/miniconda3/envs/ipy3core: The following packages will be downloaded: package | build ---------------------------|----------------- openmp-2017.0.3 | intel_8 615 KB intel numpy-1.11.3 | py35_intel_8 5.1 MB intel intelpython3_core-2017.0.3 | 2 1 KB intel ------------------------------------------------------------ Total: 5.7 MB The following packages will be UPDATED: intelpython3_core: 2017.0.3-1 intel --> 2017.0.3-2 intel numpy: 1.11.3-py35_intel_6 intel [intel] --> 1.11.3-py35_intel_8 intel [intel] openmp: 2017.0.3-intel_5 intel --> 2017.0.3-intel_8 intel Proceed (/n)?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I appreciated the effort made, but I think a more thorough review of the issue (potential overflow in the index space) is in order. With the update N.copy works, but other trivial opeartions still fail. Here is an example:
>>> import sys >>> import numpy as N >>> print sys.version 2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] >>> print N.version.version 1.11.3 >>> a = N.ones([715827882, 3]) >>> b = N.copy(a) >>> c = a + b >>> c[-1] array([ 2., 2., 2.]) >>> a = N.ones([715827883, 3]) >>> b = N.copy(a) >>> c = a + b Intel MKL ERROR: Parameter 1 was incorrect on entry to vdAdd. >>> c[-1] array([ 0., 0., 0.]) # A quick check that b looks like a copy of a: >>> a[-1] array([ 1., 1., 1.]) >>> b[-1] array([ 1., 1., 1.]) >>>
Thanks.
Yours,
-Nick

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page