Intel® Distribution for Python*
Support and discussions for achieving faster Python* applications and core computational packages.

Python3 bug: numpy.percentile()

Stephan_U
Beginner
342 Views

We work with camera images from CMOS sensors which often use 16Bit image formats (int16, uint16). For some image sizes the implementation of percentile() causes memory corruptions/segfaults. This not observed for other data types (8, 32 or 64Bit) and is present at least in numpy version 1.13 ... 1.17 (releases 2018-1 through 2019-5). Other python distributions than intel behave normal.

We could boil down the problem to the attached 3-liner 'minimal-iP.txt':

import numpy as np
test = np.array([np.arange(i, 44+i, dtype=np.int16) for i in range(56)])
np.percentile(test, (0.1, 0.99))

$: python3  minimal-iP.txt

The resulting crash log is also attached. Other image sizes might work, (43x47) is also known to trigger the crash.

With best regards,

Stephan

 

0 Kudos
1 Reply
Oleksandr_P_Intel
342 Views

Hi Stephan, 

Thank you for providing a clean reproducer. I was able to reproduce the crash. The crash is due to aggressive compiler optimization of  {{dumb_select_short}} routine in numpy/core/src/npymath/selection.c.src, which implements quadratic complexity selection.

The fix is suppress vectorization there for short integers, just like it is done for bytes already.

https://github.com/IntelPython/numpy/blob/intel/1.17.3/numpy/core/src/npysort/selection.c.src#L258

The issue will be brought up to the compiler's team attention. I will report here once the binary of numpy 1.17.3 with the fix is published.

Sorry for the inconvenience,
Oleksandr

Reply