Python3 bug: numpy.percentile()

Stephan_U — Fri, 01 Nov 2019 14:29:51 GMT

We work with camera images from CMOS sensors which often use 16Bit image formats (int16, uint16). For some image sizes the implementation of percentile() causes memory corruptions/segfaults. This not observed for other data types (8, 32 or 64Bit) and is present at least in numpy version 1.13 ... 1.17 (releases 2018-1 through 2019-5). Other python distributions than intel behave normal.

We could boil down the problem to the attached 3-liner 'minimal-iP.txt':

import numpy as np
test = np.array([np.arange(i, 44+i, dtype=np.int16) for i in range(56)])
np.percentile(test, (0.1, 0.99))

$: python3 minimal-iP.txt

The resulting crash log is also attached. Other image sizes might work, (43x47) is also known to trigger the crash.

With best regards,

Stephan

Hi Stephan,

Oleksandr_P_Intel — Tue, 05 Nov 2019 16:26:46 GMT

Hi Stephan,

Thank you for providing a clean reproducer. I was able to reproduce the crash. The crash is due to aggressive compiler optimization of {{dumb_select_short}} routine in numpy/core/src/npymath/selection.c.src, which implements quadratic complexity selection.

The fix is suppress vectorization there for short integers, just like it is done for bytes already.

https://github.com/IntelPython/numpy/blob/intel/1.17.3/numpy/core/src/npysort/selection.c.src#L258

The issue will be brought up to the compiler's team attention. I will report here once the binary of numpy 1.17.3 with the fix is published.

Sorry for the inconvenience,
Oleksandr

topic Python3 bug: numpy.percentile() in Intel® Distribution for Python*

Python3 bug: numpy.percentile()

Hi Stephan,