ICC compiler bug building Python sha3

Ed1 · ‎06-19-2018

Hi,

When trying to build Python 3.6.3 (and 3.6.4) using icc 2018.1 and .2 the compiler segfaults.

It is essentially a problem with icc blowing up with a very long line of code generated by a macro.

building '_sha3' extension
icc -pthread -fPIC -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3
-Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-parameter
-Wno-missing-field-initializers -fp-model strict -I./Include -I.
-I/usr/local/include
-I/derp/Python-3.6.4/Include
-I/derp/Python-3.6.4 -c
/derp/Python-3.6.4/Modules/_sha3/
sha3module.c -o
build/temp.linux-x86_64-3.6/derp/Python-3.6.4/Modules/_sha3/sha3module.o
": internal error: ** The compiler has encountered an unexpected problem.
** Segmentation violation signal raised. **
Access violation or stack overflow. Please contact Intel Support for assistance.

Full details of the problem are here: https://bugs.python.org/issue33174

My hacky solution to get it to build was to expand the macro into multiple lines, but even then it will only build with "-O0".

Ed1 · ‎06-21-2018

Updates from the Python bug report:

msg320100 - (view)

	Author: Kenneth Hoste (boegel)	Date: 2018-06-20 17:47
After a bit of back and forth, we have figured out the underlying cause for the problem with the Intel compiler: if the stack limit is not set to a specific value (see output of "ulimit -s"), the Intel compilers fail with a segfault or Interal Compiler Error (ICE). But if the stack limit is set to 'unlimited' (e.g. via "ulimit -s unlimited"), then the compilation works fine... See also https://github.com/easybuilders/easybuild-easyconfigs/issues/6484 and our (optional) workaround in https://github.com/easybuilders/easybuild-easyblocks/pull/1441
msg320104 - (view)	Author: Ned Deily (ned.deily) *	Date: 2018-06-20 18:52
Thanks for all the investigate work and thanks for the finding what appears to be an easy workaround, @boegel. Clearly, though, this is not a Python problem and, with an external workaround (e.g. increasing the process stack size via ulimit), there is no need for a workaround in Python itself so I'm closing this issue. Suggest making sure that the workaround is brought to the Intel compiler team's attention.
msg320106 - (view)	Author: Kenneth Hoste (boegel)	Date: 2018-06-20 19:01
Fully agreed that this is not something that should be fixed in the Python code base. Moreover, the problem has been fixed in Intel compilers 2018 update 3 as far as I can tell (tested with both Python 3.6.4 and 3.7.0rc1). I can't reproduce the problem anymore with that version of the Intel compilers, even after setting a hard stack limit (e.g. "ulimit -s 10240"), while it was easy to reproduce with the 2018 update 1 version.