Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
117 Views

Building Python

Hi everyone,
I just wanted to ask if anyone's got any expercience building Python with icc / icpc?
I'm gonna try and post if I run into trouble of any kind or what I had to do to get it to work.
I'll try Python 3.1.1 with computed gotos ;) Btw, I'm running Ubuntu 9.10 amd64 on a Core2Duo T7200 notebook.
Cheers,
nuku
0 Kudos
36 Replies
Highlighted
New Contributor I
18 Views

Hi
You can see that with xiar given by Brandon Hewitt and fPIC required for option -fast the build is ended correctly but with some faults.
If i want build python without fault with Icc , i can made without help.
I have perfectly understand all faults are given and how to make, just no the times for made several step test compile for that i not having to use.
If you not understand my bad control explain expression language , is not important.
Bests regards.
0 Kudos
Highlighted
Employee
18 Views

OK, I finally got a chance to spend some time on this. In addition to the need to set AR=xiar as already pointed out in this thread, there's also the problem with __int128_t with a workaround described in http://software.intel.com/en-us/forums/showthread.php?t=56652&o=a&s=lr.

There is one other issue related to -ipo that I found. Essentially, when you use -ipo some functions get inlined everywhere they are called in the 'python' executable, so the compiler is removing the original body. Unfortunately some of the shared libs depend on some of these functions, causing problems with a lot of loadable modules. I don't have a workaround for this yet, so for now you should avoid -ipo, but I'll work with the developers to get thsi fixed.

Thanks!

Dale
0 Kudos
Highlighted
Beginner
18 Views

I'm sorry, I was quite busy lately (found a small old robot of mine based on an ATmega8^^), but I'll try AR=xiar later today.
I already found the sulution to the problem with __int128_t in a small tutorial on how to build Firefox (which also uses libffi) with icc. Anyway, even if there was no fix, it wouldn't be so tragic, as it only affects the curses module.
As for ipo - so the compiler inlines functions which are referenced by modules - did I understand this correctly?
Thanks a lot!
Lorenz
0 Kudos
Highlighted
Employee
18 Views

>As for ipo - so the compiler inlines functions which are referenced by modules - did I understand this correctly?

Yes, essentially. The problem comes about not from the inlining itself, but from the fact after inlining everywhere in the "python" executable, it sees no calls to the function and therefore eliminates it. The use of the -export-dynamic switch at link time is supposed to prevent this. When it's fixed, it should still do the inlining but leave the symbol intact and dynamically visible.

I'll post here when a fix for that is available, in the meantime you should be able to build without -ipo. If you want to use "-fast" without "-ipo", see "icc -help" for the definition of "-fast" and you can use those options explicitly for now, leaving out "-ipo".

Dale
0 Kudos
Highlighted
Beginner
18 Views

OK, thank you!

And well, I need precise floating point numbers for numerical applications, so I'll have to live with the loss of optimizations by -fp-model precise and -fp-speculation=safe... - You can't have everything, right? (Pity, though :D)
Now I tried the following:
FLAGS="-O3 -mssse3 -no-prec-div -static -xSSSE3 -opt-multi-version-aggressive -vec-guard-write -opt-malloc-options=1 -opt-calloc -mkl -openmp -fp-model precise -fp-speculation=safe -mp1"
./configure --with-computed-gotos --without-gcc CC=icc CXX=icpc CFLAGS=$FLAGS AR=xiar
make CC=icc CXX=icpc CFLAGS=$FLAGS AR=xiar
I got this:http://bpaste.net/show/5035/(click "raw" for the actual output).

Any idea what went wrong here?
I played around with omitting several flags, and suddenly it started telling me that my c compiler was broken (couldn't find a shared library), so I restarted with a clean source tarball (I got the recent version, 3.1.2, instead of 3.1.1) and it stopped complaining. It didn't change anything about the above error, thought. Stuck again...
0 Kudos
Highlighted
Employee
18 Views

Looking at the raw output, the problem seems to be an undefined reference to __kmpc_begin, which is probably because you built objects with -openmp but didn't link with it.

icc -DNDEBUG -g  -O3 -Wall -Wstrict-prototypes   Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lresolv -ldl  -lutil -o Parser/pgen
Parser/pgenmain.o: In function `main':
Parser/pgenmain.c:(.text+0x2c): undefined reference to `__kmpc_begin'
make: *** [Parser/pgen] Error 1

Did you add openmp code to python? I don't see any omp pragmas doing a quick search myself, so I don't think there's any point to having -openmp on the command line.

What worked for me was just setting CFLAGS to "-xHOST -O3 -no-prec-div -static", but as you said you may need to set fp-model appropriately for your needs. If I were you I'd start with -O2 or -O3 and work up from that, rather than having a long list of options that you think you might need. Then with each change (if it works) measure the effect to see if it matters. If it doesn't work, please let us know so we can investigate it. If it does work and it makes a difference, that would also be useful information.

I must confess I don't know exactly how the fp-model setting would affect calculations in python. Are floating point calculations simply passed to C functions? If so then I can see where it could affect the results you get when running a python program that does fp calculation, but it would be good to verify the effects with some fp test suite of python code. If you can illuminate me on that it would be appreciated.

Thanks!
Dale
0 Kudos
Highlighted
Beginner
18 Views

Hi,
uhm the OpenMP thing seems to have somehow slipped in, complete nonsense. Thank you for that ;)
For the fp precision, I ran some included self-tests (make test) and lots of them failed for either fp precision or incorrect handling of NaN.
And I not only forgot linking against OpenMP, but also MKL, so I left that one out for now, too.
Then, the build finished cleanly, though with some fun compiler warnings like this one:
/.../pyicc312/Modules/pyexpat.c(1768): warning #1419: external declaration in primary source file
PyMODINIT_FUNC MODULE_INITFUNC(void); /* avoid compiler warnings */
:D
Flags were:-O3 -mssse3 -no-prec-div -static -xSSSE3 -opt-multi-version-aggressive -vec-guard-write -opt-malloc-options=1 -opt-calloc -fp-model precise -fp-speculation=safe -mp1
I also did a build with only -O3 right in the beginning...
I compared the above build to a gcc build (Py 3.1.1, just -O3 -msse3), and gcc was around 2% faster, but gcc wasn't limited in optimizations in floating point numbers. Details:http://bpaste.net/show/5037/
Fun fact: icc is 7% faster in comparing floats, even if its floating point optimizations are disabled^^
I am currently running the self tests to see how big the difference in fp precision actually is, but these take awfully long...
0 Kudos
Highlighted
Beginner
18 Views

OK, test results are in:
305 tests OK.
4 tests failed:
test_cmath test_ctypes test_math test_memoryview
27 tests skipped:
test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu
test_dbm_ndbm test_kqueue test_nis test_ossaudiodev test_pep277
test_smtpnet test_socketserver test_sqlite test_startfile test_tcl
test_timeout test_tk test_ttk_guionly test_ttk_textonly
test_urllib2net test_urllibnet test_winreg test_winsound
test_xmlrpc_net test_zipfile64
6 skips unexpected on linux2:
test_dbm_ndbm test_ttk_guionly test_tcl test_tk test_ttk_textonly
test_dbm_gnu
I only tested the four failed ones with the gcc compiled python, and all of them were OK.
This is what went wrong:
test_math:
self.assertTrue(math.isnan(math.atan2(0., NAN)))
AssertionError: False is not True
AssertionError: 1.1102230246251565e-16 != 0.0
test_cmath:
self.assertTrue(math.isnan(phase(z)))
AssertionError: False is not True
AssertionError: acos1004: acos(complex(0.0, nan))
Expected: complex(1.5707963267948966, nan)
Received: complex(1.5707963267948966, 0.0)
Received value insufficiently close to expected value.
test_memoryview:
AssertionError: array('i', [97, 98, 97, 98, 97, 102]) != array('i', [97, 98, 97, 98, 99, 102])
AssertionError: array('i', [97, 98, 97, 98, 97, 102]) != array('i', [97, 98, 97, 98, 99, 102])
AssertionError: array('i', [97, 98, 97, 98, 97, 102]) != array('i', [97, 98, 97, 98, 99, 102])
AssertionError: bytearray(b'ababaf') != bytearray(b'ababcf')
AssertionError: bytearray(b'ababaf') != bytearray(b'ababcf')
AssertionError: bytearray(b'ababaf') != bytearray(b'ababcf')
test_ctypes:
AssertionError: 15.0 != 21
AssertionError: 15 != 21
AssertionError: 133 != 139
AssertionError: 0.333333333333 not less than 0.01
In test_cmath, z is one of the following (for z in complex nans:):
complex_nans = [complex(x, y) for x, y in [
(NAN, -INF),
(NAN, -2.3),
(NAN, -0.0),
(NAN, 0.0),
(NAN, 2.3),
(NAN, INF),
(-INF, NAN),
(-2.3, NAN),
(-0.0, NAN),
(0.0, NAN),
(2.3, NAN),
(INF, NAN)
]]
I have no idea as to what the hell went wrong, but obviously something did ;)
Cheers
0 Kudos
Highlighted
Employee
18 Views

OK, I'll try running "make test" and see what I find. fp comparisons can be a real pain.

Thanks!

Dale
0 Kudos
Highlighted
Employee
18 Views

Yeah, I see some fails in my icc-built python. Let me investigate and get back to you.

Dale
0 Kudos
Highlighted
Employee
18 Views

First problem, the patch I used for ffi64.c (which I got from http://software.intel.com/en-us/articles/build-firefox-35-with-intel-c-compiler/) has a bug, where it does the post increment of ssecount twice instead of once. I don't know if you have the same problem or not, but you might want to check that. I still have other failures, so I'll continue investigating.

Dale

0 Kudos
Highlighted
Employee
18 Views

BTW, Feilong has fixed the problem on http://software.intel.com/en-us/articles/build-firefox-35-with-intel-c-compiler/, so if you look there for the problem, you might be confused as to what I'm talking about. All better now.

Dale
0 Kudos
Highlighted
Beginner
18 Views

Hm yeah no I used the old version. Fixed it and ctypes-tests are no longer going wrong ;) The other ones still aren't fixed, naturally. Thanks for keeping at it!
0 Kudos
Highlighted
Employee
18 Views

Maybe I'm missing something, but it seems to me there's a problem with the python build. Regardless of whether I set CFLAGS in the environment or on the makefile line (or both) it doesn't get passed to setup.py properly, so when I set "-fp-model precise" the modules still get built without it. If I run setup.py by hand and set CFLAGS in the environment, then it seems to work, though it still adds it's own (e.g. if I set "-O0", it calls "icc -O3 -O0").

Does anyone else know if maybe there's some other way I should be setting CFLAGS for the module buillds?

Thanks!

Dale
0 Kudos
Highlighted
Beginner
18 Views

I just asked a friend who's pretty well into Python and this is what he came up with:
"See the README. Set OPT to influence the optimization flags; set EXTRA_CFLAGS otherwise.", according to a guy called Martin von Lwis, he told me.
In Misc/HISTORY it says on line threethousandfourhundredandsomething, under "What's new in Python 2.5 Alpha 1":
- EXTRA_CFLAGS has been introduced as an environment variable to hold compiler
flags that change binary compatibility. Changes were also made to
distutils.sysconfig to also use the environment variable when used during
compilation of the interpreter and of C extensions through distutils.
And some three thousand lines further on:
- On systems which build using the configure script, compiler flags which
used to be lumped together using the OPT flag have been split into two
groups, OPT and BASECFLAGS. OPT is meant to carry just optimization- and
debug-related flags like "-g" and "-O3". BASECFLAGS is meant to carry
compiler flags that are required to get a clean compile. On some
platforms (many Linux flavors in particular) BASECFLAGS will be empty by
default. On others, such as Mac OS X and SCO, it will contain required
flags. This change allows people building Python to override OPT without
fear of clobbering compiler flags which are required to get a clean build.
So looks like there is quite some tidiness in those makefiles, though it's a bit hidden...
0 Kudos
Highlighted
Beginner
18 Views

Nuku, I'm no expert on computation but I've tried every possible combination that comes to my mind in order to build Python with Intel compilers in Ubuntu 10.10. Here's what I've tried:

./configure --with-gcc=icc--with-cxx-main=icc\

--prefix=/home/user/python\

OPT="-O3 -xHost"

./configure --with-gcc=icc--with-cxx-main=icpc\

--prefix=/home/user/python\

OPT="-O3 -xHost"

./configure --with-computed-gotos --without-gcc CC=icc CXX=icpc CCFLAGS="-O3"\

--prefix=/home/user/python

After configuring the installation I type make install and everything goes ok, at least no error messages appear. Nonetheless I still cannot get Python to be built with the Intel compiler.... here's what I get when I run the executable:

Python 2.6.2 (r262:71600, May 27 2011, 11:50:43)

[GCC 4.4.5] on linux2

When I should be getting something like this:

Python 2.6.2 (r262:71600, Jun 30 2010, 19:36:03)

[GCC Intel C++ gcc 4.1 mode] on linux2

Do you have any ideas about what I'm doing wrong. Thanks in advance,

Csar.

0 Kudos
Highlighted
18 Views

Complete Python Programming For Beginners Step-by-Step – OOPS, Loops, Functions, Python Libraries, Exceptions & More!

0 Kudos