Python/C API

[rudi-gaelzer]

I assume that the Python/C API that is available for the gcc suite also works for intel python, correct?

If I'm correct, does it work only for C sources compiled with icc?

I have the following test code:

#include <Python.h>
#include <stdio.h>

void main ()
{
   PyObject *sys, *version;
   char *ver;
   
   Py_Initialize();

   sys = PyImport_ImportModule("sys");
   version = PyObject_GetAttrString(sys, "version");

   ver= PyString_AsString(version);
   printf(ver);
   printf("\n\n");
}

That simply prints the version of the python interpreter.

I tried to compile the code with gcc with:

gcc  -I/usr/local/intel/intelpython2/include/python2.7 call_py_print_version.c -L/usr/local/intel/intelpython2/pkgs/python-2.7.13-intel_1/lib/python2.7/config -lpython2.7

and got a ton or error messages.

It goes without saying that compiling with the gcc libraries (at /usr/include/python2.7) works.

Does that mean that I cannot use gcc?  Or is it something else?

Thanks.

[Rohit_J_Intel]

Hi Rudi,

We build CPython with gcc. I was able to build call_py_print_version.c with gcc using the following:

gcc call_py_print_version.c `<IDP_installation>/bin/python-config --includes` `<IDP_installation>/bin/python-config --libs` -o call_py_print_version 

Thanks,
Rohit

[rudi-gaelzer]

Rohit J. (Intel) wrote:

Hi Rudi,

We build CPython with gcc. I was able to build call_py_print_version.c with gcc using the following:

gcc call_py_print_version.c `<IDP_installation>/bin/python-config --includes` `<IDP_installation>/bin/python-config --libs` -o call_py_print_version 

Thanks,
Rohit

Dear Rohit, as per your suggestion, I tried the compilation line

gcc call_py_print_version.c `/usr/local/intel/intelpython2/bin/python-config --includes` `/usr/local/intel/intelpython2/bin/python-config --libs`

The code compiled and ran successfully, but the result was unexpected.

This is what I got:

2.7.13 (default, Jun 26 2017, 10:20:05) 
[GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]

Which is what I get from the GNU python installation.

OTOH, opening a python terminal with the intel python:

$> usr/local/intel/intelpython2/bin/python
Python 2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import sys
>>> sys.version
'2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46) \n[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]'

So, from the C executable, I expected the result

2.7.13 |Intel Corporation| (default, Apr 27 2017, 15:33:46)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

Does this make any sense?

[Rohit_J_Intel]

Rudi,

There's a difference in outputs when you build "call_py_print_version" (with libraries and header-files from IDP installation and GNU Python installation v/s executing <IDP_installation>/bin/python -c "from sys import version; print(version)" for two reasons:

1. The values reported by `sys.version` are computed at build-time of CPython - the output implies that gcc-4.8.2 was used for building it. I'm assuming that when you're building "call_py_print_version", you're using gcc-7.1.1.
2. The output of Py_GetVersion() is different, "sys.version" reports an extra "Intel Corporation" string in the output because we patched sources of CPython to have this persisted in the output of "sys.version" - it was an add-on feature that could help differentiate IDP's python executable at runtime with any other Python installation, should the need be.

Thanks,
Rohit

[rudi-gaelzer]

Rohit J. (Intel) wrote:

There's a difference in outputs when you build "call_py_print_version" (with libraries and header-files from IDP installation and GNU Python installation v/s executing <IDP_installation>/bin/python -c "from sys import version; print(version)" for two reasons:

1. The values reported by `sys.version` are computed at build-time of CPython - the output implies that gcc-4.8.2 was used for building it. I'm assuming that when you're building "call_py_print_version", you're using gcc-7.1.1.

Indeed, using the interpreter with a command line (with python -c) gives the information about the specific python build (gnu or intel).  However, I must confess I did not fully understand why the executable gives always the same info, even when compiled against intel python libs.

Rohit J. (Intel) wrote:

The output of Py_GetVersion() is different, "sys.version" reports an extra "Intel Corporation" string in the output because we patched sources of CPython to have this persisted in the output of "sys.version" - it was an add-on feature that could help differentiate IDP's python executable at runtime with any other Python installation, should the need be.

Py_GetVersion() also gives always the same information :

[GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]

Regardless of which version I use in the compilation.

Anyway, perhaps I should explain the real reason behind this thread.

I wrote some C functions that access special functions supported by the mpmath libray.  The functions work all right, but I'd like to benchmark the executables build with GNU python against intel python, since the latter is supposed to be faster.

So, the bottomline is: If I compile the said functions with your suggested line

gcc something.c `<IDP_Installation>/bin/python-config --includes` `<IDP_Installation>/bin/python-config --libs`

can I be assured that the executable will access intel's libraries?

[Rohit_J_Intel]

Rudi,

However, I must confess I did not fully understand why the executable gives always the same info, even when compiled against intel python libs.

Some background: the "version" information comes from libpython2.7.so (definitions of core Python's C-API functions are baked in libpython2.7.so, and the value returned by Py_GetVersion() is stored in a static variable which is initialized at build-time of CPython : https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Python/getversion.c :: the variable is "version").

I was able to reproduce this behavior. The problem stems from the fact that you might have another Python on your PATH. At the execution time, the loader goes through its default search-path trying to locate libpython2.7.so, and subsequently "Py_GetVersion" symbol within the library, it's able to find the library stored at system-path before it gets to <IDP_installation>/lib directory. Ideally, Py_GetVersion() should have been invoked from libpython2.7.so stored at <IDP_installation>/lib, which would have given the expected value.  There are two ways to fix this :
1. Ensuring that <IDP_installation>/lib is on top of loader's search-path before invoking the executable :

if [ "$LD_LIBRARY_PATH" != "" ]; then
  export OLD_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
fi
export LD_LIBRARY_PATH=`<IDP_installation>/bin/python-config --exec-prefix`/lib:$LD_LIBRARY_PATH
./call_py_print_version
if [ -z "$OLD_LD_LIBRARY_PATH" ]; then
  unset LD_LIBRARY_PATH
else
  export LD_LIBRARY_PATH=$OLD_LD_LIBRARY_PATH
  unset OLD_LD_LIBRARY_PATH
fi

2. Ensuring that you've set RPATH correctly while building "call_py_print_version" executable.

When sys.version is invoked from Python interpreter, the runtime ensures that libraries stored at <IDP_installation>/lib are used before looking into those available at system-level, hence the expected output.
 
Anyhow, as you intend to build a custom C-extension that's to be used for comparison with IDP and GNU Python, you'd have to build the respective libraries, import it into the interpreter and run your benches from there (unless you set the RPATH correctly for those custom libraries at build-time, you might have to set LD_LIBRRAY_PATH explicitly, at execution time). We have optimization for Numpy, Scipy and Scikit-learn, we haven't optimized mpmath. If your code implicitly uses any of the modules that we've optimized, you should see performance benefits.

Thanks,
Rohit