Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Ryo_A_
Beginner
209 Views

Intel Theano installation for Intel Python

Hello,

First, I am not entirely certain if this is the correct forum for this, so I apologize in advance if this post is out of place.

We are trying to install the Intel-optimized Theano on Intel Python with Intel C++ Compiler support. We were following the instructions in the PDF on the github page for Theano: https://github.com/intel/theano/blob/master/Install_Guide.pdf

And we were able to follow it without any issues until the last step. However when we try to run `python -c "import theano"` we get the following error:

Problem occurred during compilation with the command line below:
icpc -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -qopenmp -march=native -O3 -qopt-report=3 -fno-alias -qopt-prefetch=2 -fp-trap=none -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/opt/intel/intelpython35/lib/python3.5/site-packages/numpy/core/include -I/opt/intel/intelpython35/include/python3.5m -I/opt/intel/intelpython35/lib/python3.5/site-packages/Theano-0.9.0.dev2-py3.5.egg/theano/gof -L/opt/intel/intelpython35/lib -fvisibility=hidden -o /home/user/.theano/compiledir_Linux-3.10-el7.xppsl_1.4.3.3482.x86_64-x86_64-with-centos-7.2.1511-Core-x86_64-3.5.2-64/tmpm_kkdy41/m698f212fff64c1daf999e0b323a6632c.so /home/user/.theano/compiledir_Linux-3.10-el7.xppsl_1.4.3.3482.x86_64-x86_64-with-centos-7.2.1511-Core-x86_64-3.5.2-64/tmpm_kkdy41/mod.cpp -lmkl_rt -lpython3.5m
icpc: command line warning #10006: ignoring unknown option '-Wno-unused-label'
icpc: remark #10397: optimization reports are generated in *.optrpt files in the output location
/home/user/.theano/compiledir_Linux-3.10-el7.xppsl_1.4.3.3482.x86_64-x86_64-with-centos-7.2.1511-Core-x86_64-3.5.2-64/tmpm_kkdy41/mod.cpp(520): error: identifier "PyInt_FromLong" is undefined
  V1 = PyInt_FromLong(atoi(v.Build));
       ^

Here are the versions of Intel Python and libraries:
[user@localhost ~]$ python -c "import sys; print(sys.version)"
3.5.2 |Intel Corporation| (default, Oct 20 2016, 03:10:33)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

[user@localhost ~]$ python -c "import numpy; print(numpy.__version__)"
1.11.2

[user@localhost ~]$ python -c "import scipy; print(scipy.__version__)"
0.18.1

[user@localhost ~]$ python -c "import theano; print(theano.__version__)"
// ... the above error ... //
0.9.0dev2.dev-07abbd2f4202021c81eb5002b080f72a81469e6c

And the Intel compiler version:
[user@localhost ~]$ icc -v
icc version 17.0.1 (gcc version 4.8.5 compatibility)

The physical system is Intel Xeon Phi processor (KNL). The OS is CentOS 7.2 with XPPS 1.4.3 patch.

Any suggestions will be very much appreciated.

Thanks!
Ryo Asai

0 Kudos
7 Replies
209 Views

Hello Ryo,

PyInt_FromLong in python 3 is supposed to be aliased to PyLong_FromLong.  Since you're seeing this error, it makes me think that a python 2 header is being included somewhere, or Theano thinks you're using python 2.  However, the compile command clearly includes the correct python 3 headers, and you're clearly using python 3.  I would suggest running `theano-cache clear` and removing the compile directory, ~/.theano, then trying to import theano again.  You might also check the PYTHONPATH and PYTHONHOME evnironment variables.  They should be empty.

Thanks,

Chris

 

Ryo_A_
Beginner
209 Views

Hello Chris,

Thank you for your fast response. We found that there was an environment variable, INTEL_PYTHONHOME but unsetting it (and removing .theano and clearing cache) did not help. I am getting the exact same errors.

We did make some progress, and tracked the issue down to some environment variable set by the executable "compilervars.sh" that comes with Parallel Studio. When we do not have this script sourced, we were able to run our theano application with gcc + mkl (by copying "theanorc_gcc_mkl" file to "~/.theanorc", verified that it' using MKL with MKL_VERBOSE=1). But as soon as we source "compilervars.sh" we get the same errors again.

We captured the output of "env" with and without sourcing compilervars.sh and did a "diff" but we can't seem to track down what could be causing the issue.

0a1,2
> MKLROOT=/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl
> MANPATH=/opt/intel/man/common:/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/man:/opt/intel/documentation_2017/en/debugger//gdb-ia/man/:/opt/intel/documentation_2017/en/debugger//gdb-mic/man/:/opt/intel/documentation_2017/en/debugger//gdb-igfx/man/:/usr/local/share/man:/usr/share/man:
2a5,6
> INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.1.132/linux/licenses:/opt/intel/licenses:/root/intel/licenses
> IPPROOT=/opt/intel/compilers_and_libraries_2017.1.132/linux/ipp
5a10
> GDBSERVER_MIC=/opt/intel/debugger_2017/gdb/targets/mic/bin/gdbserver
6a12,13
> LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.1.132/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/daal/../tbb/lib/intel64_lin/gcc4.4
> MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.1.132/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.1.132/linux/tbb/lib/mic
8a16
> LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/tbb/lib/intel64/gcc4.7:/opt/intel/debugger_2017/iga/lib:/opt/intel/debugger_2017/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.1.132/linux/daal/../tbb/lib/intel64_lin/gcc4.4
9a18,20
> MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.1.132/linux/tbb/lib/mic
> CPATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.1.132/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.1.132/linux/daal/include
> NLSPATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64_lin/locale/%l_%t/%N:/opt/intel/debugger_2017/gdb/intel64_mic/share/locale/%l_%t/%N:/opt/intel/debugger_2017/gdb/intel64/share/locale/%l_%t/%N
11c22,23
< PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
---
> PATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/intel64/bin:/opt/intel/debugger_2017/gdb/intel64_mic/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
> TBBROOT=/opt/intel/compilers_and_libraries_2017.1.132/linux/tbb
13a26
> GDB_CROSS=/opt/intel/debugger_2017/gdb/intel64_mic/bin/gdb-mic
14a28,29
> DAALROOT=/opt/intel/compilers_and_libraries_2017.1.132/linux/daal
> MPM_LAUNCHER=/opt/intel/debugger_2017/mpm/mic/bin/start_mpm.sh
16a32
> INTEL_PYTHONHOME=/opt/intel/debugger_2017/python/intel64/
19a36
> CLASSPATH=/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.1.132/linux/daal/lib/daal.jar
21a39
> INFOPATH=/opt/intel/documentation_2017/en/debugger//gdb-ia/info/:/opt/intel/documentation_2017/en/debugger//gdb-mic/info/:/opt/intel/documentation_2017/en/debugger//gdb-igfx/info/
22a41
> I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi

 

So far we have tried unsetting INTEL_PYTHONHOME and PATH but neither helped.

Thanks again for the help!
Ryo

209 Views

Hi Ryo,

I wanted to let you know that Intel Python 2017 Update 2 is now released and contains a prebuilt Theano package. Can you try this new version to see if it solves your problem?  You can create a conda environment with Intel Theano as follows:

$ conda create -n theano_test -c intel --override-channels theano
$ cp <intel_python_install_root>/envs/theano_test/share/config/theano/theanorc_icc_mkl ~/.theanorc
$ . activate theano_test
$ python -c 'import theano'

Or install it into your existing environment:

$ conda install -c intel --override-channels theano

Chris

gaston-hillar
Black Belt
209 Views

Hi Chris,

I couldn't find information about the inclusion of the prebuilt Theano package. As Intel added more prebuilt packages, is there any place in which I can check which are the prebuilt packages added in version 2017 Update 2? I mean, any link with this information?

gaston-hillar
Black Belt
209 Views

Hi Chris,

I found the info. No need to answer my previous question. The following link provides the information I was searching: https://software.intel.com/en-us/articles/intel-distribution-for-python-2017-update-2-readme

 

Sven_M_
Beginner
209 Views

I installed Theano as Chris suggested, but get the error message the icpc can not be found

$ cat ~/.theanorc
[global]
device = cpu
floatX = float32
cxx = icpc
mode = FAST_RUN
openmp = True
openmp_elemwise_minsize = 10
allow_gc = False
[dnn]
enabled = mkl
[gcc]
cxxflags = -qopenmp -march=native -O3  -qopt-report=3 -fno-alias -qopt-prefetch=2 -fp-trap=none
[blas]
ldflags = -lmkl_rt

 

$ python theano_test.py
WARNING (theano.gof.cmodule): WARNING: your Theano flags `gcc.cxxflags` specify an `-march=X` flags.
         It is better to let Theano/g++ find it automatically, but we don't do it now
Problem occurred during compilation with the command line below:
icpc -shared -g -qopenmp -march=native -O3 -qopt-report=3 -fno-alias -qopt-prefetch=2 -fp-trap=none -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/numpy/core/include -I/home/sumeyer/.conda/envs/theano_test/include/python3.5m -I/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof -L/home/sumeyer/.conda/envs/theano_test/lib -fvisibility=hidden -o /home/sumeyer/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-3.5.2-64/lazylinker_ext/lazylinker_ext.so /home/sumeyer/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-3.5.2-64/lazylinker_ext/mod.cpp -lpython3.5m
Traceback (most recent call last):
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof/lazylinker_c.py", line 75, in <module>
    raise ImportError()
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof/lazylinker_c.py", line 92, in <module>
    raise ImportError()
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof/vm.py", line 659, in <module>
    from . import lazylinker_c
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof/lazylinker_c.py", line 127, in <module>
    preargs=args)
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof/cmodule.py", line 2291, in compile_str
    p_out = output_subprocess_Popen(cmd)
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/misc/windows.py", line 77, in output_subprocess_Popen
    p = subprocess_Popen(command, **params)
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/misc/windows.py", line 43, in subprocess_Popen
    proc = subprocess.Popen(command, startupinfo=startupinfo, **params)
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'icpc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "theano_test.py", line 1, in <module>
    from theano import function, config, shared, sandbox
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/__init__.py", line 70, in <module>
    from theano.compile import (
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/compile/__init__.py", line 10, in <module>
    from theano.compile.function_module import *
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/compile/function_module.py", line 21, in <module>
    import theano.compile.mode
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/compile/mode.py", line 10, in <module>
    import theano.gof.vm
  File "/home/sumeyer/.conda/envs/theano_test/lib/python3.5/site-packages/theano/gof/vm.py", line 674, in <module>
    if x.fullname == 'linker'][0].default.startswith('cvm'), e
AssertionError: [Errno 2] No such file or directory: 'icpc'

 

Rohit_J_Intel
Employee
209 Views

icpc is Intel's C++ compiler. From the log that's pasted, it seems that you don't have icpc in PATH. Since, you've mentioned in the .theanorc to use icpc for compiling cxx files, you're seeing these failures. If you don't have icpc, you could use g++ as well, you'd have to change the .theanorc to this : https://github.com/intel/Theano/blob/ver-1.0/theanorc_gcc_mkl

Reply