Intel® Distribution for Python*
Engage in discussions with community peers related to Python* applications and core computational packages.
428 Discussions

Theano on Intel Python (l_python3_b_2018.0.005.tgz)

sergio_r_
Novice
1,362 Views

 

Working under linux, Ubuntu 14.04 , I installing theano according to [ https://software.intel.com/en-us/articles/getting-started-with-intel-optimized-theano ]  I then created the ~/.theanorc with the following lines:

$ more ~/.theanorc
[global]
device = cpu
floatX = float32
cxx = icpc
mode = FAST_RUN
openmp = True
openmp_elemwise_minsize = 10
[gcc]
cxxflags = -qopenmp -march=native -O3 -fno-alias -qopt-prefetch=2 -fp-trap=none
[blas]
ldflags = -lmkl_rt

1.- For GPU computing, is it possible to enable the graphic card

$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Inte
grated Graphics Controller (rev 06)

?

2.- What this error might signify?

Testing theano via "theano-nose --theano  -v", the test ends with:

...
...
theano.tensor.nnet.tests.test_abstract_conv.TestAbstractConvNoOptim.test_all((8, 1, 6, 6), (5, 1, 2, 2), [1, 1], (0, 0), True, False) ... ok
Test 1D kernels used in bilinear upsampling ... ok
Test 2D kernels used in bilinear upsampling ... ok
Test bilinear upsampling using 1D kernels ... Error in file [/home/.theano/compiledir_Linux-3.13--generic-x86_64-with-debian-jessie-sid-x86_64-3.5.2-64/tmpuygyxl3h/mod.cpp:750], err code (-1)Wed Apr 26 21:03:45 VET 2017

Regards,

Sergio
Enhance your #MachineLearning and #BigData skills via #Python #SciPy
1) https://www.packtpub.com/big-data-and-business-intelligence/numerical-and-scientific-computing-scipy-video
2) https://www.packtpub.com/big-data-and-business-intelligence/learning-scipy-numerical-and-scientific-computing-second-edition

0 Kudos
8 Replies
Christophe_H_Intel2
1,362 Views

Hi Sergio,

For your second question, I would suggest using the Theano that comes with Intel Python.  As of 2017 U2 (including 2018 Beta), Intel Theano is bundled with Intel Python on Linux. It is no longer necessary to build your own Theano using the instructions you linked. Hopefully that will fix the test failure you are seeing.

I will ask the Intel Theano team to assist you with your first question.

Thanks,

Chris

0 Kudos
sergio_r_
Novice
1,362 Views

Hi Chris,

Thanks for reading an answering.

Following your advise, I unistalled-reinstalled l_python3_b_2018.0.005. Soon after that
I executed in a terminal the test "theano-nose --theano  -v".
This time, after reaching the "test_mpi.test_mpi_roundtrip"
the system just stayed there for more than an hour (apparently
doing nothing as the typical noise from the cooler fan of the laptop went down)

...
Regression test for a bug where output was borrowed by mistake. ... ok
theano.tensor.tests.test_misc.test_deepcopied_type_filter ... ok
theano.tensor.tests.test_mpi.test_recv ... ok
theano.tensor.tests.test_mpi.test_send ... ok
theano.tensor.tests.test_mpi.test_can_make_function ... ok
theano.tensor.tests.test_mpi.test_mpi_roundtrip ...

I just hit ctrl-C to stop it. Do you know,  Chris, whether is this a normal behaviour? Is there a way to skip it?

Regards,

Sergio
Enhance your #MachineLearning and #BigData skills via #Python #SciPy
1) https://www.packtpub.com/big-data-and-business-intelligence/numerical-and-scientific-computing-scipy-video
2) https://www.packtpub.com/big-data-and-business-intelligence/learning-scipy-numerical-and-scientific-computing-second-edition

 

 

0 Kudos
Peng_Z_Intel
Employee
1,362 Views

Hi Serigio,

Thanks for using Intel Optimized Theano.

Regarding Q1, Theano takes CUDA and cuDNN as default GPU backend now. Unfortunately, it can not work in non-NVIDIA GPUs.

If you want to work on GEN, I think you can implement OPs with OpenCL API.

https://software.intel.com/en-us/intel-opencl

Thanks

--Patric

 

 

0 Kudos
Christophe_H_Intel2
1,362 Views

Hi Sergio,

Thanks for bringing this to our attention.

The test in question relies on mpiexec to run a simple MPI program:

p = subprocess.Popen("mpiexec -np 2 python " + theano_root +
                         "tensor/tests/_test_mpi_roundtrip.py",
                         stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE,
                         shell=True,
                         close_fds=True)

However, the Intel MPI bundled with Intel Python uses mpirun (a wrapper around mpiexec.hydra) instead of mpiexec to launch MPI programs. I assume the test is finding an MPI installation outside of Intel Python.  What does `which mpiexec` output?

I will work on a fix for this, but in the meantime, you can either change "mpiexec" to "mpirun" in <intel_python_root>/lib/python3.5/site-packages/theano/tensor/tests/test_mpi.py:42, or skip the test by changing the first line in that function to "if True:".

Chris

0 Kudos
sergio_r_
Novice
1,362 Views

Hi Chris,

 Here are the outputs of some commands:

$ which mpiexec
/home/intel2017/compilers_and_libraries_2018.0.061/linux/mpi/intel64/bin/mpiexec

$ which mpirun
/home/intel/intelpython3/bin/mpirun

$ which mpiicc
/home/intel2017/compilers_and_libraries_2018.0.061/linux/mpi/intel64/bin/mpiicc

First option:
In file /home/intel/intelpython3/lib/python3.5/site-packages/theano/tensor/tests/test_mpi.py
Changing the line (42) that reads
  p = subprocess.Popen("mpiexec -np 2 python " + theano_root +
to read:
  p = subprocess.Popen("mpirun -np 2 python " + theano_root +

The result is the same already reported: that is,
after reaching the "test_mpi.test_mpi_roundtrip"
the system hangs there.

Second option SKIPPING the test:
In file /home/intel/intelpython3/lib/python3.5/site-packages/theano/tensor/tests/test_mpi.py

Changing the line (39) that reads
 if not mpi_enabled:
to read
 if True:

and executing that particular test in the form:

theano-nose -sv /home/intel/intelpython3/lib/python3.5/site-packages/theano/tensor/tests/test_mpi.py

===============================
Problem occurred during compilation with the command line below:
icpc -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -qopenmp -march=native -O3 -fno-alias -qopt-prefetch=2 -fp-trap=none -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/home/intel/intelpython3/lib/python3.5/site-packages/numpy/core/include -I/home/intel/intelpython3/include/python3.5m -I/home/intel/intelpython3/lib/python3.5/site-packages/theano/gof -L/home/intel/intelpython3/lib -fvisibility=hidden -o /home/.theano/compiledir_Linux-3.13--generic-x86_64-with-debian-jessie-sid-x86_64-3.5.2-64/tmpirvcdp0u/m698f212fff64c1daf999e0b323a6632c.so /home/.theano/compiledir_Linux-3.13--generic-x86_64-with-debian-jessie-sid-x86_64-3.5.2-64/tmpirvcdp0u/mod.cpp -lmkl_rt -lpython3.5m

icpc: command line warning #10006: ignoring unknown option '-Wno-unused-label'
/home/.theano/compiledir_Linux-3.13--generic-x86_64-with-debian-jessie-sid-x86_64-3.5.2-64/tmpirvcdp0u/mod.cpp(520): error: identifier "PyInt_FromLong" is undefined
  V1 = PyInt_FromLong(atoi(v.Build));
       ^

compilation aborted for /home/.theano/compiledir_Linux-3.13--generic-x86_64-with-debian-jessie-sid-x86_64-3.5.2-64/tmpirvcdp0u/mod.cpp (code 2)

theano.tensor.tests.test_mpi.test_recv ... ok
theano.tensor.tests.test_mpi.test_send ... ok
theano.tensor.tests.test_mpi.test_can_make_function ... ok
theano.tensor.tests.test_mpi.test_mpi_roundtrip ... ok
theano.tensor.tests.test_mpi.test_mpi_send_wait_cmp ... ok
theano.tensor.tests.test_mpi.test_mpi_tag_ordering ... ok
theano.tensor.tests.test_mpi.test_mpi_schedule ... ok

----------------------------------------------------------------------
Ran 7 tests in 1.206s

OK


Next time I'll report the full test resulting result (it takes a while to finish).

Regards,

Sergio
Enhance your #MachineLearning and #BigData skills via #Python #SciPy
1) https://www.packtpub.com/big-data-and-business-intelligence/numerical-and-scientific-computing-scipy-video
2) https://www.packtpub.com/big-data-and-business-intelligence/learning-scipy-numerical-and-scientific-computing-second-edition

 

 

0 Kudos
sergio_r_
Novice
1,362 Views

Hi Chris,

 Here is how the test ended skipping the
theano.tensor.tests.test_mpi.test_mpi_roundtrip:

======================================================================
FAIL: test_local_log_erfc (theano.tensor.tests.test_opt.T_local_erfc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/intel/intelpython3/lib/python3.5/site-packages/theano/tensor/tests/test_opt.py", line 4899, in test_local_log_erfc
    assert all(numpy.isfinite(f(val)))
AssertionError

----------------------------------------------------------------------
Ran 3895 tests in 7420.868s

FAILED (SKIP=368, failures=1)

Sergio
Enhance your #MachineLearning and #BigData skills via #Python #SciPy
1) https://www.packtpub.com/big-data-and-business-intelligence/numerical-and-scientific-computing-scipy-video
2) https://www.packtpub.com/big-data-and-business-intelligence/learning-scipy-numerical-and-scientific-computing-second-edition

 

 

0 Kudos
sergio_r_
Novice
1,362 Views

By the way Chris,

 In case this could help, the "test_mpi_roundtrip" is passed using a python Anaconda distribution without any changes in the file "test_mpi.py" located in this set up at /home/myProg/Anaconda341/lib/python3.5/site-packages/theano/tensor/tests/test_mpi.py

Here is what I did:

$ which python
/home/myProg/Anaconda341/bin/python

$ which mpirun
/home/myProg/Anaconda341/bin/mpirun

$ which mpiexec
/home/myProg/Anaconda341/bin/mpiexec

$ which mpicc
/home/myProg/Anaconda341/bin/mpicc

$ more ~/.theanorc
[global]
device = cpu
floatX = float32
cxx = g++
mode = FAST_RUN
openmp = True
openmp_elemwise_minsize = 10
[gcc]
cxxflags = -fopenmp -lpthread -march=native -O3

theano-nose -sv /home/myProg/Anaconda341/lib/python3.5/site-packages/theano/tensor/tests/test_mpi.py

theano.tensor.tests.test_mpi.test_recv ... ok
theano.tensor.tests.test_mpi.test_send ... ok
theano.tensor.tests.test_mpi.test_can_make_function ... WARNING (theano.gof.cmodule): WARNING: your Theano flags `gcc.cxxflags` specify an `-march=X` flags.
         It is better to let Theano/g++ find it automatically, but we don't do it now
ok
theano.tensor.tests.test_mpi.test_mpi_roundtrip ... ok
theano.tensor.tests.test_mpi.test_mpi_send_wait_cmp ... ok
theano.tensor.tests.test_mpi.test_mpi_tag_ordering ... ok
theano.tensor.tests.test_mpi.test_mpi_schedule ... ok

----------------------------------------------------------------------
Ran 7 tests in 15.170s

OK

Let me mention that the execution of "theano.tensor.tests.test_mpi.test_mpi_tag_ordering" last a couple of seconds (meaning that it takes a short while before reporting "ok").

Regards,

Sergio
Enhance your #MachineLearning and #BigData skills via #Python #SciPy
1) https://www.packtpub.com/big-data-and-business-intelligence/numerical-and-scientific-computing-scipy-video
2) https://www.packtpub.com/big-data-and-business-intelligence/learning-scipy-numerical-and-scientific-computing-second-edition

 

0 Kudos
Christophe_H_Intel2
1,362 Views

Hi Sergio,

Thanks for your detailed report.

The error "identifier "PyInt_FromLong" is undefined" has been addressed and the fix will be available in 2017 U3 and 2018 Beta U1.  I will look into the MPI problem and report back my findings. 

Chris

0 Kudos
Reply