Environment initialized, no module named torch or tensorflow

[AdamMiltonBarker]

Environment is active, pytorch not found.

It also says Tensorflow is installed and is the latest, this is not true, Tensorflow is not found either, if I look at the conda envs it shows Tensorflow 2.3.0, not the latest.

 

[AdamMiltonBarker]

More issues, cannot install packages in conda env pytorch.

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
environment location: /opt/intel/oneapi/intelpython/latest/envs/pytorch

Even giving the current user permissions to the environment and the envs directory does not resolve this:

sudo chown -R $USER:$USER /opt/intel/oneapi/intelpython/latest/envs/pytorch

conda activate pytorch

conda install nb_conda

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
environment location: /opt/intel/oneapi/intelpython/latest/envs/pytorch

[AlekhyaV_Intel]

Hi,

Thank you for posting in Intel Forums. We tried reproducing your issue. There might not be write permissions for the pre-installed environments hence you could've faced these issues. We have provided few commands to create new environments in the user directory so that you can install any other packages without any issues.

You could use the following to create an AI Kit Tensorflow* environment named “aikit-tf” or an Intel:

conda create -n aikit-tf -c intel intel-aikit-tensorflow

Similarly, you can create an AI Kit PyTorch environment named "aikit-pt":

conda create -n aikit-pt -c intel intel-aikit-pytorch

After creating, you can activate that respective environment using

conda activate <environment_name>

While working on your issue, we created pytorch environment "aikit-pt" and installed nb_conda. It worked for us so we're providing the below steps:

conda create -n aikit-pt -c intel intel-aikit-pytorch

conda activate aikit-pt

conda install nb_conda

Please try the above commands and let us know if this works.

Regards,

Alekhya

[AdamMiltonBarker]

Hi thank you for the reply, I am aware of the ways to create the new envs, however the output shown in the first image is misleading, TF and Pytorch are not installed, you would expect to be able to use TF and Torch without doing anything as when initializing the environment it tells you the latest of both are installed ad active, which they actually are not and are not the latest versions when using the provided envs.

[AdamMiltonBarker]

Hi Alekhya.

I have oneAPI on my laptop (The machine where the issue with VSCode extensions exists) and from that machine, if I load oneAPI in command prompt I can use torch and TF out of the box without the requirement for conda env. The machine I am having this issue with is on an Intel NUC NUC8i7BEH with Ubuntu 20.04.2 on. 

[AdamMiltonBarker]

Just to clarify also, on the NUC setting up a new env does work and I can switch to the AI kit env. So just something weird with the root installation that I can only reproduce on the NUC so far, I will try another machine also. 

[AlekhyaV_Intel]

Hi Adam,

We could reproduce your issue.

Though PyTorch & Tensorflow are installed in the oneAPI AI Analytics toolkit directory,  we must still activate the environments every time we use. Please refer the below steps.

To use PyTorch:

conda activate pytorch

To use the Intel® Optimization for Tensorflow:

conda activate tensorflow

To return to the Intel® Distribution for Python after activating PyTorch or Tensorflow:

conda activate root

The reason you're getting "EnvironmentNotWritable Error" is that normal users don't have root permissions to install to the oneapi AI kit installation directory(/opt/intel/oneapi/). That's why we provided appropriate commands in our previous responses as a workaround to create a new environment when a user doesn't have root privileges.

However, this issue can be solved by changing folder permissions using sudo. We have provided few steps below that worked for us. Please find the screenshot attached.

cd opt/intel/oneapi/intelpython/latest/envs

sudo chmod -R o+w pytorch

{ chmod o+w means that you are going to give others the write permission WITHOUT altering user and group permissions. }

conda activate pytorch

conda install nb_conda

Please try the above commands and let us know if this works for you as well. Please refer this documentation for more information regarding PyTorch and Tensorflow environments. We will check internally regarding the tf version issue and get back to you.

Regards,

Alekhya

[AdamMiltonBarker]

Thanks I am using a custom env, I did already give root permissions to my user for the environment dir but it failed to allow me to install anything.  The issue with version numbers is there, I am making a pytorch model and deprecated parameters exist in the Intel Pytorch version   https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.RandomAffine according to docs fillColor is deprecated and fill should be used, but fill does not exists in the oneAPI version, also RandomEqualize should exist but it does not. 

[AlekhyaV_Intel]

Hi,

We have addressed all your issues below:

1) Issue regarding root permissions

Could you please try applying the below commands to different python environment folders and let us know if the issue still persists.

sudo chmod -R u+w <python_env_folder>
sudo chmod -R o+w <python_env_folder> 
     {u+w -> to give write permissions to $USER
      o+w -> to give write permissions to other users}

And please provide us the ls -l log.

2) Issue regarding RandomEqualize() and fillcolor().

You're facing this issue because the torchvision provided with OneAPI is of an earlier version. If you want to use this package, you can upgrade your torchvision and pytorch using the following command:

Step 1:

python -m pip install torchvision --upgrade

In PyTorch 1.7 version, the RandonEqualize & RandomAffine(consists of fill attribute) functions might be unavailable. So you have to upgrade to 1.9 versions to use these functions. And the Intel Optimization AI frameworks are merged with the main branch, so the native PyTorch would be Intel Optimized.

Step 2:

Activate python environment and import transforms. Please follow the below commands. We have also attached a screenshot of the commands that worked for us.

import torchvision.transforms as T
equalizer = T.RandomEqualize()
T.RandomAffine(45, fill=(0, 0, 255))
dir (T) {to check the required functions in the T directory}

3)Issue regarding incorrect version numbers. 

We are checking on the version numbers issue internally and we'll get back to you with an update. 

Regards,

Alekhya

[AdamMiltonBarker]

Thanks I was just verifying with you my original report that the installation of oneapi states the latest versions are installed but they are not.  

[AdamMiltonBarker]

I can confirm that updating worked.

[AlekhyaV_Intel]

Hi,

Regarding the issue with the wrong versions displayed, we have checked on this internally. The Intel AI toolkit which is released is 2021.2 version and the latest version 2021.3 with latest version of Tensorflow and PyTorch is work in progress. And with the current installation you'll be getting tensorflow 2.3 & Pytorch 1.7 versions. Please refer the release notes here: https://software.intel.com/content/www/us/en/develop/articles/oneapi-ai-analytics-toolkit-release-notes.html

Regards,

Alekhya

[AdamMiltonBarker]

Thank you for the replies, regarding the version numbers it was more a bug report to let you know that although it says latest versions they weren't. My message yesterday was just to confirm that the latest versions were not installed. Thanks everyone. 

[AdamMiltonBarker]

Ok so with nothing changed, today suddenly the conda env was not available within notebooks. I had been using it for two days it was still running, closed the server and when logging back in it just now, the oneapi env was not available anymore. Will work out what happened and update.

The solution was (Which I didn't do last time) to run  ipython kernel install --user --name=all-jetson-nano--display-name="all-jetson-nano" inside the env.

[AdamMiltonBarker]

This is weird, on a fresh conda env on a different machine it worked as it did the other day, no requirement for the ipython kernal command, only needs nb_conda installed and all envs are there, strange what happened on the other machine the other day. 

[AdamMiltonBarker]

There are more issues, this time with Tensorflow, on the NUC, 

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/nuci71/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 444, in <module>
_ll.load_library(_main_dir)
File "/home/nuci71/.local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 154, in load_library
py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /home/nuci71/.conda/envs/all-nano-33-ble=sense/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

This happened just from creating a new env with intel-aikit-tensorflow, I then upgraded to 2.5.0 incase it was an issue there but the error persists.  I have Tensorflow installed on the NUC in the base env, have been using it for a while with no issues, this issue comes from intel-aikit-tensorflow. Now it has messed up the installation I already had., downgrading numpy fixed it in the base env, I have destroyed the env 3 times and each time created a new env with intel-aikit-tensorflow, the issue is reproducible in all cases. Can't import tensorflow to even check the version, any time trying to use intel-aikit-tensorflow it fails with the above. 

The issue with the intel-aikit-tensorflow is resolved by pip install -U tensorflow==2.3.0 this then completely removed the working version of 2.5.0 in the base env.

[AdamMiltonBarker]

I have replicated this again on another machine, freshly created conda env with aikit tensorflow, enter the env, import tensorflow:

(base) nas@hias:~/ALL-Arduino-Nano-33-BLE-Sense-Classifier$ conda activate all-nano-33-ble-sense
(all-nano-33-ble-sense) nas@hias:~/ALL-Arduino-Nano-33-BLE-Sense-Classifier$ python
Python 3.7.11 (default, Jul 13 2021, 16:27:58)
[GCC 9.3.0] :: Intel Corporation on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import tensorflow as tf
2021-07-16 21:11:22.358166: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/nas/.conda/envs/all-nano-33-ble-sense/lib/libfabric:/opt/intel/oneapi/vpl/2021.2.2/lib:/opt/intel/oneapi/tbb/2021.2.0/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/mpi/2021.2.0//libfabric/lib:/opt/intel/oneapi/mpi/2021.2.0//lib/release:/opt/intel/oneapi/mpi/2021.2.0//lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/ipp/2021.2.0/lib/intel64:/opt/intel/oneapi/ippcp/2021.2.0/lib/intel64:/opt/intel/oneapi/ipp/2021.2.0/lib/intel64:/opt/intel/oneapi/dnnl/2021.2.0/cpu_dpcpp_gpu_dpcpp/lib:/opt/intel/oneapi/dal/2021.2.0/lib/intel64:/opt/intel/oneapi/compiler/2021.2.0/linux/lib:/opt/intel/oneapi/compiler/2021.2.0/linux/lib/x64:/opt/intel/oneapi/compiler/2021.2.0/linux/lib/emu:/opt/intel/oneapi/compiler/2021.2.0/linux/compiler/lib/intel64_lin:/opt/intel/oneapi/compiler/2021.2.0/linux/compiler/lib:/opt/intel/oneapi/ccl/2021.2.0/lib/cpu_gpu_dpcpp:/opt/intel/oneapi/vpl/2021.2.2/lib:/opt/intel/oneapi/tbb/2021.2.0/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/mpi/2021.2.0//libfabric/lib:/opt/intel/oneapi/mpi/2021.2.0//lib/release:/opt/intel/oneapi/mpi/2021.2.0//lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/ipp/2021.2.0/lib/intel64:/opt/intel/oneapi/ippcp/2021.2.0/lib/intel64:/opt/intel/oneapi/ipp/2021.2.0/lib/intel64:/opt/intel/oneapi/dnnl/2021.2.0/cpu_dpcpp_gpu_dpcpp/lib:/opt/intel/oneapi/debugger/10.1.1/dep/lib:/opt/intel/oneapi/debugger/10.1.1/libipt/intel64/lib:/opt/intel/oneapi/debugger/10.1.1/gdb/intel64/lib:/opt/intel/oneapi/dal/2021.2.0/lib/intel64:/opt/intel/oneapi/compiler/2021.2.0/linux/lib:/opt/intel/oneapi/compiler/2021.2.0/linux/lib/x64:/opt/intel/oneapi/compiler/2021.2.0/linux/lib/emu:/opt/intel/oneapi/compiler/2021.2.0/linux/lib/oclfpga/host/linux64/lib:/opt/intel/oneapi/compiler/2021.2.0/linux/lib/oclfpga/linux64/lib:/opt/intel/oneapi/compiler/2021.2.0/linux/compiler/lib/intel64_lin:/opt/intel/oneapi/compiler/2021.2.0/linux/compiler/lib:/opt/intel/oneapi/ccl/2021.2.0/lib/cpu_gpu_dpcpp
2021-07-16 21:11:22.358250: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/nas/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 444, in <module>
_ll.load_library(_main_dir)
File "/home/nas/.local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 154, in load_library
py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /home/nas/.conda/envs/all-nano-33-ble-sense/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

[AlekhyaV_Intel]

Hi,

We would like to know which tensorflow version you would want to work on. 

If you want to work on the tensorflow 2.3.0, you could use it long with AI toolkit or if you want to work on the latest tensorflow version i.e. 2.5.0, you could create a new virtual environment without AI toolkit and install tensorflow 2.5.0 with the following command:

conda create -n <env_name> pip python=3.7
conda activate <env_name>
pip install --ignore-installed --upgrade tensorflow

You could check the tensorflow version through conda list.

AI toolkit 2021.3.0 with tensorflow 2.5 will be released soon. So it would be great if you could wait for the latest version to get released as there would be package conflicts when installed with different versions.

Regards,

Alekhya

[AdamMiltonBarker]

Thanks for the info, this post was more to document issues I found while using oneAPI. I have already completed the projects using the versions that come with it, and will definitely upgrade when the new 2021.3.0 is available. 

My main points were:

- "Latest version" is misleading, maybe instead of displaying latest version you could list the actual version number.

- You have to use envs to use the installed versions of tf and pytorch, maybe add a note when the sertvars is run that this is the case, to avoid confusion if someone tries to use TF or Pytorch without an env.

- There were conflicts between already installed versions of TF. 

Thanks again for those that have offered help in this topic, I did try to say a number of times it was more to report bugs and issues to you rather than to find solutions. 

[AdamMiltonBarker]

Some other issues I found during the rounds, in the documentation for Intel Optimized Tensorflow:

Available for Linux*

TensorFlow* version: 2.5

Installation instructions:

Open Anaconda prompt and use the following instruction. Available for Python 3.7, 3.8 and 3.9.

conda install tensorflow -c intel

This is also not correct, it install 2.3.0. 

[AlekhyaV_Intel]

Hi,

Thank you for sharing all your observations. Could you please share with us the documentation links which you're referring?

Regards,

Alekhya