- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When a user creates a new conda environment, he expects the environment to be be completely isolated from the global path. But in this case, the path remains fixed (/glob/intel-python/python3/bin//python) and I'm not able to change.
I've attached the screenshot below.
- Tags:
- General Support
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tried it just now with the default environments.
qsub: waiting for job 557645.v-qsvr-1.aidevcloud to start
qsub: job 557645.v-qsvr-1.aidevcloud ready
########################################################################
# Date: Thu Mar 26 08:10:23 PDT 2020
# Job ID: 557645.v-qsvr-1.aidevcloud
# User: u39403
# Resources: mem=196gb,neednodes=4:ppn=2,nodes=4:ppn=2,walltime=24:00:00
########################################################################
(base) u39403@s001-n006:~$ conda info --envs
# conda environments:
#
base * /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest
2021.1-beta05 /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest/envs/2021.1-beta05
pytorch /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest/envs/pytorch
pytorch-1.4.0 /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest/envs/pytorch-1.4.0
tensorflow /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest/envs/tensorflow
tensorflow-1.15.0 /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest/envs/tensorflow-1.15.0
torch /home/u39403/.conda/envs/torch
(base) u39403@s001-n006:~$ conda activate pytorch
(pytorch) u39403@s001-n006:~$ python3
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>
(pytorch) u39403@s001-n006:~$ conda activate pytorch-1.4.0
(pytorch-1.4.0) u39403@s001-n006:~$ python3
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
Not sure how is it working on your end.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have the same dilemma, not sure how it's not working on your end. ;)
Might be related to packages you've previously installed in your user space.
At this point I would look at a few options:
1. cleanup or refresh your side of things. You mentioned previously you installed some packages and perhaps there's a collision between various things.
To be fair I'm not an expert on Conda env management, so the following may or may not help:
https://docs.conda.io/projects/conda/en/latest/commands/clean.html
conda clean --yes --all
Remove custom environments:
conda remove --name myenv --all
etc. other related cleanup methods
2. You could give us permissions to enter your home folder and try and find a fix
3. We could try and reset your account to default but that may lead to lost files on your side.
Which option would you like to try?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, you have the permission. I'm tired of replying now. Please try it yourself and let me know if you figured out.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have two accounts. The one I'm referring to has User Name: u39403. The other one also has the same problem but I use it primarily.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No worries. I got that info from the terminal output you have posted previously. ;)
Would you mind if we simply backup your files and reset your account to factory settings? Might be faster than trying to debug what's going on.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please go ahead
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Prajjwal, I've decided to wait a bit before going into your account.
Could you please try the following first?
1. mv ~/.conda ~/tmp/
if that does not help and there is pip involved, then
2. mv ~/.local ~/tmp/
Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Done
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Does Conda work now as you would expect it?
Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have not tried yet. I'll try using the instance and let you know any problems that I will encounter.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No it's not working.
Please see below.
(base) u39403@login-2:~$ conda info --envs
WARNING: The conda.compat module is deprecated and will be removed in a future release.
# conda environments:
#
base * /glob/intel-python/versions/2018u2/intelpython3
(base) u39403@login-2:~$ q
qsub: waiting for job 561084.v-qsvr-1.aidevcloud to start
qsub: job 561084.v-qsvr-1.aidevcloud ready
########################################################################
# Date: Tue Mar 31 06:45:03 PDT 2020
# Job ID: 561084.v-qsvr-1.aidevcloud
# User: u39403
# Resources: mem=196gb,neednodes=4:ppn=2,nodes=4:ppn=2,walltime=24:00:00
########################################################################
(base) u39403@s001-n055:~$ conda create --name torch
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/u39403/.conda/envs/torch
Proceed (
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate torch
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) u39403@s001-n055:~$ conda activate torch
(torch) u39403@s001-n055:~$ conda install pytorch torchvision cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/u39403/.conda/envs/torch
added / updated specs:
- cpuonly
- pytorch
- torchvision
The following packages will be downloaded:
package | build
---------------------------|-----------------
cpuonly-1.0 | 0 2 KB pytorch
jpeg-9b | h024ee3a_2 214 KB
libtiff-4.1.0 | h2733197_0 447 KB
olefile-0.46 | py37_0 50 KB
pillow-7.0.0 | py37hb39fc2d_0 598 KB
pytorch-1.4.0 | py3.7_cpu_0 39.0 MB pytorch
torchvision-0.5.0 | py37_cpu 10.6 MB pytorch
zstd-1.3.7 | h0b5b093_0 401 KB
------------------------------------------------------------
Total: 51.2 MB
The following NEW packages will be INSTALLED:
bzip2 conda_channel/linux-64::bzip2-1.0.8-0
certifi conda_channel/linux-64::certifi-2019.11.28-py37_0
cpuonly pytorch/noarch::cpuonly-1.0-0
freetype conda_channel/linux-64::freetype-2.10.1-1
icc_rt conda_channel/linux-64::icc_rt-2021.1b5-intel_387
intel-openmp conda_channel/linux-64::intel-openmp-2021.1b5-intel_387
intelpython conda_channel/linux-64::intelpython-2021.1b5-0
jpeg pkgs/main/linux-64::jpeg-9b-h024ee3a_2
libffi conda_channel/linux-64::libffi-3.3-11
libgcc-ng conda_channel/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libpng conda_channel/linux-64::libpng-1.6.37-2
libstdcxx-ng conda_channel/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
libtiff pkgs/main/linux-64::libtiff-4.1.0-h2733197_0
mkl conda_channel/linux-64::mkl-2021.1b5-intel_336
mkl-service conda_channel/linux-64::mkl-service-2.3.0-py37_0
mkl_fft conda_channel/linux-64::mkl_fft-1.1.0-py37ha68da19_0
mkl_random conda_channel/linux-64::mkl_random-1.1.0-py37ha68da19_0
ninja conda_channel/linux-64::ninja-1.9.0-py37hfd86e86_0
numpy conda_channel/linux-64::numpy-1.17.5-py37ha68da19_1
numpy-base conda_channel/linux-64::numpy-base-1.17.5-py37_1
olefile pkgs/main/linux-64::olefile-0.46-py37_0
openssl conda_channel/linux-64::openssl-1.1.1d-0
pillow pkgs/main/linux-64::pillow-7.0.0-py37hb39fc2d_0
pip conda_channel/linux-64::pip-19.3.1-py37_2
python conda_channel/linux-64::python-3.7.5-hf484d3e_3
pytorch pytorch/linux-64::pytorch-1.4.0-py3.7_cpu_0
setuptools conda_channel/linux-64::setuptools-45.2.0-py37_0
six conda_channel/linux-64::six-1.13.0-py37_2
sqlite conda_channel/linux-64::sqlite-3.30.1-6
tbb conda_channel/linux-64::tbb-2021.1b5-intel_342
tbb4py conda_channel/linux-64::tbb4py-2021.1b5-py37_intel_0
tcl conda_channel/linux-64::tcl-8.6.9-24
tk conda_channel/linux-64::tk-8.6.9-29
torchvision pytorch/linux-64::torchvision-0.5.0-py37_cpu
wheel conda_channel/linux-64::wheel-0.33.6-py37_4
xz conda_channel/linux-64::xz-5.2.4-h14c3975_7
zlib conda_channel/linux-64::zlib-1.2.11.1-h14c3975_1
zstd pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0
Proceed (
Downloading and Extracting Packages
pillow-7.0.0 | 598 KB | ############################################################################################################################################# | 100%
libtiff-4.1.0 | 447 KB | ############################################################################################################################################# | 100%
cpuonly-1.0 | 2 KB | ############################################################################################################################################# | 100%
olefile-0.46 | 50 KB | ############################################################################################################################################# | 100%
torchvision-0.5.0 | 10.6 MB | ############################################################################################################################################# | 100%
zstd-1.3.7 | 401 KB | ############################################################################################################################################# | 100%
jpeg-9b | 214 KB | ############################################################################################################################################# | 100%
pytorch-1.4.0 | 39.0 MB | ############################################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(torch) u39403@s001-n055:~$ python3
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>
(torch) u39403@s001-n055:~$ which python
/glob/intel-python/python3/bin/python
(torch) u39403@s001-n055:~$
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Prajjwal,
Have you looked at the following getting started guide?
https://devcloud.intel.com/oneapi/get-started/analytics-toolkit/
It covers our recommended way of using the PyTorch and TensorFlow versions that are installed through the Intel® AI Analytics Toolkit(Beta).
with apologies, could you please try the following steps:
1. mv ~/.conda ~/tmp/
2. mv ~/.local ~/tmp/
3. ssh devcloud
4. source /opt/intel/inteloneapi/setvars.sh
5. conda activate pytorch
The source .../setvars.sh command is required in order for this to work. We have added this command to the ~/.bash_profile of all (or at least most) users a while back but it seems that in your case that may not be true.
The AI DevCloud is being retired, I believe today is the last day. The old way of doing things on the DevCloud is not supported anymore.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please check your recommended solution before you inform me. You already have access to my account.
(base) u39403@s001-n055:~$ source /opt/intel/inteloneapi/setvars.sh
:: setvars has already been run. Skipping any further invocation. To force its re-execution, pass --force
(base) u39403@s001-n055:~$ conda activate pytorch
(pytorch) u39403@s001-n055:~$ python3
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Good catch. Yes, you can't use the login node for your work.
You need to access a compute node.
Here's an updated list of instructions.
1. ssh devcloud
This will lead you to the login node.
2. mv ~/.conda ~/tmp/
3. mv ~/.local ~/tmp/
This was just to clean up your conda/pip environment.
4. qsub -I
Request an interractive session on a compute node.
Note: -I is upper case i not lower case L.
5. source /opt/intel/inteloneapi/setvars.sh
6. conda activate pytorch
The 4th step requests an interactive session on any compute node. This is great for debugging your projects but there's a better way if you want to execute some large workloads (for example ML training): the batch mode.
For more info about job submission see this link: https://devcloud.intel.com/oneapi/learn/job-submission/.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please carefully see what I posted. I tried your method, it doesn't work. I never asked about compute/login node. That's clearly evident from the snippet I shared. I'm again posting it here.
(base) u39403@s001-n055:~$ source /opt/intel/inteloneapi/setvars.sh
:: setvars has already been run. Skipping any further invocation. To force its re-execution, pass --force
(base) u39403@s001-n055:~$ conda activate pytorch
(pytorch) u39403@s001-n055:~$ python3
Python 3.6.3 |Intel Corporation| (default, Feb 12 2018, 06:37:09)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>
Pytorch is not imported. This should not happen.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Somehow you're starting the system python not the intelpython from oneAPI despite of sourcing the setvars.sh. I can only assume that certain changes on your profile are hiding the intelpython. Any thoughts?
What would be a good time window for us to go in using your account? I'd like to avoid interfering with your work.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please go ahead (log in to my account). I'm not using it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you please print exact steps on how a user is supposed to create a fresh conda environment and install packages ? Please test it out properly. Whatever I receive on these forums are either half cooked solutions or they don't work at all.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It seems that users can't install any packages now in intel provided environments. They're greeted with this:
EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
environment location: /glob/development-tools/versions/oneapi/beta05/inteloneapi/intelpython/latest/envs/pytorch-1.4.0
Why are there so many restrictions ? Why can't a user do a fresh install by themselves ? I think docker support would be more handy rather than using what is provided.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Prajjwal, thank you for your patience.
I have accessed your account and found a conda initialization section in your .bashrc config file. This section configured the python in /glob/intel-python as your default python, a valid setting during the lifetime of AI DevCloud. Since the AI DevCloud was retired on 03/31/2020, the initialization section is not necessary anymore, at least in the current form.
In order to do some testing, I have commented out that initialization section and this time the steps I mentioned in a previous post passed. Please try it and let me know if that addresses your issue.
Based on your experience we're conducting additional investigations & fixes. One related issue we found is that after conda deactivate the oneAPI environment variables might be disabled. Our recommendation is to explicitly run "source /opt/intel/inteloneapi/setvars.sh" before any meaningful work.
As a thank you for your continuing efforts to improve the Intel(R) DevCloud we've extended your access by one additional year. Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Could you please confirm, whether the steps mentioned previously are working fine now?
Thanks
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page