Sugestion - different channels for different python versions

[Orion_P_]

When installing packages with conda you may end up with packages from the main conda repository instead of versions from the Intel repository.  To avoid this I have been using the "--strict-channel-priority" option.  However with Python 3.7 this can lead to conflicts I believe primarily due to packages in the Intel repository that exist for Python 3.6 but not Python 3.7.  If there were additional Intel repositories split up by python version (plus a Python independent version) I think you would then have more success.

[ArunJ_Intel]

Conda treats Python the same as any other package, so it is easy to manage and update multiple installations. It wouldn't be actually practical to have a different channel/repository specific to multiple versions of python.

Conda while installing package from a channel checks if prerequisites are present in your machine. For example let us say you try to install a conda python based package it checks if a compatible version of python(a dependency/prerequisite) exists on your machine. If the compatible version of python does not exist, the conda should prompt to allow installation of a compatible version of python or would suggest to you the conflicts which might be unresolved. The below document gives more description on the process flow that happens when installing package with conda.

https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/installing-with-conda.html

Having different repositories for different versions would make the installation and package management more complex process. With ever increasing number of packages in conda repositories I feel it would be quite normal to have some of the packages/versions in default channels to be not present in intel channel. If you could point out the specific packages that you are having issues with, we could try verifying this and raise the issue with engineering.

Thanks

Arun

[Orion_P_]

I don't see why it wouldn't be practical to provide the multiple repositories in addition to the combined one.  I'm not very familiar with conda repo generation but if it is anything like yum repos it should be straightforward to sort the packages into the appropriate repos.

conda also suffers from increased depsolving times as the number of packages increases, having limited repos would help with this. 

The main issue that I'm trying to solve is making environments that are mixed with Intel and Anaconda packages that use Intel components whenever possible. I have done this in the past with something like:

conda create -n scipy37-intel-notebook --override-channels --strict-channel-priority -c intel -c main python=3.7 notebook

but this fails because while notebook exists in the Intel channel (and so conda restricts it to that channel) it doesn't exist for Python 3.7 and so it can't satisfy the requirement.

My proposal would allow for something like:

conda create -n scipy37-intel-notebook --override-channels --strict-channel-priority -c intel-py37 -c intel-nopy -c main python=3.7 notebook

[Orion_P_]

I just discovered the "channel=" selector so I'm dealing with this now by specifying package selections like:

conda create -y -n scipy37-intel --override-channels -c intel -c main -c conda-forge python=3.7 'astropy>=3.2.3' 'async_generator>=1.10' 'basemap>=1.2.0' 'mkl>=2020.2' mysql-connector-python 'netcdf4>=1.3.1' 'setuptools>=41.0.1' 'six>=1.11.0' 'xarray>=0.13.0' 'h5py>=2.10.0' 'icc_rt[channel=intel]>=2021.2' 'libprotobuf[channel=intel]' 'matplotlib[channel=intel]>=2.2.2' 'nbbrowserpdf>=0.2.0' 'numba[channel=intel]' 'pandas[channel=intel]>=0.22.0' 'pyodbc>=4.0.30' 'pyyaml[channel=intel]>=3.12' 'scikit-learn[channel=intel]>=0.19.1' scikit-umfpack 'scipy[channel=intel]>=1.0.1' spyder sunpy 'tensorflow[channel=intel]>=2.3.0' 'typing_extensions[channel=intel]' 'zlib[channel=intel]'

[ArunJ_Intel]

Hi

Glad to know you have found a workaround , we will check internally regarding feasibility of multiple repositories  for python versions and let you know when have an update.

One issue I foresee with the proposal is python version might not be the only dependency for a package. Other package versions or modules in the environment could also act as an unresolved dependency causing conflicts. This is why I feel it wouldn't be practical enough to split repositories only on the basis of python packages.

Thanks

Arun

[surbhinahta]

Good to know about the information you have shared with us about Python. I think your tips and strategies could help the writers to do their works properly