topic in Intel® Distribution for Python*

GDAL not working with Intel Distribution for Python

soman__kritik — Mon, 08 Jan 2018 09:08:54 GMT

I was trying to install GDAL with Intel Distribution for Python on anaconda using

conda install -c conda-forge gdal

After installation, I am unable to import gdal, it shows that a label in not present in hdf5 package. I then tried older versions of GDAL, but I was unable to install them due to dependencies on mkl_fft.

Please update Intel Distribution for Python to support GDAL.

Hi Kritik,

DavidLiu — Mon, 08 Jan 2018 20:30:15 GMT

Hi Kritik,

Using the Intel® Distribution for Python* with Python 3.6 nets me a very different issue in which libgdal.20.dylib having issues with a dynamic link. Can you post your conda info along with the subsequent error output?

Thanks,

David

Hi David

soman__kritik — Tue, 09 Jan 2018 05:41:15 GMT

Hi David

Thanks for the reply. I am actually trying to perform clustering on geospatial data by using intel’s sklearn. I want to import a geotif file for which I need gdal.

I was using python 2.7. I am not able to reproduce the last error that I was facing earlier, but no version of gdal seems to work in Intel distribution for python. Please find below the error I am getting now. My code was running on normal python 2.7. I shall be grateful if you could help setting up any version of gdal with intel for python.

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda install -c intel scikit-learn

Fetching package metadata .............

Solving package specifications: .

Package plan for installation in environment /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo:

The following NEW packages will be INSTALLED:

daal: 2018.0.1.20171012-1 intel

mkl_fft: 1.0.0-np113py27_intel_15 intel [intel]

mkl_random: 1.0.0-np113py27_intel_6 intel [intel]

pydaal: 2018.0.1.20171012-np113py27_intel_0 intel [intel]

scikit-learn: 0.19.0-np113py27_intel_6 intel [intel]

scipy: 0.19.1-np113py27_intel_23 intel [intel]

tbb: 2018.0.1-py27_intel_3 intel [intel]

The following packages will be UPDATED:

hdf5: 1.8.15.1-2 --> 1.10.1-intel_0 intel [intel]

libnetcdf: 4.3.3.1-3 --> 4.5.0-h42fd751_7

mkl: 2017.0.2-intel_1 intel [intel] --> 2018.0.1-intel_5 intel

numpy: 1.11.3-py27_0 --> 1.13.3-py27_intel_6 intel [intel]

The following packages will be DOWNGRADED:

gdal: 2.0.0-np111py27_3 --> 1.10.1-py27_1

libgdal: 2.0.0-0 --> 1.11.2-1

Proceed (/n)? y

daal-2018.0.1. 100% |################################| Time: 0:01:58 653.65 kB/s

mkl-2018.0.1-i 100% |################################| Time: 0:02:30 897.26 kB/s

libnetcdf-4.5. 100% |################################| Time: 0:00:04 241.18 kB/s

tbb-2018.0.1-p 100% |################################| Time: 0:00:02 198.79 kB/s

mkl_fft-1.0.0- 100% |################################| Time: 0:00:00 250.12 kB/s

mkl_random-1.0 100% |################################| Time: 0:00:01 250.54 kB/s

numpy-1.13.3-p 100% |################################| Time: 0:00:10 550.26 kB/s

pydaal-2018.0. 100% |################################| Time: 0:01:02 780.27 kB/s

scipy-0.19.1-n 100% |################################| Time: 0:00:14 1.90 MB/s

scikit-learn-0 100% |################################| Time: 0:00:06 922.40 kB/s

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ python

Python 2.7.14 |Intel Corporation| (default, Oct 16 2017, 10:29:14)

[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

Intel(R) Distribution for Python is brought to you by Intel Corporation.

Please check out: https://software.intel.com/en-us/python-distribution

>>> import gdal

>>>

[15]+ Stopped python

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ python clustering.py

Traceback (most recent call last):

File "clustering.py", line 17, in <module>

original_dem = original_dem.ReadAsArray()

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdal.py", line 781, in ReadAsArray

import gdalnumeric

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdalnumeric.py", line 1, in <module>

from gdal_array import *

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdal_array.py", line 25, in <module>

_gdal_array = swig_import_helper()

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdal_array.py", line 17, in swig_import_helper

import _gdal_array

ImportError: No module named _gdal_array

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda remove -n geo gdal

Fetching package metadata ...........

Solving package specifications: .

Package plan for package removal in environment /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo:

The following packages will be REMOVED:

gdal: 1.10.1-py27_1

Proceed (/n)? y

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda install -c conda-forge gdal

Fetching package metadata .............

Solving package specifications: .

Package plan for installation in environment /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo:

The following NEW packages will be INSTALLED:

gdal: 2.2.3-py27_0 conda-forge

The following packages will be UPDATED:

curl: 7.45.0-0 --> 7.55.1-0 conda-forge

freetype: 2.8-intel_0 intel [intel] --> 2.8.1-0 conda-forge

glib: 2.53.6-h33f6a65_2 --> 2.55.0-0 conda-forge

kealib: 1.4.5-0 --> 1.4.7-4 conda-forge

libgdal: 1.11.2-1 --> 2.2.3-2 conda-forge

poppler: 0.60.1-hb654a48_0 --> 0.61.1-3 conda-forge

The following packages will be SUPERSEDED by a higher-priority channel:

cairo: 1.14.10-h913ea44_6 --> 1.14.10-0 conda-forge

libnetcdf: 4.5.0-h42fd751_7 --> 4.4.1.1-10 conda-forge

Proceed (/n)? y

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ python clustering.py

Traceback (most recent call last):

File "clustering.py", line 2, in <module>

import gdal

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/gdal.py", line 2, in <module>

from osgeo.gdal import deprecation_warn

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/__init__.py", line 21, in <module>

_gdal = swig_import_helper()

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/__init__.py", line 17, in swig_import_helper

_mod = imp.load_module('_gdal', fp, pathname, description)

ImportError: dlopen(/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/_gdal.so, 2): Library not loaded: @rpath/libpng16.16.dylib

Referenced from: /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/libgdal.20.dylib

Reason: Incompatible library version: libgdal.20.dylib requires version 51.0.0 or later, but libpng16.16.dylib provides version 49.0.0

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda remove -n geo gdal

Fetching package metadata ...........

Solving package specifications: .

Package plan for package removal in environment /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo:

The following packages will be REMOVED:

gdal: 2.2.3-py27_0 conda-forge

Proceed (/n)? y

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda search gdal

Fetching package metadata ...........

gdal 1.10.1 py27_0 defaults

1.10.1 py26_1 defaults

1.10.1 py27_1 defaults

1.10.1 py33_1 defaults

1.10.1 np18py26_2 defaults

1.10.1 np18py27_2 defaults

1.10.1 np18py33_2 defaults

1.10.1 np18py34_2 defaults

1.11.0 np19py26_0 defaults

1.11.0 np19py27_0 defaults

1.11.0 np19py33_0 defaults

1.11.0 np19py34_0 defaults

1.11.1 np19py26_0 defaults

1.11.1 np19py27_0 defaults

1.11.1 np19py33_0 defaults

1.11.1 np19py34_0 defaults

1.11.1 np19py26_1 defaults

1.11.1 np19py27_1 defaults

1.11.1 np19py33_1 defaults

1.11.1 np19py34_1 defaults

1.11.2 np19py26_0 defaults

1.11.2 np19py27_0 defaults

1.11.2 np19py33_0 defaults

1.11.2 np19py34_0 defaults

1.11.2 np19py26_1 defaults

1.11.2 np19py27_1 defaults

1.11.2 np19py33_1 defaults

1.11.2 np19py34_1 defaults

1.11.2 np19py26_2 defaults

1.11.2 np19py27_2 defaults

1.11.2 np19py33_2 defaults

1.11.2 np19py34_2 defaults

1.11.2 np19py26_3 defaults

1.11.2 np19py27_3 defaults

1.11.2 np19py33_3 defaults

1.11.2 np19py34_3 defaults

2.0.0 np110py27_0 defaults

2.0.0 np110py34_0 defaults

2.0.0 np110py35_0 defaults

2.0.0 np19py26_0 defaults

2.0.0 np19py27_0 defaults

2.0.0 np19py33_0 defaults

2.0.0 np19py34_0 defaults

2.0.0 np19py35_0 defaults

2.0.0 np110py27_3 defaults

2.0.0 np110py34_3 defaults

2.0.0 np110py35_3 defaults

2.0.0 np111py27_3 defaults

2.0.0 np111py34_3 defaults

2.0.0 np111py35_3 defaults

2.1.0 py27_0 defaults

2.1.0 py34_0 defaults

2.1.0 py35_0 defaults

2.2.2 py27h98c8286_0 defaults

2.2.2 py36hff4f997_0 defaults

2.2.2 py35h90dc150_0 defaults

2.2.2 py36h503167d_0 defaults

2.2.2 py35hd88591f_0 defaults

2.2.2 py27hc303eaf_0 defaults

2.2.2 py27h65d9da8_1 defaults

2.2.2 py36h17ea0c9_1 defaults

2.2.2 py35he67b238_1 defaults

2.2.2 py35h7aa31b1_1 defaults

2.2.2 py27h9d2a75c_1 defaults

2.2.2 py36hd505dc6_1 defaults

libgdal 1.11.2 0 defaults

1.11.2 1 defaults

1.11.2 2 defaults

2.0.0 0 defaults

2.0.0 4 defaults

2.1.0 0 defaults

2.2.2 h2c60913_0 defaults

2.2.2 h3aae6cd_0 defaults

2.2.2 h7ea3a73_0 defaults

2.2.2 hdcd1c41_0 defaults

2.2.2 ha430587_1 defaults

2.2.2 h839ae26_1 defaults

2.2.2 h3559a57_1 defaults

2.2.2 hc3eca24_1 defaults

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda install gdal=2.0.0

Fetching package metadata ...........

Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:

- gdal 2.0.0*

- pydaal

Use "conda info <package>" to see the dependencies for each package.

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda install gdal=2.1.0

Fetching package metadata ...........

Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:

- gdal 2.1.0* -> libgdal 2.1.0 -> geos 3.5.*

- gdal 2.1.0* -> libgdal 2.1.0 -> proj4 4.9.2

- libspatialite -> geos >=3.6.2,<3.6.3.0a0

Use "conda info <package>" to see the dependencies for each package.

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda install gdal=1.11.2

Fetching package metadata ...........

Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:

- gdal 1.11.2*

- pydaal

Use "conda info <package>" to see the dependencies for each package.

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda install gdal=1.10.1

Fetching package metadata ...........

Solving package specifications: .

Package plan for installation in environment /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo:

The following NEW packages will be INSTALLED:

gdal: 1.10.1-py27_1

Proceed (/n)? y

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ python clustering.py

Traceback (most recent call last):

File "clustering.py", line 17, in <module>

original_dem = original_dem.ReadAsArray()

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdal.py", line 781, in ReadAsArray

import gdalnumeric

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdalnumeric.py", line 1, in <module>

from gdal_array import *

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdal_array.py", line 25, in <module>

_gdal_array = swig_import_helper()

File "/Volumes/MacBookData/kritiksoman/anaconda3/envs/geo/lib/python2.7/site-packages/osgeo/gdal_array.py", line 17, in swig_import_helper

import _gdal_array

ImportError: No module named _gdal_array

(geo) Kritiks-MacBook-Air:clustering kritiksoman$ conda info

Current conda install:

platform : osx-64

conda version : 4.3.30

conda is private : False

conda-env version : 4.3.30

conda-build version : 3.0.27

python version : 3.6.3.final.0

requests version : 2.18.4

root environment : /Volumes/MacBookData/kritiksoman/anaconda3 (writable)

default environment : /Volumes/MacBookData/kritiksoman/anaconda3/envs/geo

envs directories : /Volumes/MacBookData/kritiksoman/anaconda3/envs

/Volumes/MacBookData/kritiksoman/.conda/envs

package cache : /Volumes/MacBookData/kritiksoman/anaconda3/pkgs

/Volumes/MacBookData/kritiksoman/.conda/pkgs

channel URLs : https://repo.continuum.io/pkgs/main/osx-64

https://repo.continuum.io/pkgs/main/noarch

https://repo.continuum.io/pkgs/free/osx-64

https://repo.continuum.io/pkgs/free/noarch

https://repo.continuum.io/pkgs/r/osx-64

https://repo.continuum.io/pkgs/r/noarch

https://repo.continuum.io/pkgs/pro/osx-64

https://repo.continuum.io/pkgs/pro/noarch

config file : /Volumes/MacBookData/kritiksoman/.condarc

netrc file : None

offline mode : False

user-agent : conda/4.3.30 requests/2.18.4 CPython/3.6.3 Darwin/17.3.0 OSX/10.13.2

UID:GID : 501:20

(geo) Kritiks-MacBook-Air:clustering kritiksoman$

Rohit_J_Intel — Tue, 09 Jan 2018 14:15:00 GMT

Hi Kritik,

I was able to install gdal from conda-forge into IDP-2.7 using following steps:

<IDP_installation>/bin/conda create -n gdal_env -c intel python=2.7 scikit-learn -y
source <IDP_installation>/bin/activate gdal_env
conda install --no-deps -c conda-forge libgdal -y
conda install -c conda-forge gdal -y

For testing if gdal can be imported into the interpreter :

python -c "import gdal"

Thanks,

Rohit

Hi Rohit

soman__kritik — Thu, 11 Jan 2018 04:58:00 GMT

Hi Rohit

Thanks for the help. I also want to install opencv, matplotlib, skimage, and scipy after sklearn and gdal. But on doing so, I am running into the following errors

1) Incompatible library version: libgdal.20.dylib requires version 51.0.0 or later, but libpng16.16.dylib provides version 49.0.0

2)

conda install -c intel scikit-image
Fetching package metadata .............
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:
  - geotiff
  - harfbuzz
  - libspatialite
  - mkl_fft
  - scikit-image
Use "conda info <package>" to see the dependencies for each package.

Please advise how these can be resolved.

Thanks

Kritik

Rohit J. (Intel) wrote:

Hi Kritik,

I was able to install gdal from conda-forge into IDP-2.7 using following steps:
<IDP_installation>/bin/conda create -n gdal_env -c intel python=2.7 scikit-learn -y
source <IDP_installation>/bin/activate gdal_env
conda install --no-deps -c conda-forge libgdal -y
conda install -c conda-forge gdal -y
For testing if gdal can be imported into the interpreter :
python -c "import gdal"
Thanks,

Rohit

Hi Kritik,

Rohit_J_Intel — Thu, 11 Jan 2018 13:44:00 GMT

Hi Kritik,

Since you have "gdal_env" already setup with scikit-learn, scipy is already installed in your environment. We currently don't ship scikit-image and opencv on OSX, but you can install all your requirements through:

source <IDP_installation>/bin/activate gdal_env
conda install -c intel matplotlib -y
conda install -c conda-forge opencv scikit-image -y

Thanks,

Rohit

Hi Rohit

soman__kritik — Thu, 11 Jan 2018 14:59:38 GMT

Hi Rohit

When I run

conda install -c conda-forge opencv scikit-image -y

gdal 2.2.2 downgrades to 1.10.1, which is a problem because it gives error:ImportError: No module named _gdal_array

I want to run gdal >=2.2. I am attaching the terminal log.

Thanks

Kritik

Rohit J. (Intel) wrote:

Hi Kritik,

Since you have "gdal_env" already setup with scikit-learn, scipy is already installed in your environment. We currently don't ship scikit-image and opencv on OSX, but you can install all your requirements through:
source <IDP_installation>/bin/activate gdal_env
conda install -c intel matplotlib -y
conda install -c conda-forge opencv scikit-image -y
Thanks,

Rohit

Hi Kritik,

Rohit_J_Intel — Thu, 11 Jan 2018 19:41:54 GMT

Hi Kritik,

It seems that opencv (from conda-forge) causes some conflict with one of dependencies of gdal. The following is a bit awkward but it should get what you require:

<IDP_installation>/bin/conda create -n gdal_env2 -c intel python=2.7 matplotlib scikit-learn -y
source <IDP_installation>/bin/activate gdal_env2
conda install -c conda-forge opencv scikit-image gdal -y

This should give you gdal==2.2.3 and libgdal==2.2.3.

Thanks,

Rohit

Hi Rohit

soman__kritik — Sat, 20 Jan 2018 16:05:19 GMT

Hi Rohit

Thanks for the response. Your suggestion works well.

Thanks

Kritik

Rohit J. (Intel) wrote:

Hi Kritik,

It seems that opencv (from conda-forge) causes some conflict with one of dependencies of gdal. The following is a bit awkward but it should get what you require:
<IDP_installation>/bin/conda create -n gdal_env2 -c intel python=2.7 matplotlib scikit-learn -y
source <IDP_installation>/bin/activate gdal_env2
conda install -c conda-forge opencv scikit-image gdal -y
This should give you gdal==2.2.3 and libgdal==2.2.3.

Thanks,

Rohit