pip install ipp is creating redundant libraries, not symlinks

Fisticleese · ‎09-27-2022

Good morning,

Ever since I've started installing IPP using pip, I am noticing that it creates redundant copies of libraries rather than 1 library and symlinks that is normal for Linux.

I'll pick just one file as an example, after IPP installs there are 3 files libippi_tl_tbb.so, libippi_tl_tbb.so.10, libippi_tl_tbb.so.10.5 that are full size with the same data.

Instead there should be 1 file and symlinks as such:

libippi_tl_tbb.so -> /usr/local/lib/tl/tbb/libippi_tl_tbb.so.10.5
libippi_tl_tbb.so.10 -> /usr/local/lib/tl/tbb/libippi_tl_tbb.so.10.5
libippi_tl_tbb.so.10.5

The end result is that IPP has 3X the footprint on disk than it should, which in my case causes my Docker image to explode in size by several extra GB.

As a temporary workaround, I'm running some fix scripts that look similar to this:

for file in $(ls /usr/local/lib/tl/tbb | grep '10.5'); do \
NAME=`echo "$file" | cut -d'.' -f1`; \
rm /usr/local/lib/tl/tbb/${NAME}.so.10; \
ln -s /usr/local/lib/tl/tbb/${NAME}.so.10.5 /usr/local/lib/tl/tbb/${NAME}.so.10; \
rm /usr/local/lib/tl/tbb/${NAME}.so; \
ln -s /usr/local/lib/tl/tbb/${NAME}.so.10.5 /usr/local/lib/tl/tbb/${NAME}.so; \
done

Can you (Intel) please fix the pip install so that the symlinks are used instead of redundant files?

Thanks!

Chris

VidyalathaB_Intel · ‎09-29-2022

Hi Chris,

Thanks for reaching out to us.

>>Ever since I've started installing IPP using pip, I am noticing that it creates redundant copies of libraries....The end result is that IPP has 3X the footprint on disk than it should, which in my case causes my Docker image to explode in size by several extra GB.

Could you please confirm if you have observed this issue with any of the earlier versions of IPP using pip or if is it related to the latest version of IPP 2021.6?

>> Instead there should be 1 file and symlinks as such:

libippi_tl_tbb.so -> /usr/local/lib/tl/tbb/libippi_tl_tbb.so.10.5
libippi_tl_tbb.so.10 -> /usr/local/lib/tl/tbb/libippi_tl_tbb.so.10.5
libippi_tl_tbb.so.10.5

Also, I think this is how it looks now

Regards,

Vidya.

Fisticleese · ‎09-29-2022

Good morning Vidya, thank you for responding!

The first python ipp package I used that has the problem was from July 2021, and the libraries had .10.2. I did not record the python ipp package version at that time.

The second version I used that has the problem was installed in Nov 2021, and the libraries had .10.3. Unfortunately I did not record the python ipp package version that time either.

The 3rd version I use that has the problem was installed in Dec 2021, and the libraries show as .10.4. I am running a docker container with this version, so I am able to get the python package version:

# pip3 list | grep ipp
ipp 2021.5.2

This week I upgraded the python ipp to 2021.6.1, and that's when I decided to raise the issue on this forum. The libraries now have .10.5.
# pip3 list | grep ipp
ipp 2021.6.1

Your screen shot shows a path that includes /opt/intel/oneapi, that is different than what I see after an installation with pip. When I run the pip install, the Intel libraries are in /usr/local/lib. Here's a fresh install inside an Ubuntu 22.04 docker container showing the redundancy issue:

EDIT: My previous screenshot didn't show the multiple copies. Here's a correction:

My impression based on installing 4 successive versions from pypi is that the issue has always been present with the pip install.

Note that if I install a different way, such as from apt.repos.intel.com using apt install intel-ipp-2020.3-912 then the symlinks appear as normal.

The pip install is what needs the debugging.

Thanks,

Chris

VidyalathaB_Intel · ‎10-02-2022

Hi Chris,

Thanks for letting us know.

>>Note that if I install a different way, such as from apt.repos.intel.com using apt install intel-ipp-2020.3-912 then the symlinks appear as normal.

Thanks for giving it a try and letting us know.

>>the issue has always been present with the pip install...The pip install is what needs the debugging.

Yes, unfortunately, Python wheels (PIP packages) don't support symbolic links.

This issue is valid for all components, not only for IPP.

Maybe you can refer to the below links with similar discussions on Python GitHub (https://github.com/python/cpython/issues/82102) and on the official python forum https://discuss.python.org/t/symbolic-links-in-wheels/1945/24 for more details.

Regards,

Vidya.

VidyalathaB_Intel · ‎10-10-2022

Hi Chris @Fisticleese ,

As we haven't heard back from you, could you please provide us with an update regarding the issue? Please confirm if we can go ahead and close this thread from our end if you do not have any issues.

Regards,

Vidya.

Fisticleese · ‎10-10-2022

Good morning Vidya,

Thank you for following up. I understand pip packages have a general issue with symlinks. I found a similar complaint to mine with regard to MKL being too large and not using symlinks:
https://github.com/oneapi-src/oneMKL/issues/64
For that package, the file duplication issue was solved by removing the extra copies:
https://github.com/oneapi-src/oneMKL/issues/64#issuecomment-812692748

In that link, Intel is saying "as long term solution we are investigating how symlinks (or alternative options) could be enabled in pip wheel package format".

For my reported bug, I would say we cannot close the issue yet since it is not resolved. Please keep my issue open until the symlink problem is solved. A documented post-install fixup would be acceptable.

Regards,

Chris