Intel® oneAPI Base Toolkit
Support for the core tools and libraries within the base toolkit that are used to build and deploy high-performance data-centric applications.
417 Discussions

oneAPI: improving the setvars.sh script

ax3l
Novice
4,135 Views

From working a couple of months with the new new setvars.sh environment activation file on Linux, I would like to share some feedback on it to improve its quality.

It has the following flaws with respect to common workflows:
- accesses undefined variables
- does not work when sourced in return-code checking scripts
- uses bash-ishm


For the first two points, I think it's helpful to check out this guide:
  https://github.com/azet/community_bash_style_guide

Here is a reproducer:

 

# start a bash shell
bash

# set quality options
set -eu -o pipefail

# source script
. /opt/intel/inteloneapi/setvars.sh
# bash: ZSH_EVAL_CONTEXT: unbound variable

 

All CI runs with "set -eu -o pipefail" these days. A high-quality script would ideally pass this and this avoids hard-to-spot and debug script issues on the user side. Workarounds like "just set +e for our scripts" are not ideal for CI/Docker/... users.

The third point, bash-ism, is problematic because not all people use bash. For example, boost build scripts run as pure sh. Containers also do not necessarily activate a full bash first. Avoiding Bash-specifics and staying sh compatible is not hard.
As a reproducer, just try to source in a sh-shell and simple things like double [[ will pop up.

 

# start a SH shell
# this is the common denominator for CI, Docker, etc.
sh

# source script
. /opt/intel/inteloneapi/setvars.sh
# sh: 16: /opt/intel/inteloneapi/setvars.sh: [[: not found
# sh: 17: /opt/intel/inteloneapi/setvars.sh: [[: not found
# sh: 19: /opt/intel/inteloneapi/setvars.sh: [[: not found
# sh: 22: /opt/intel/inteloneapi/setvars.sh: Bad substitution
# Incorrect usage. is not meant to be executed directly, use source instead:
# sh: 23: /opt/intel/inteloneapi/setvars.sh: Bad substitution source
# sh: 24: exit: Illegal number: -1

 


Thanks a lot for considering,
Axel

13 Replies
GouthamK_Intel
Moderator
4,110 Views

Hi Axel,

Thanks for reaching out to us.

We would like to appreciate you for providing your feedback and suggestions.!

We are forwarding your request to the concerned internal team.


Thanks & Regards

Goutham


PaulF_IntelCorp
Employee
4,046 Views

Hi @ax3l,

Most of the issues you refer to in your post have already been addressed in the beta08 release. There are a few wrinkles regarding POSIX compliance that still need to be worked out for the top-level setvars.sh script.

Beginning with beta08, running in /bin/sh will be detected but will exit with an error message. Ultimately, when the script is fully POSIX compliant, that error will go away. Until then, sourcing the script from /bin/sh will be a limitation (not due to Bashisms, but because of differences in behavior between the various shells regarding how argument passing is handled).

Paul

ax3l
Novice
3,953 Views

Hi @PaulF_IntelCorp,

thank you and we appreciate the update. Looking forward to further updates to increase portability with sh-support.

As of oneAPI beta09, there seems to be some problem shipping and activating the dpcpp compiler.  Just in case you are not aware of this, I opened a ticket no. 04801443 in which I was now re-directed to the forum.


We use the oneAPI apt repos for tests on CI and the latest release 2021.1-beta09 does not activate the dpcpp compiler anymore with the environment scripts.

We install the following apt packages:
  https://software.intel.com/content/www/us/en/develop/articles/installing-intel-oneapi-toolkits-via-apt.html

- build-essential
-
intel-oneapi-dpcpp-compiler
- intel-oneapi-mkl

and the compiler does not come with an env/vars.sh script in the latest beta09 release, nor does source /opt/intel/oneapi/setvars.sh activate it anymore.

This looks like a packaging bug on apt on Linux or something related to those changes, can you confirm?

0 Kudos
PaulF_IntelCorp
Employee
3,947 Views

@ax3l -- I just experienced the same, myself, this morning. I think the beta09 APT packages were released prematurely and are not actually ready yet. Regardless, I'll ping the install team to confirm if that is the case.

UPDATE 2020-09-15: A bug has been filed. Sorry, I don't have insight as to when it will be fixed.

UPDATE 2020-09-16: @ax3l -- I was told that for beta releases we should only be installing at the toolkit level, not the individual component level (e.g., apt install intel-basekit or apt install intel-basekit-2021.1-beta09)

With the beta09 release there was a change to the naming of the individual compiler packages. On my Ubuntu system I see these compiler package names for beta08:

$ apt search intel-oneapi | grep beta08 | grep compiler

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

intel-oneapi-compiler-shared-2021.1-beta08/all,now 2021.1-863.beta08 amd64 [installed,automatic]
intel-oneapi-compiler-shared-32bit-2021.1-beta08/all 2021.1-863.beta08 i386
intel-oneapi-compiler-shared-common-2021.1-beta08/all,all,all,now 2021.1-863.beta08 all [installed,automatic]
intel-oneapi-compiler-shared-common-runtime-2021.1-beta08/all,all,all,now 2021.1-863.beta08 all [installed,automatic]
intel-oneapi-compiler-shared-runtime-2021.1-beta08/all,now 2021.1-863.beta08 amd64 [installed,automatic]
intel-oneapi-compiler-shared-runtime-32bit-2021.1-beta08/all 2021.1-863.beta08 i386
intel-oneapi-dpcpp-compiler/all 2021.1-863.beta08 amd64 [upgradable from: 2021.1-494.beta06]
intel-oneapi-dpcpp-compiler-2021.1-beta08/all,now 2021.1-863.beta08 amd64 [installed,automatic]
intel-oneapi-dpcpp-compiler-eclipse-cfg/all,all,all 2021.1-2198.beta09 all [upgradable from: 2021.1-863.beta08]

And these compiler package names for beta09:

$ apt search intel-oneapi | grep beta09 | grep compiler

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

intel-oneapi-compiler-shared-2021.1-beta09/all 2021.1-2198.beta09 amd64
intel-oneapi-compiler-shared-32bit-2021.1-beta09/all 2021.1-2198.beta09 i386
intel-oneapi-compiler-shared-common-2021.1-beta09/all,all,all 2021.1-2198.beta09 all
intel-oneapi-compiler-shared-common-runtime-2021.1-beta09/all,all,all 2021.1-2198.beta09 all
intel-oneapi-compiler-shared-runtime-2021.1-beta09/all 2021.1-2198.beta09 amd64
intel-oneapi-compiler-shared-runtime-32bit-2021.1-beta09/all 2021.1-2198.beta09 i386
intel-oneapi-dpcpp-compiler-eclipse-cfg/all,all,all 2021.1-2198.beta09 all [upgradable from: 2021.1-863.beta08]
intel-oneapi-dpcpp-cpp-compiler/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-compiler-2021.1-beta09/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-compiler-common-2021.1-beta09/all,all,all 2021.1-2198.beta09 all
intel-oneapi-dpcpp-cpp-compiler-pro/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-compiler-pro-2021.1-beta09/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-compiler-pro-32bit/all 2021.1-2198.beta09 i386
intel-oneapi-dpcpp-cpp-compiler-pro-32bit-2021.1-beta09/all 2021.1-2198.beta09 i386
intel-oneapi-dpcpp-cpp-compiler-pro-common-2021.1-beta09/all,all,all 2021.1-2198.beta09 all
intel-oneapi-dpcpp-cpp-compiler-pro-eclipse-cfg/all,all,all 2021.1-2198.beta09 all
intel-oneapi-dpcpp-cpp-compiler-pro-runtime-32bit/all 2021.1-2198.beta09 i386
intel-oneapi-dpcpp-cpp-compiler-pro-runtime-32bit-2021.1-beta09/all 2021.1-2198.beta09 i386
intel-oneapi-dpcpp-cpp-compiler-runtime/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-compiler-runtime-2021.1-beta09/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-pro-fortran-compiler-shared-common-2021.1-beta09/all,all,all 2021.1-2198.beta09 all
intel-oneapi-dpcpp-cpp-pro-fortran-compiler-shared-runtime-2021.1-beta09/all 2021.1-2198.beta09 amd64
intel-oneapi-dpcpp-cpp-pro-fortran-compiler-shared-runtime-32bit-2021.1-beta09/all 2021.1-2198.beta09 i386

 

p.s. My issue was unrelated to yours. It has to do with problems upgrading to beta09 against a beta08 installation. However, the recommended solution probably should be heeded in your case, as well. Uninstall an existing beta08 before installing beta09.

ax3l
Novice
3,869 Views

Thanks for the hint @PaulF_IntelCorp .

The reason I pick the component packages is that the toolkit-level packages are often too heavy for (GitHub Action/Azure/Travis) CI jobs (in terms of download size and temporary disk space used). I now updated the component that I pick to intel-oneapi-dpcpp-cpp-compiler, which works well.

ax3l
Novice
3,858 Views

Just a quick follow-up, unrelated to DPC++: is icc also still shipped with beta09? Cannot spot it in the packages anymore (last one is beta08) and it does not come with intel-basekit.

0 Kudos
PaulF_IntelCorp
Employee
3,850 Views

My understanding is that icc is shipped with the HPC toolkit, not with the Base toolkit. It's confusing, because it gets installed into the compiler folder, like dpcpp does, but not from the same source. I don't have a proper beta09 version of /opt/intel/oneapi installed at the moment so I can't easily check the contents of the packages to see what the specific package is named that includes it, since the HPC toolkit is a large meta-package.

When I get a chance to check that I'll get back to you.

0 Kudos
PaulF_IntelCorp
Employee
3,836 Views

@ax3l -- Installing intel-basekit gets you the dpcpp, icx and icpx C/C++ compilers. See query below (removed irrelevant bits):

$ egrep "bin.*(dpcpp|icx|icpx|icc|icpc)" /var/lib/dpkg/info/intel*list
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/dpcpp
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/icpx
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/icx

 And after installing intel-hpckit, which adds icc and icpc:

$ egrep "bin.*(dpcpp|icx|icpx|icc|icpc)" /var/lib/dpkg/info/intel*list
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/dpcpp
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/icpx
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/icx
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-compiler-pro-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/intel64/icc
/var/lib/dpkg/info/intel-oneapi-dpcpp-cpp-compiler-pro-2021.1-beta09.list:/opt/intel/oneapi/compiler/2021.1-beta09/linux/bin/intel64/icpc

You can see that icc is in the "compiler-pro" package.

Sorry, I'm not an expert on the compilers, but I hope this helps.

ax3l
Novice
3,810 Views

Thank you for the hint, can confirm that helps.

 

It would be great if icpc/icc could be installed separately though, because the intel-hpckit package and dependencies are too huge to be used in public CI, with limited resources (usually data diff <4GByte).

0 Kudos
ax3l
Novice
3,764 Views

Hi @PaulF_IntelCorp,

 

do you think we could give that feedback and request e.g. a intel-oneapi-icpc-cpp-compiler package?


The current package intel-basekit shipping icc/icpc is too large to be useful for CI (Azure Pipelines, GitHub Actions, etc.) for testing our software continuously for proper Intel Compiler support.

The dpcpp package is fine.

Update: actually, using the intel-oneapi-dpcpp-cpp-compiler-pro package to get ICC/ICPC installed works well as a solution! It's <800MB at the moment.

PaulF_IntelCorp
Employee
3,741 Views

Hi @ax3l :

I believe the organization of packages may not be final, so watch for further changes.

Paul

0 Kudos
PaulF_IntelCorp
Employee
3,072 Views

@ax3l  -- Your comments regarding setvar.sh and the env/vars.sh scripts are expected to be fixed in the 2021.3 release, which is the next version to be released. Unfortunately, I am not authorized to provide a release date. 

0 Kudos
PrasanthD_intel
Moderator
2,973 Views

Hi Axel,


Thanks for reporting the issue

As the issue is going to be fixed in a future version of OneAPI we are closing this thread. We will no longer respond to this thread. If you face any issues in future releases please raise a new thread. Any further interaction in this thread will be considered community only.


Regards

Prasanth


0 Kudos
Reply