parallel studio cluster install to shared directory for all users

[Wayne_Louis_H_]

I have a system with 150 nodes or so.  Each node has a shared filesystems mounted.  Each node is booted from a central image and the OS is loaded into memory so there is no persistent data on nodes.

I have the Parallel Studio XE Cluster Edition.

I would like to install the compiler to /usr/local/intel-2015 (/usr/local is one of the mount points to the shared filesystem).

I don't want the installation to be making changes to system directories (such as /etc, /usr (save for /usr/local) and I'd prefer not to have the install throw a lot of files into the home directory of the User doing the install.

I would like to know what environment variables are required to be able to work in the parallel studio as I can't really use the various scripts (compilervars, iccvars, ifortvars).  We use modules (http://modules.sourceforge.net/) to manage the user's environment because one of the requirements is that we have multiple versions of various software (including the compilers) installed at the same time and use modules to determine which one is to be used.  I understand the scripts set the variables, but modules allow for the removal of those variable settings so that another version if the studio can be used.

So, in summary, how can I install the Parallel Studio XE Cluster Edition into a shared directory (so that I only need to install it once rather than once per node) which is able to be used by all users?

What environment variables need to be set so that a user is able to use the all the features of that shared install of the Parallel Studio?

I see that there is a "--SHARED_INSTALL" option to the offline installer.  I tried that, but it still created an "intel" directory in the home of the user that ran the install.  Is the data in that directory is necessary to use the parallel studio for any user?

[Gergana_S_Intel]

Hey Wayne,

We meet again :)  Let me try to answer your questions above.

If your /usr/local is already mounted on the shared filesystem, all you have to do is follow the default prompts for an installation on the "Current node only" but make sure to change the default install directory from /opt/intel to /usr/local.  That will install the software once, only on the local node; the rest of the nodes will have that directory available via the shared mount.

If you install as root (with RPMs), the installer will handle the permissions so all users have access to the scripts in the installation path.

We've seen a lot of people use modules instead of running the provided scripts.  We have an internal feature request to our installer team to add support for modules but, alas, that time hasn't yet come.  If you'd like, I'll add your name to that request.

Unfortunately, I don't have a list of all environment variables that are being set when sourcing the scripts.  Your best bet will be to look at each script (e.g. compilervars.sh) and grab a list there.  Those scripts are created after installation so you should see what paths each env variable is being set to.  As a base, all of those scripts edit PATH and LD_LIBRARY_PATH to add pointers to the Intel libraries and runtime components.  Additionally, each tool might set additional env variables (e.g. MPI_ROOT for the Intel MPI Library) specific for its usage.

I hope this helps.  I'll check in tomorrow before I leave on vacation.  Then I'll hand this off to my colleague who can help further.

Regards,
~Gergana

[Wayne_Louis_H_]

I really *hate* to run installs as root.  Most installs tend to think that they own the machine and make various changes to system directories such as /usr/lib, /lib, /usr/include, etc and then depend on those changes having been made.  With non-root installs, the install scripts generally figure out that system changes aren't allowed.

As for parsing the scripts to find the variables -- Yeah, I tried parsing those scripts a time or two before. They call scripts which call other scripts in various other directories and its a royal pain to try to follow, much less to parse.

I inserted a "set -x" at the beginning of compilervars.sh and the output was over 200 lines long.

only including lines with an "=" still showed more than 116 lines.

I was just hoping somewhere there might be documented what variables were needed.

Do I really need to set LANGUAGE_TERRITORY, LIBRARY_PATH, LD_LIBRARY_PATH, MIC_LIBRARY_PATH, MIC_LD_LIBRARY_PATH, OLD_LIBRARY_PATH, OLD_MIC_LD_LIBRARY_PATH, OLD_LD_LIBRARY_PATH and the seemingly myriad varieties of the various environment variables touched by all those scripts?

I can do it -- I can use 'set -x' and grep to try to winnow things down.  Of course it doesn't help that you do a "set VARIABLE=blah" followed by an "export VARIABLE" in some cases and a combined "export VARIABLE=blah" in others.

The largest frustration is not knowing if I got it all right until some users tries some variation that required a variable I missed and reports the error.  The last time I tried this, *that* particular process got fairly old as it took more iterations than I want to admit.

[James_T_Intel]

Hi Wayne,

I'm taking over this from Gergana while she's out.  I don't think we have anything documenting all of the environment variables that are set.

However, I would recommend this.  Since you're using modules, start at an individual product level, rather than at the full suite level.  Each product has its own main environment variable script.  There is some overlap in these to keep in mind.  Specifically, compilervars eventually uses ippvars, mklvars, and tbbvars.

I would recommend setting all of the variables we do, as there are some situations where they will be used.  However, this isn't a strict requirement, and some may not matter, depending on your system configuration.  The primary ones I would focus on are PATH and and any *ROOT* variables.  If you have any Xeon Phi™ coprocessors installed, or intend to, I'd also include any *MIC* variables.

I'll poke around and see if we have anything better.

[KitturGanesh]

What James and Gergana point out is correct, we don't have any particular one module setting up all environment variables. The Intel Parallel Studio entails individual components within the product and each product identifies to its own *vars.sh script file, like iccvars.sh for ICC, ippvars.sh for IPP and so on. A feature request has been filed against the compiler to provide an environment module file along with existing iccvars.sh script file that exists presently, fyi.

_Kittur

[Jared_B_]

All,

I would not only like to see the documentation of environment variables set by the scripts or a required list, as we also use modules on all of our HPC systems, it would be nice to be able to have multiple installation directories for versions. I really *HATE* the way the installation works for the suite as a whole. On one of our systems, the installation of Intel 2012 was installed into a directory (/apps) which is user applications and quickly became over populated, thus we attempted to be more correct about the installation when upgrading to 2013 by using /apps/INTEL/2013. Everything seemed to go well, then I attempted to install parallel studio 2015 and the installer **FORCES** the installation directory to be /apps/INTEL/2013. Why can I not make the installation directory /apps/INTEL/2015?! The installer forces it into /apps/INTEL/2013 because that is where other 2013 tools reside. I don't want the 2015 tools in the 2013 directory to keep users from getting confused at times and I feel that it ruins the directory hierarchy system, especially with all the symlinks that the installer places between different versions. It is nearly the most unclean method of install that I've worked with. We always have multiple versions of the compilers on our system because we use modules and need to house several versions of the suite for code/performance repetition and benchmarks. I know I can always specify the path down to the explicit version number, but I would rather have 2013 be entirely separated from 2015 software, but the installer does not allow that and it is ugly.

--

ls -l

total 76K
lrwxrwxrwx  1 root root   44 Oct  9 13:09 advisor_xe -> /apps/INTEL/2013//advisor_xe_2015.1.0.367266
drwxr-xr-x 16 root root 4.0K Jan  6  2014 advisor_xe_2013
lrwxrwxrwx  1 root root   44 Oct  9 13:09 advisor_xe_2015 -> /apps/INTEL/2013//advisor_xe_2015.1.0.367266
drwxr-xr-x 15 root root 4.0K Oct  9 13:10 advisor_xe_2015.1.0.367266
drwxr-xr-x  2 root root 4.0K Oct  9 13:13 bin
lrwxrwxrwx  1 root root   16 Oct  9 13:13 composerxe -> composer_xe_2015
drwxr-xr-x  3 root root 4.0K May  3  2013 composer_xe_2013
drwxr-xr-x 13 root root 4.0K Oct 15  2012 composer_xe_2013.1.117
drwxr-xr-x 13 root root 4.0K Mar 15  2013 composer_xe_2013.3.163
drwxr-xr-x  3 root root 4.0K Jan  6  2014 composer_xe_2013_sp1
drwxr-xr-x 15 root root 4.0K Jan  6  2014 composer_xe_2013_sp1.0.080
drwxr-xr-x  3 root root 4.0K Oct  9 13:13 composer_xe_2015
drwxr-xr-x 14 root root 4.0K Jul 25 04:13 composer_xe_2015.0.090
drwxr-xr-x  4 root root 4.0K Jan  6  2014 icsxe
drwxr-xr-x  4 root root 4.0K Oct  9 13:12 imb
lrwxrwxrwx  1 root root   31 Oct  9 13:13 imb_4.0.0 -> /apps/INTEL/2013/imb/4.0.0.022/
lrwxrwxrwx  1 root root   31 Oct  9 13:13 imb_latest -> /apps/INTEL/2013/imb/4.0.0.022/
drwxr-xr-x  6 root root 4.0K Oct  9 13:08 impi
lrwxrwxrwx  1 root root   32 Oct  9 13:13 impi_5.0.1 -> /apps/INTEL/2013/impi/5.0.1.035/
lrwxrwxrwx  1 root root   32 Oct  9 13:13 impi_latest -> /apps/INTEL/2013/impi/5.0.1.035/
lrwxrwxrwx  1 root root   18 Oct  9 13:13 include -> composerxe/include
lrwxrwxrwx  1 root root   46 Oct  9 13:09 inspector_xe -> /apps/INTEL/2013//inspector_xe_2015.1.0.366509
drwxr-xr-x 16 root root 4.0K Jan  6  2014 inspector_xe_2013
lrwxrwxrwx  1 root root   46 Oct  9 13:09 inspector_xe_2015 -> /apps/INTEL/2013//inspector_xe_2015.1.0.366509
drwxr-xr-x 15 root root 4.0K Oct  9 13:09 inspector_xe_2015.1.0.366509
lrwxrwxrwx  1 root root   14 Oct  9 13:13 ipp -> composerxe/ipp
drwxr-xr-x  6 root root 4.0K Oct  9 13:08 itac
lrwxrwxrwx  1 root root   32 Oct  9 13:13 itac_9.0.1 -> /apps/INTEL/2013/itac/9.0.1.033/
lrwxrwxrwx  1 root root   32 Oct  9 13:13 itac_latest -> /apps/INTEL/2013/itac/9.0.1.033/
lrwxrwxrwx  1 root root   14 Oct  9 13:13 lib -> composerxe/lib
lrwxrwxrwx  1 root root   14 Oct  9 13:13 man -> composerxe/man
lrwxrwxrwx  1 root root   14 Oct  9 13:13 mkl -> composerxe/mkl
lrwxrwxrwx  1 root root   16 Oct  9 13:13 mpirt -> composerxe/mpirt
drwxr-xr-x  5 root root 4.0K Oct  9 13:12 parallel_studio_xe_2015
lrwxrwxrwx  1 root root   14 Oct  9 13:13 tbb -> composerxe/tbb
lrwxrwxrwx  1 root root   52 Oct  9 13:09 vtune_amplifier_xe -> /apps/INTEL/2013//vtune_amplifier_xe_2015.1.0.367959
drwxr-xr-x 21 root root 4.0K Jan  6  2014 vtune_amplifier_xe_2013
lrwxrwxrwx  1 root root   52 Oct  9 13:09 vtune_amplifier_xe_2015 -> /apps/INTEL/2013//vtune_amplifier_xe_2015.1.0.367959
drwxr-xr-x 19 root root 4.0K Oct  9 13:09 vtune_amplifier_xe_2015.1.0.367959

--

The above output is absolutely silly as naming conventions are not even consistent!

[Maciej_Cytowski]

Hello,
recently I've written modulefiles setting up the environment for Parallel Studio
XE Cluster Edition. They mimic the Linux bash scripts (I assume csh scripts do
exactly the same thing) included with the Parallel Studio software very
precisely (with one exception - more on that later).

The scripts are pure Tcl (via Environment Modules). No shell scripts are called
during the script evaluation and no system/exec keywords are used (except two
calls of exec necessary to check system version of gcc needed by Threading
Building Blocks script). The whole setup is shell independent.

Mac OS is not officially supported by the Environment Modules team, but my
modulefiles should work there anyway (I haven't tested it due to problems with
Tcl on Mac I use, but I've scoped forums - some people actually use Environment
Modules on Macs, so I'll return to pinpoint the problem in some time).

Since the scripts included in Parallel Studio package are copyrighted,
I'm asking for permission to upload my modulefiles here (they are, in fact,
Intel's bash scripts rewritten in Tcl).

regards,
Szymon Jaranowski

[James_T_Intel]

For more information on creating module files, please see https://software.intel.com/en-us/articles/using-environment-modules-with-the-intel-compiler.

[Maciej_Cytowski]

Thanks, it works for me (well, almost - I have some issues with older products) but this might be problem on heterogeneous clusters, since the method only loads environment for one, certain realization of the script (I think MICs are unusable then, i.e. if you have single, shared modulefile loading the environment).

Environment Modules are considered final (3 years with no updates) and, as far as I'm concerned, they are very popular in HPC (I guess mostly due to easy maintenance of multiple versions, as mentioned in the article from the post above). It would be nice, if modulefiles could be provided by Intel - at least for Cluster products...

OK, last thing - quote from the article:

There are two ways you can create modulefiles, the obvious way is by hand, using the provided script files *vars.[sh | csh] as reference. DO NOT DO IT THIS WAY [...]. Unraveling this nest of scripts is nearly impossible and error prone.

That's a bit exaggerated. It was laborious, but it was completely doable (I took Parallel Studio XE Cluster Edition 2015 Initial Release as an example). I don't know how the scripts are parsed during the installation phase, but it is possible, that the modulefiles would be parsed by the same parser.