Installing and setting up oneAPI base kit on GitHub Actions and general feedback

[inakleinbottle]

Ok so I recently posted about working on a CI/CD pipeline for a library that I maintain that makes use of MKL/LAPACK. I'm trying to build Python wheels for the library on 32 and 64 bit Windows/Linux and MacOS (just x86 for now, since I can't even get this working).

I have many, many problems but most revolve around MKL. I'm just going to enumerate them here, since I feel the scale of these problems would be diluted if I separate them. Starting from the top, I've had a pretty horrible experience downloading MKL and actually getting it into the environment, and I haven't managed to get it working yet.

1) The download links for command line downloads are way too hard to find on the Intel website. The download page does not obviously give the download link and there doesn't seem to be anywhere in the documentation that lists them either. In the end I had to look in the oneapi-ci repository where (some) of the download links are listed. Why are is there not a listing of these URLs in the documentation, it seems like a very simple thing.

2) The command line installer for the oneAPI base toolkit does not function properly on Windows. This presumably stems from from the GUI first philosophy of Windows, and the fact that one has to install several 100MBs of Qt framework libraries along with the installer. The components flag does not function as advertised in the documentation on Windows; specifying more than one component does nothing and causes the installer to install everything. The config file option sort-of works, but this is very poorly documented and isn't a very helpful solution in a headless environment. Related, but still separate: the documentation for command line on Windows doesn't seem to be written for Windows and instead seems to be basically the Linux instructions with some names changed. These platforms are very different.

2) The installation of MKL (both architectures) and TBB components of oneAPI on Windows takes far too long. On the Github actions runner that I'm using, the Windows installation of MKL takes 20 minutes! (Compare this to a similar installation on MacOS that takes 2 minute.) Why on earth does it take so long? Time is money on these action runners. Also, I would love an option to skip installing all the Installer and other stuff that is not useful on a CI/CD machine. Here's the log from a Windows build demonstrating this https://github.com/datasig-ac-uk/esig/runs/5986615873 compared to a recent install on Mac https://github.com/datasig-ac-uk/esig/runs/5962203538. I am using the offline installer on Windows rather than the online installer on Mac, but there is no appreciable difference in time when using the online intaller.

3) The setvars.bat script does not seem to set environment variables correctly on Windows. Moreover, it would be really nice to have a .sh version since GitHub action runners come with Bash which is greatly superior to command prompt. Also, sourcing these scripts inside the runner is not easy because each step of the pipeline is a separate process and environment variables are not so easily persisted. Moreover, it's quite hard to find out exactly which environment variables need to be set on each platform (I don't have ready access to a Windows or MacOs machine to test install MKL and look) so I can do some manual steps to propagate those variables. (Logs from a recent run showing that variables are not set https://github.com/datasig-ac-uk/esig/runs/5993950348 after `@call`ing the setvars.bat script. I also tried various other combinations of invocations to try and get this to work.)

4) The CMake config file for MKL does not reliably detect the version of MKL installed in the same directory on Windows. After setting MKLROOT to the head of the install file C:\Program Files (x86)\Intel\oneAPI\mkl\latest, I used

find_package(MKL CONFIG) 

in my CMakeLists.txt and strangely it found libiomp5md.lib stub from the LLVM install (presumably bundled with Visual Studio on the runners). Here's there relevant log section https://github.com/datasig-ac-uk/esig/runs/5986615873?check_suite_focus=true#step:10:292 (Note that findBLAS/findLAPACK can't find MKL either, which is also strange. They also fail on MacOS for some reason, but this is a CMake issue.) I appreciate that this is also partly a CMake problem, but it seems unnecessary fragile for the MKL script within the same directory as the binaries and headers to fail. I've tried many dozens of configurations setting MKLROOT, CMAKE_PREFIX_PATH, the MKL_ROOT CMake variable. Might I suggest making a CMake config for oneAPI base that defines targets for all components of oneAPI. This would make including oneAPI much easier for many people.

I understand completely that my use case is not really indicative of most users of oneAPI, but going forward I think more library maintainers will want to make use of its facilities and at the moment this is just far too difficult. I appreciate any efforts to address some of these issues.

[VidyalathaB_Intel]

Hi Sam,

Thanks for reaching out to us.

Could you please let us know how you are installing MKL in CI/CD pipeline and elaborate a bit more on the CI/CD machine and GitHub action runners? Please let us know the steps that you are following in this case.

Regarding the findBLAS and findLAPACK errors, there is a recent thread that got resolved when doing it in Intel oneAPI command prompt

https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/How-to-link-the-BLAS-library-in-mkl-to-compile-other-packages/m-p/1373890#M32962

But this issue is just on a local windows machine. So it would be a great help if you explain a bit more which helps us in better understanding.

Regards,

Vidya.

[inakleinbottle]

The specs for the GitHub action runners are listed here: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners.

I have a caching installation setup, but the installation is handled by the following code:

curl -v -o /d/a/intel-mkl-installer.exe "$URL"
/d/a/intel-mkl-installer.exe -s -x -f /d/a/webimage_extracted --log extract.log 
rm -v /d/a/intel-mkl-installer.exe
/d/a/webimage_extracted/bootstrapper.exe -s --action install --config ./build/oneapi-install-config-x64-windows.ini --log-dir=.
curl -v -o /d/a/intel-mkl-installer-x86.exe "$URL32"
/d/a/intel-mkl-installer-x86.exe -s -x -f /d/a/webimage_extracted_x86 --log extractx86.log
rm -v /d/a/intel-mkl-installer-x86.exe 
/d/a/webimage_extracted_x86/bootstrapper.exe -s --action install --eula=accept -p=NEED_VS2017_INTEGRATION=0 -p=NEED_VS2019_INTEGRATION=0 -p=NEED_VS2022_INTEGRATION=0 --log-dir=. --ignore-errors

 where the URL and URL32 environment variables are set as

URL: https://registrationcenter-download.intel.com/akdlm/irc_nas/18599/w_BaseKit_p_2022.1.3.210_offline.exe
URL32: https://registrationcenter-download.intel.com/akdlm/irc_nas/18599/w_BaseKit_32bit_p_2022.1.3.44_offline.exe

These are the URLs that I commented are very difficult to find in the documentation. I had to get them from the https://github.com/oneapi-src/oneapi-ci repository.

The install config I'm using for the main toolkit is 

eula=accept
components=intel.oneapi.win.mkl.devel:intel.oneapi.win.tbb.devel:intel.oneapi.win.dpl
p="NEED_VS2017_INTEGRATION=0 NEED_VS2019_INTEGRATION=0 NEED_VS2022_INTEGRATION=0"

I made a little progress on the CMake side of things yesterday. I came to the conclusion that the standard FindBLAS and FindLAPACK modules don't work to find the OneAPI distribution of MKL. To combat this, I've borrowed updated FindBLAS/FindLAPACK modules from a very recent pull request on CMake that looks for the MKLConfig. A combination of using these updated find modules and setting CMAKE_PREFIX_PATH to the MKL root, I managed to to find MKL successfully. In the process I uncovered a slight bug that you might consider addressing.

In the MKLConfig.cmake on line 241 you use a string substitution

string(REPLACE "\\" "/" MKL_ROOT ${MKL_ROOT})

to replace Windows style string with CMake style strings. Unfortunately, the MKL_ROOT variable can fail to be set if the MKLROOT environment variable is set but empty. In this case, the cmake fails with a rather unhelpful error message. I'd suggest modifying the if statement on the lines above to check if the environment variable is not set or if it is set but empty. (I don't have the latest file to hand or I'd show you exactly the replacement I suggest.) Something like

if (DEFINED ENV{MKLROOT} AND ENV{MKLROOT})
   set(MKL_ROOT $ENV{MKLROOT})
else()
   ... # fallback logic
endif()
string(REPLACE "\\" "/" MKL_ROOT "${MKL_ROOT}")

 (Also including the quotations around the expanded variable will at least prevent the unhelpful CMake error.)

[inakleinbottle]

Ok, so yet more two steps forward one step back. I'm using the MKLConfig.cmake file successfully now after a lot of playing with the environment variables. I had some more frustrating problems with MacOs where the command I was using to unpack and install MKL was failing unexpectedly. After several iterations, I found moving into the bootstrapper.app/Contents/MacOs folder and then calling install.sh as is actually described in the documentation. Quick note here, the top of the install instructions on MacOs refers to m_BaseKit_<version>.sh which is plainly copied from the install instructions on Linux but is actually not how the online/offline package is distributed for MacOs (it comes as a dmg file).

I started putting the Linux pipeline together yesterday. On Linux, the builds take place in separate docker containers (running CentOS) for 64 and 32 bit architectures. I'm using YUM to install MKL inside these docker containers. This works fine in the 64 bit docker container but in the 32 bit container, the install fails. I've tried installing both intel-oneapi-mkl-devel and the intel-oneapi-mkl-devel-32bit packages but neither works. Both give an error "No package ... available". (See https://github.com/datasig-ac-uk/esig/runs/6040159168?check_suite_focus=true#step:10:3594 for intel-oneapi-mkl-devel and https://github.com/datasig-ac-uk/esig/runs/6040397857?check_suite_focus=true#step:10:3595 for intel-oneapi-mkl-devel-32bit.) Is there a different repository I need to use for 32 bit operating systems?

I believe an older version of MKL might work (currently testing) but ideally I'd like to keep the installation routine as up-to-date and routine as possible.

[VidyalathaB_Intel]

Hi Sam,

>>I'm using the MKLConfig.cmake file successfully

Thanks for letting us know.

For installation-related questions, we suggest you post your concerns in the below forum link from where you can get better assistance.

https://community.intel.com/t5/Intel-oneAPI-Registration/bd-p/registration-download-licensing-instal

Regards,

Vidya.

[VidyalathaB_Intel]

Hi Sam,

>>I uncovered a slight bug that you might consider addressing.

We are working on this issue. we will get back to you soon.

Thanks for your patience.

Regards,

Vidya.

[Robert_C_Intel]

> Why are is there not a listing of these URLs in the documentation, it seems like a very simple thing.

You can get the URL's for standalone installers from this page: https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html

I right click on the link for the tool I want and copy/paste.