Intel Phi on Ubunt 14.04

[Nagy-Egri__Mate_Fere]

Hi!

I am the admin of a GPU cluster, and the OS of choice is 64-bit Ubuntu Server 14.04. It is rather strange that Intel provides Windows binaries sooner than they would port the RedHat implementation over to the Debian world. I will not have a single not on a completely different ecosystem, so I decided to compile the drivers on Ubuntu, as I've seen that it is not impossible.

Getting the mpss module to build was fairly simple (so I even have the feeling I did something wrong). The source code needed minimal change, as there was a kernel version check that triggered a compile error due to too many arugments being passed to a system function. Other than this, the module got built and it could be loaded and /dev/mic0 exists.

I installed the OpenCL SDK and got clinfo by installing the AMD APPSDK as well. However, the Intel platform only holds the CPU device.

Before fetching the SDK, I wanted to compile miccheck to obtain an invaluable tool for debugging faulty installations, however I could not find out how on earth I should install and run it. (I am an absolute Python noob.) I don't even know which package I should build from source, miccheck or miccheck-bin? The Makefile holds hardcoded version for Python being 2.6, which must be modified to 2.7 on Ubuntu 14.04. Then the modules compile, but the resulting Python scripts wont run. How can I parametrize Make so that I can just call miccheck from anywhere and have the tests execute?

Is there anything else worth building and installing?

[Frances_R_Intel]

Recompiling the kernel modules really isn't all that hard, generally, so I don't think the fact that it wasn't overly painful should cause you to worry. I am actually surprised that you needed to make source code changes rather than just do a straight rebuild. The mic kernel module is available for RHEL 7 which, I believe, uses a 3.10.0 base Linux kernel. It might benefit other Ubuntu folks if you posted the change you make here. 

It shouldn't be necessary to rebuild miccheck for different versions of operating system. Only the kernel modules require rebuilding. When you install the MPSS, miccheck gets copied to /usr/bin.

I recently ran into another user having problems with OpenCL not detecting the coprocessors. In that case, it looks like there were some missing library files. Which version of the Intel OpenCL runtime are you running? The OpenCL runtime installs in /opt/intel by default. Can you check to see if the lib64 directory from that install location is in your LD_LIBRARY_PATH and the mic directory from that install location in your MIC_LD_LIBRARY_PATH?

[Nagy-Egri__Mate_Fere]

Hello Frances,

here is what I've done:

1. I downloaded the tarball containing the mpss-3.4 sources.
2. I extracted the archive mpss-modules-3.4.
3. In order to get things to compile I had to
   1. vim $(HOME)/mpss-3.4/src/mpss-modules-3.4/host/linux.c
   2. The function call "mic_ctx->sysfs_state = sysfs_get_dirent(..." has an #ifdef of some version that seemed to misbehave for me, so I removed it.
4. Then I issued "sudo make -j12 MIC_CARD_ARCH=k1om install" and saw

   1. mnagy@phi:~/mpss-3.4/src/mpss-modules-3.4$ sudo make -j12 MIC_CARD_ARCH=k1om install
      [sudo] password for mnagy:
      make -C /lib/modules/3.13.0-32-generic/build M=/home/mnagy/mpss-3.4/src/mpss-modules-3.4 modules_install \
                      INSTALL_MOD_PATH=
      install -d /usr/local/etc/sysconfig/modules
      install -d /lib/modules/3.13.0-32-generic
      install -m644 Module.symvers /lib/modules/3.13.0-32-generic/scif.symvers
      install mic.modules /usr/local/etc/sysconfig/modules
      install -d /usr/src/linux-headers-3.13.0-32-generic/include/modules
      install -m644 include/scif.h /usr/src/linux-headers-3.13.0-32-generic/include/modules
      install -d /usr/local/etc/modprobe.d
      install -m644 mic.conf /usr/local/etc/modprobe.d
      install -d /usr/local/etc/udev/rules.d
      install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules
      make[1]: Entering directory `/usr/src/linux-headers-3.13.0-32-generic'
        INSTALL /home/mnagy/mpss-3.4/src/mpss-modules-3.4/mic.ko
      Can't read private key
        DEPMOD  3.13.0-32-generic
      make[1]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
      

I'm still not seeing miccheck under /usr/bin/, but /dev/mic0 exists, and the kernel module seems to be loaded. Is there something that I missed? Do I need other defines while building? If installing the MIC driver were so easy, why does Intel not provide .deb packages? Would giving official .deb packages also imply full Debian support? The courtesy of making .deb packages and saying that they are unsupported and they are provided as is would be highly appreciated by the community. At least a How-To on getting a minimalistic dev environment running on Ubuntu.

[Frances_R_Intel]

Let me ask a question. There is a problem with our documentation that we are working to address - did you start with the steps in the readme.txt document? It has the actual installation steps.  That is where miccheck should have been installed.

[Nagy-Egri__Mate_Fere]

As far as the readme.txt goes, it is pretty much completely irrelevant to an Ubuntu host system. Even compiling the kernel is not possible the way it states. (the rpmbuild tool is looking for packages that are not inside the Ubuntu repo) If I take the readme.txt as general guidelines, it is still not trivial what I should be doing. (I am noobish, I am aware of that)

I converted the kernel module rpm to deb and installed it, and so I did with the rest of the rpm packages found in the root of mpss-3.4_Linux.tar. miccheck says the driver is not loaded, however syslog shows no error messages, /dev/mic0 exists, and modprobe mic exits with exit code 0.

Then I thought that recompiling the kernel must surely be in order, thus I extracted the deb package of the driver and rebuilt it with the aforementioned source hack. I have overwritten the mic.ko from the previous install with the new one tailored to this kernel version. (A bold move, which might be outright brutish, I am aware)

Could you tell me how to get as far as having clinfo recognize the MIC devices all the way from having a fresh install? Dealing with the rpm packages is surely not the way to go.

Sidenote: How come cross-platform projects, such as drivers and tools like these do not use CMake and CPack to compile sources and create .rpm/.deb/.msi packages? Boy, I would love to live in a world like this.

[JJK]

did you install the Intel OpenCL package on the host system? for 'clinfo' to see the Intel MIC the OpenCL ICD files must be installed.

You can download the latest Intel OpenCL package from   https://software.intel.com/en-us/articles/opencl-drivers

[Nagy-Egri__Mate_Fere]

I did install it. The /etc/OpenCL/vendors is correctly populated.

[JJK]

the mpss daemon needs to be running on the host as well ; did you start/port this yet? you might be able to get away with extracting the binaries from the RPMs and starting those...

[Frances_R_Intel]

I can sense your frustration but I am afraid the only feasible solution I can come up with is to either install rpm or use a program like alien to convert the rpm files to deb files, which I guess is what you have done. Now that you have installed all the rpm files from the mpss-3.4 root directory, is miccheck in the right place?

When you built the mpss-modules from the contents of the rpm file, did you still comment out the call to  sysfs_get_dirent in the linux.c file? The sysfs_get_dirent function should be in your system libraries, so there shouldn't be any #ifdef that don't work. Commenting out that line is a bad thing.

After you execute 'modprobe mic', miccheck fails test1:  'Check mic driver is loaded ... fail' ? And at this point your are stopped cold? You haven't run 'micctrl --initdefaults' yet?

Obviously, until this is resolved, the OpenCL issue is going to need to wait. And once it is resolved, the OpenCL issue will probably resolve itself as well.

So what now? If you are still commenting out that line in the linux.c file, please see if you can find the source of the #ifdef problem and fix it. (I'll try to help if I can.) Apparently modprobe believes it is loading the mic module. Does it show up with lsmod? What does modinfo say about the module file? What happens if you use  insmod instead of modprobe to load the module? Using insmod instead of modprobe is kind of brute force. I have seen modules load with insmod when they wouldn't with modprobe. This usually happened because there was a mismatch between the kernel version and the module version which could be the case here.

[Nagy-Egri__Mate_Fere]

Hi!

• The sysfs_get_dirent is among the system libraries. The #ifdef in the kernel code is about the number of parameters passed to the function. The sysfs_get_dirent that is inside Ubuntu 14.04 uses the interface with only 2 parameters, and the NULL that is #ifdef-ed inside the driver source must be using the 2 parameter version, but it fails to detect which one it should be using. I didn't comment out the entire function, just fixed it's signature to match the one found in Ubuntu
• I installed the RPM-s and miccheck and micctrl are in the right places.
• Indeed, I have not run 'micctrl --initdefaults' yet. Now I have and it argued about libmpssconfig.so not being found that is because it was installed under /usr/lib64 which is again a portability issue. (Ubuntu uses /usr/lib and /usr/lib32) I created an ldconfig entry under /etc/ld.so.conf.d/mic.conf to include the libdir where the rpm-s installed their binaries. micctrl ran just fine.
• miccheck however still says the driver is not loaded, even though lsmod reports 'mic_host' being loaded (as well as vringh).

mnagy@phi:/usr/lib64$ modinfo mic_host
filename:       /lib/modules/3.13.0-32-generic/kernel/drivers/misc/mic/host/mic_host.ko
license:        GPL v2
description:    Intel(R) MIC X100 Host driver
author:         Intel Corporation
srcversion:     2BC8077D255BA3F33703230
alias:          pci:v00008086d0000225Esv*sd*bc*sc*i*
alias:          pci:v00008086d0000225Dsv*sd*bc*sc*i*
alias:          pci:v00008086d0000225Csv*sd*bc*sc*i*
alias:          pci:v00008086d0000225Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000225Asv*sd*bc*sc*i*
alias:          pci:v00008086d00002259sv*sd*bc*sc*i*
alias:          pci:v00008086d00002258sv*sd*bc*sc*i*
alias:          pci:v00008086d00002257sv*sd*bc*sc*i*
alias:          pci:v00008086d00002256sv*sd*bc*sc*i*
alias:          pci:v00008086d00002255sv*sd*bc*sc*i*
alias:          pci:v00008086d00002254sv*sd*bc*sc*i*
alias:          pci:v00008086d00002253sv*sd*bc*sc*i*
alias:          pci:v00008086d00002252sv*sd*bc*sc*i*
alias:          pci:v00008086d00002251sv*sd*bc*sc*i*
alias:          pci:v00008086d00002250sv*sd*bc*sc*i*
depends:        vringh
intree:         Y
vermagic:       3.13.0-32-generic SMP mod_unload modversions
signer:         Magrathea: Glacier signing key
sig_key:        5E:3C:0F:9C:A6:E3:65:43:53:5F:A2:BB:5B:70:9E:84:F1:6D:A7:C7
sig_hashalgo:   sha512
mnagy@phi:/usr/lib64$

I am unsure how to go about using insmod. (sysadmin in training)

mnagy@phi:/usr/lib64$ sudo insmod /lib/modules/3.13.0-32-generic/kernel/drivers/misc/mic/host/mic_host.ko
insmod: ERROR: could not insert module /lib/modules/3.13.0-32-generic/kernel/drivers/misc/mic/host/mic_host.ko: File exists

[Nagy-Egri__Mate_Fere]

I would like to bump this thread in hope of getting another go at fixing the problem.

I am close to ditching Ubuntu for MIC, but essentially having a single node running an RPM based distro, while the entire cluster runs Ubuntu will most certainly be the source of many aggravation in the future. If this would be the case, I'd definitely install SLC (Scientific Linux CERN), which is an SL variant, but doing so will again bring back my haunted dreams starred by Anaconda (the installer of SLC). Any subsequent porting efforts of cluster installation with Cobbler and management with Puppet will be doubled by having just a single machine on different terms as the rest.

Any guesses how to load the MIC module? All help is greatly appreciated.

[TaylorIoTKidd]

Have you looked at https://software.intel.com/en-us/forums/topic/368002 ?

The thread addresses the changes that were needed to get MPSS working with Ubuntu 12.04.

As an aside, what are the advantages of using Ubuntu versus a more enterprise oriented OS such as CentOS/RHEL?

Regards
--
Taylor
 

[Nagy-Egri__Mate_Fere]

I will try to incorporate what's in the linked thread to my attempts. I'll report back should I have relevant new information (or problems).

As for the sidenote. I'd have a question as an answer: what's the disadvantage? (Aside from MPSS not working that is.)

I started off as a CERN sysadmin for a Tier-2 site in Hungary, and naturally for the CERN grid, SLC is the falvour of choice (Scientific Linux CERN), which is a nice Scientific Linux (RedHat based distro) variant, so most people around me tend to use that. I was an exchange student twice in the Helmholtz-Zentrum Dresden, where I had the chance to work together with IT guys, specifically a sysadmin far better than me (got his PhD in software engineering if I'm not mistaken, though his thesis was highly related to mathematics), and there they had a cluster somewhat bigger than our site, all machines running Ubuntu Server, and that was 5 years ago. (Distributions usually don't age, they mature. At least that's my experience.) And I thought to myself: hey, Ubuntu might not just be a desktop distro. They had high-quality services and a reliable and flexible infrastructure. I was falttered how promptly the guy installed stuff on the cluster, even crazy stuff not in the distro, in a non-intrusive way towards other users.

So when I was put to charge of a new cluster, all new hardware, all new... whatever I wanted, I decided to tackle Ubuntu, as it was familiar as a desktop, plus it has a humongous community, and you can google pretty much any problem you run into, and the first 50 hits usually solves it in 10 different ways. I have not regretted this decision ever since. MPSS is the first real challange.

To rephrase the question: if CentOS or SL is considered a 'serious' cluster distro, then why make a Windows driver before a Debian-like one? Windows is clearly as much a desktop environment as Ubuntu is. It is only a small niche of enterprise companies that use Windows cluster for HPC. (I mean, just look at the number of HPC libraries that have Windows ports or variants.) I am truly amazed at how Ubuntu Server is considered second-rate. (In the linked thread you claimed SLES and RHEL are more commercially supported. Is Canonical a charity organization?)

As a sidenote, as an everyday workstation, I am a Windows user, and will remain one, as stuff like this on Linux give me the creeps.

[Frances_R_Intel]

As to why a port to Windows was done but not a port to a Debian-style Linux such as Ubuntu -

In theory, given that Linux is open source as is most of the code in the MPSS, it should be possible for a power user to port the MPSS to different flavors of Linux. The port to Windows is not necessarily something the user could do themselves. Unfortunately, as you have found, the port to Ubuntu is not without issues, starting with the MPSS use of rpm files and going on to kernel differences.

In our little group, we do not have a system running Ubuntu. I have asked our lab person to put Ubuntu on a spare disk that I can swap into one of our systems to play with. Unfortunately this will need to be on an as-time-allows basis. I am still hopeful that one of the people already using Ubuntu will be able to respond here. 

[Nagy-Egri__Mate_Fere]

There is a great article found here:

http://arrayfire.com/getting-started-with-the-intel-xeon-phi-on-ubuntu-14-04linux-kernel-3-13-0/

However as I have posted there, after flashing the card, the kernel module won't load, even though it is exactly the same version as miccheck was complaining about (390).

[Nagy-Egri__Mate_Fere]

Hi!

Rebuilding the kernel module and reinstalling solved the issue, and now everything is installed as it should, both miccheck and micctrl say that everything is fine, the ICD is installed. The ultimate goal is to run OpenCL apps on the MIC, however OpenCL does not see the deivce. The ICD is installed as it should be, apps and a simple clinfo finds the Intel OpenCL platform, however it does not see the accelerator card. Should the MICs be found as GPUs, or as CL_DEVICE_TYPE_ACCELERATOR?

[JJK]

the Xeon Phi is seen as type CL_DEVICE_TYPE_ACCELERATOR ; you do need to install the opencl-1.2-intel-mic package , otherwise the MIC is not seen by the ICD.

The OpenCL samples from Intel, AMD and NVidia should work , provided that they check for CL_DEVICE_TYPE_ACCELERATOR (in some cases this is a one-line code change )

[Nagy-Egri__Mate_Fere]

I did run the installation scripts, and everything under /opt/intel seems to be populated correctly (although very not Ubuntu like). I see these things installed:

intel-opencl-1.2-4.6.0.92  intel_sdp_products.db  ism  mic  opencl-1.2-4.5.0.8

I tried converting the opencl-1.2-intel-mic.rpm package uing alien by hand and installing it along with the other 3 packages that ship with opencl_runtime_14.2_x64_4.5.0.8.tgz, but 2 of the 4 package failed their post installation scripts: opencl-1.2-base-pset and opencl-1.2-intel-cpu. Is there no need for these packages if the install.sh seemingly ran fine and install seems valid?

[JJK]

hi,

I've checked my OpenCL installation (Scientific Lniux 6.6) and it looks like the opencl-1.2-intel-cpu-4.5.0.8-1.x86_64 is crucial . This package sets up the .icd file and installs the appropriate ocl libs. On my box:

$ ls -al /etc/OpenCL/vendors/intel64.icd  
lrwxrwxrwx 1 root root 42 Oct  8 11:48 /etc/OpenCL/vendors/intel64.icd -> /etc/alternatives/opencl-intel-runtime-icd

$ ls -al /etc/alternatives/opencl-intel-runtime-icd
lrwxrwxrwx 1 root root 45 Oct  8 11:48 /etc/alternatives/opencl-intel-runtime-icd -> /opt/intel/opencl-1.2-4.5.0.8/etc/intel64.icd

$ cat /opt/intel/opencl-1.2-4.5.0.8/etc/intel64.icd
/opt/intel/opencl-1.2-4.5.0.8/lib64/libintelocl.so

perhas if you replicate this on your setup the OpenCL install might work.

[nealmcb]

Ubuntu Server is the biggest distribution in many regions of the cloud.  E.g. it is the #1 OS for docker development, and has been a leader in low-power server support.

It certainly deserves good support from Intel for OpenCL and the like, so I'm glad you're making some progress here.

When might we see a clean .deb for installs?

And I'd love to see it supported on desktop machines also (Intel core with HD 4000 et al.)

[Minh_H_]

Hi,

I was successfully compiled and installed whole MPSS 3.4.1 on Ubuntu 14.04. However I got problem with scif communication under normal user mode, which I posted in
https://software.intel.com/en-us/forums/topic/539636
scif_open: Permission denied. It works fine under sudo mode. It seems that this problem specific to Ubuntu not Centos.

Any advice please?