Intel Phi on Ubunt 14.04

[Nagy-Egri__Mate_Fere]

Hi!

I am the admin of a GPU cluster, and the OS of choice is 64-bit Ubuntu Server 14.04. It is rather strange that Intel provides Windows binaries sooner than they would port the RedHat implementation over to the Debian world. I will not have a single not on a completely different ecosystem, so I decided to compile the drivers on Ubuntu, as I've seen that it is not impossible.

Getting the mpss module to build was fairly simple (so I even have the feeling I did something wrong). The source code needed minimal change, as there was a kernel version check that triggered a compile error due to too many arugments being passed to a system function. Other than this, the module got built and it could be loaded and /dev/mic0 exists.

I installed the OpenCL SDK and got clinfo by installing the AMD APPSDK as well. However, the Intel platform only holds the CPU device.

Before fetching the SDK, I wanted to compile miccheck to obtain an invaluable tool for debugging faulty installations, however I could not find out how on earth I should install and run it. (I am an absolute Python noob.) I don't even know which package I should build from source, miccheck or miccheck-bin? The Makefile holds hardcoded version for Python being 2.6, which must be modified to 2.7 on Ubuntu 14.04. Then the modules compile, but the resulting Python scripts wont run. How can I parametrize Make so that I can just call miccheck from anywhere and have the tests execute?

Is there anything else worth building and installing?

[Minh_H_]

It is resolved now. See https://software.intel.com/en-us/forums/topic/539636

[jofre]

Dear Minh,

I (and I do not think I am alone) would appreciate if you could sumarize how did you manage to install the latest MPSS 3.4.3 in an Ubuntu 14.04 machine.

For info, after donwloading the source, and do:

$ sudo make MIC_CARD_ARCH=k1om

It does not compile (it starts ok, until a cryptic, to me at least, error appears:

error: too many arguments to function ‘sysfs_get_dirent’
     "state");

Thanks in advance

[jofre]

Nevermind.

- Fresh installation of Lubuntu 14.04.2 (installs kernel 3.16.x)

- sudo apt-get install linux-image-3.13.0-46-generic linux-headers-3.13.0-46-generic

change grub to boot from the 3.13 kernel and re-boot

Download source form intel site and extract mpss-modules-3.4.3.tar.bz2 file

open the file:  path/mpss-3.4/src/mpss-modules-3.4/host/linux.c

comment line 335 to 339 (as suggested above by meterohead)

now you can compile with:

$sudo make MIC_CARD_ARCH=k1om install

After that, just follow the instructions from

http://arrayfire.com/getting-started-with-the-intel-xeon-phi-on-ubuntu-14-04linux-kernel-3-13-0/

I hope it helps someone.

Jofre

[Minh_H_]

jofre wrote:

Dear Minh,

I (and I do not think I am alone) would appreciate if you could sumarize how did you manage to install the latest MPSS 3.4.3 in an Ubuntu 14.04 machine.

For info, after donwloading the source, and do:

$ sudo make MIC_CARD_ARCH=k1om

It does not compile (it starts ok, until a cryptic, to me at least, error appears:

error: too many arguments to function ‘sysfs_get_dirent’
     "state");

Thanks in advance

Sorry for late reply.  Here are what I did. I download it from git Hub

git clone https://github.com/pentschev/mpss-modules.git

The following is step-by-step.

----------------------------------------------------------------------------------------------------------------------------------------------

1. Install Ubuntu 14.04.1
   Kernel:  Linux hal 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
2. Xeon Phi MPSS Installation
Ref: https://software.intel.com/en-us/forums/topic/532329
http://arrayfire.com/getting-started-with-the-intel-xeon-phi-on-ubuntu-14-04linux-kernel-3-13-0/
2.1 Install kernel modules
    2.1.1 Compile kernel module (Intel MPSS 3.4 for Linux 3.13.0)
        git clone https://github.com/pentschev/mpss-modules.git
        required:
    * build-essential
    * linux-headers-generic
         sudo apt-get install build-essential linux-headers-generic

        make
        sudo make install
        sudo depmod
    sudo rmmod mic_host
        #add blacklist mic_host  to /etc/modprobe.d/blacklist-mic-host.conf
        sudo modprobe mic

    2.1.2: Convert RPMs packages from MPSS 3.4.2 to .deb
        sudo apt-get install alien
    #download mpss-3.4.2-linux.tar from Intel's website
        tar xvf  mpss-3.4.2-linux.tar
        cd mpss-3.4.2
        sudo alien --scripts *.rpm
        dpkg -i *.deb
        #create new file /etc/ld.so.conf.d/zz_x86_64-compat.conf with content /usr/lib64
        sudo ldconfig
2.2
2.3 Flash: normaly flash required by new MPSS. Should check the current firmware version of Xeon Phi before flash new one.
    run micinfo to check C0 stepping
    To run miccheck:
        sudo ln -s /usr/bin/lspci  /sbin/lspci
    sudo miccheck

     /opt/intel/mic/bin/micflash -update -smcbootloader -device all

2.4. Config mic
     micctrl --initdefaults
     sudo cp /etc/mpss/mpss.ubuntu  /etc/init.d/mpss
     sudo update-rc.d mpss defaults 99 10
     sudo service mpss start

For micsmc
sudo apt-get install libsm6:amd64 libxrender1:amd64 libfontconfig1:amd64
----------------------
# Ubuntu: sudo chmod o+w  /dev/mic/scif

--------------------------------------------------------------------------------------------------------------------------------

[Nagy-Egri__Mate_Fere]

Dear Jofre and Minh,

thank you both for the explanation. I was able to install MPSS, flash the MIC card, install OpenCL runtime with the ICD, and have it all recognize the MIC card.

However, even though I issued the chmod you suggested, I can still only run OpenCL apps as root, if I wish to use the MIC cards, otherwise only the CPU is visible. A very similar trick was needed for the AMD drivers to make them visible in a headless server node, but what is the trick for Intel MIC and OpenCL to work in a non-root environment?

Thank you in advance.

[JJK]

can you run 'strace -o ocl.log -f ' on your OpenCL application? I can run OpenCL stuff as non-root without any issues (on CentOS 6.6). The OpenCL programs wants access to the following files:

15108 open("/sys/class/mic/ctrl/version", O_RDONLY) = 10
15108 open("/sys/class/mic/mic0/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 10
15108 open("/sys/class/mic/mic0/device/device", O_RDONLY) = 10
15108 open("/sys/class/mic/mic0/device/subsystem_device", O_RDONLY) = 10
15108 open("/sys/class/mic/mic0/device/vendor", O_RDONLY) = 10
15108 open("/sys/class/mic/mic0/family", O_RDONLY) = 6
15108 open("/sys/class/mic/mic0/sku", O_RDONLY) = 10
15108 open("/sys/class/mic/mic0/stepping", O_RDONLY) = 10
15109 open("/dev/mic/scif", O_RDWR)     = 9

my bet is that /dev/mic/scif is not accessible (enough) to a non-root user.

[Nagy-Egri__Mate_Fere]

Indeed, /dev/mic/scif needs rw access for things to work properly. I tried writing an upstart script that changes the permissions on this file, however I am a complete upstart noob. I came up with this script:

description "Set up /dev/mic/scif permissions"

start on starting network-interface
instance mic0

script
        chmod 666 /dev/mic/scif
end script

There is a strange behavior to this, because the permissions are set the instant I log onto the machine after a reboot, but if I execute a simple clinfo app (written by me), the first 3-4 runs do not detect the MIC card. How can that be?

It would rock if we could see all these changes in an official bundle with .deb packages that are as trivial to install as .rpm-s on RedHat per say.

[Minh_H_]

meteorhead wrote:

Dear Jofre and Minh,

thank you both for the explanation. I was able to install MPSS, flash the MIC card, install OpenCL runtime with the ICD, and have it all recognize the MIC card.

However, even though I issued the chmod you suggested, I can still only run OpenCL apps as root, if I wish to use the MIC cards, otherwise only the CPU is visible. A very similar trick was needed for the AMD drivers to make them visible in a headless server node, but what is the trick for Intel MIC and OpenCL to work in a non-root environment?

Thank you in advance.

Hi,

I do not use OpenCL so I could not comment. You only need to change mod (chmod) if you need to use scif under normal user.  I use the mpss-module from the git.  For some reason only root can access/use scif, I have to change mode to allow  normal users to use it.

[Loc_N_Intel]

If you use OpenCL Runtime and  MPSS 3.4, it may not work for Intel(R) Xeon Phi(TM) coprocessor. According to this release notes  https://software.intel.com/en-us/articles/opencl-runtime-release-notes/ , you must install MPSS 3.3 .

Having said that, MPSS has been tested and supported RHEL and SuSE only. I am not sure about Ubuntu. Thank you.

[Eric_B_1]

I can confirm that this works:

But the current Ubuntu Kernal is 3.19 and you will get the error as listed. 

You will have to roll back to 3.13 as stated and it works.  No issues on my machine.

But for future referance the only issue is with this line in mic_ctx->sysfs_state = sysfs_get_dirent linux.c

[Eric_B_1]

I can confirm that this works:

But the current Ubuntu Kernal is 3.19 and you will get the error as listed. 

You will have to roll back to 3.13 as stated and it works.  No issues on my machine.

But for future referance the only issue is with this line in mic_ctx->sysfs_state = sysfs_get_dirent linux.c

[P__Robert]

#Confirmed with ubuntu-14.04.4-desktop-amd64.iso
# using MPSS 3.4.6
# and kernel 3.16.0-62-generic

apt-get remove linux-generic-lts-wily
apt-get install linux-generic-lts-utopic

gedit /home/mpss-modules-3.4.6/host/linux.c:

    /*
#if (LINUX_VERSION_CODE > KERNEL_VERSION(2,6,35))
                NULL,
#endif
    */
make MIC_CARD_ARCH=k1om
  LD   /home/mpss-modules-3.4.6/mic.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/mpss-modules-3.4.6/mic.mod.o
  LD   /home/mpss-modules-3.4.6/mic.ko
make[1]: Leaving directory `/usr/src/linux-headers-3.16.0-62-generic'


root@coast77:/home/mpss-modules-3.4.6# uname -a
Linux coast77 3.16.0-62-generic #83~14.04.1-Ubuntu SMP Fri Feb 26 22:52:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@coast77:/home/mpss-modules-3.4.6# 

[Elias_K_]

I can also confirm that MPSS and Phi work fine on my Kubuntu 14.04 LTS server, although I haven't tested for OpenCL yet since I don't need it.

To install on (x)Ubuntu you need to use either the 3.13 or 3.16 Linux kernel and the module only needs the minimal modifications mentioned above or found in the ArrayFire article's GitHub repo that I link below. The said article is very good and complete with step by step instructions. It has all the information that is also mentioned here, except for the /dev/mic/scif permissions issue (chmod 666 /dev/mic/scif) and the need to link lspci (sudo ln -s /usr/bin/lspci  /sbin/lspci) to be able to run miccheck. If you plan to flash your card please use the instructions on the official user guide / readme which has separate instructions if you have a C0 stepping or not.

http://arrayfire.com/getting-started-with-the-intel-xeon-phi-on-ubuntu-14-04linux-kernel-3-13-0/
https://github.com/pentschev/mpss-modules/commit/637b27a8e2a95618472c6b1234a933ad2017a8cf

If you use (x)Ubuntu 14.04.x LTS with greater kernel versions (e.g 3.19 or 4.2 or 4.4) due to the LTS Hardware Enablement Stack you need to downgrade your kernel to 3.13 or 3.16:

3.13: sudo apt-get install linux-generic (that's what you already have if you installed with a 14.04.0/1 LTS ISO)
OR
3.16: sudo apt-get install linux-generic-lts-utopic (that's what I already had due to installing with a 14.04.2 LTS ISO)

You can of-course try your luck with greater kernel versions and or distribution versions but the mic kernel module will need a lot of patches. There are two repositories for 3.19 and 4.2 mentioned in the ArrayFire article comments but I haven't tested them and the replies seem to indicate that they don't work.

And yes I totally agree it would be great if Intel created the needed .deb packages, even if they don't make this an official port (which would be even greater!).

Finally does anybody know what am I missing with MPSS 3.4.x vs e.g 3.7.x release? I know that the 3.4 is LTS but what features does it miss? I was contemplating on installing the 3.7 release due having a much improved user guide. However I figured that I could just read the new user guide while installing the 3.4 release and be on the safe side with the (x)Ubuntu instructions.

[P__Robert]

The big change that caused me to switch from 3.4.6 to 3.7 is use of steady_clock

chrono::steady_clock  #not available in 3.4

However, 3.7 enables chrono::steady_clock.

[Rolly_N_]

Hello, 

I am trying to install 4 Xeon Phi 7120p on a Ubuntu 16.04 LTS with kernel 4.4.0-62-generic. I found the latest MPSS 3.8 supports 

SUSE* Linux* Enterprise Server 12 SP2 64-bit | 4.4.21-69-default

So, does this guide works for Ubuntu 16.04?

Thank you!