Intel MIC + Lustre 2.5.2

[Atul_Y_]

Hi Team,

Greetings,

we are trying to build Intel MIC with lustre support.

Lustre 2.5.2 client,Cent OS 6.5, MPSS-3.2.3, OFED1.5.4.1 + OFED-3.5-2-MIC-beta1

With Plain kernel(2.6.32-431) of cent os, we are able to install mpss and ofed stack.

But when we are following the same things with the updated kernel(2.6.32-431.17.1) mpss stack is not loading.

How can we resolve this issue.

please reply 

Thank You

Atul Yadav

[Paul_P_3]

You'll need to rebuild the mpss modules for the newer kernel as outlined in the readme.txt.  Relevant section below:

Steps:                                                                 *
    *   o Red Hat* Enterprise Linux* (RHEL) 6.0, 6.1, 6.2, 6.3, 6.4, and 6.5   *
    *       1) Ensure the prerequisites are installed:                         *
    *          user_prompt> sudo yum install kernel-headers kernel-devel       *
    *                                                                          *
    *       2) Regenerate the Intel(R) MPSS driver module package:             *
    *          user_prompt> cd <folder where extracted tar file expanded>/src/ *
    *          user_prompt> rpmbuild --rebuild \                               *
    *          mpss-modules-*.el6.src.rpm                                      *
    *                                                                          *
    *       3) The mpss-modules binary rpm is located at:                      *
    *          $HOME/rpmbuild/RPMS/x86_64                                      *
    *          user_prompt> ls $HOME/rpmbuild/RPMS/x86_64                      *
    *                                                                          *
    *       4) Copy the mpss-modules binary RPM from the                       *
    *          $HOME/rpmbuild/RPMS/x86_64 directory to the directory where     *
    *          the MPSS tar file was extracted (overwriting the mpss-modules   *
    *          binary RPM that was provided with the precompiled driver).

[Frances_R_Intel]

Since you are using OFED, if you are using OFED1.5.4.1, you will want to rebuild the drivers in ofed-driver-*.fc13.src.rpm in the same src directory as mpss-modules-*.el6.src.rpm.

But I am confused. You say you are running OFED1.5.4.1 + OFED-3.5-2-MIC-beta1, Actually those two are mutually exclusive. You should either use OFED1.5.4.1 + <folder where extracted tar file expanded>/ofed/*rpm or you should be running OFED-3.5-2-MIC-beta1. If you installed OFED-3.5-2-MIC-beta1 after OFED1.5.4.1, it probably overwrote the files from OFED1.5.4.1. Either should work with Lustre, I believe.

[Atul_Y_]

Dear Team,

After release of MPSS 3.3, we are able to build MPSS 3.3 with Mellonx and Lustre 2.5.2 mic rpms also.

But we are facing some issue in 

In First client after installing lustre mic rpms, lustre modules are not available inside the mic environment

In Second client after installing lustre mic rpms, lustre module are available but we are unable to mount the lustre file system,

Need technical help with steps to configure intel mic with lustre.

Steps for creating lustre mic rpms are given below:-

yum install libselinux-devel -y
rpm -ivh /root/phi/mpss-3.3/modules/mpss-modules-dev-2.6.32-431.el6.x86_64-3.3-1.x86_64.rpm
rpm -ivh /root/phi/mpss-3.3/modules/mpss-modules-2.6.32-431.el6.x86_64-3.3-1.x86_64.rpm
rpm -Uvh mpss-3.3/ofed/ofed-ibpd-3.3-r0.glibc2.12.2.x86_64.rpm
rpmbuild --rebuild --define "MOFED 1" mpss-3.3/src/dapl-2.0.42.2-1.glibc2.12.2.src.rpm mpss-3.3/src/libibscif-1.0.0-1.fc13.src.rpm mpss-3.3/src/ofed-driver-3.3-1.src.rpm
rpm -ivh /root/phi/mpss-3.3/mpss-sdk-k1om-3.3-1.x86_64.rpm
tar -xvjf /root/phi/mpss-3.3/src/linux-2.6.38+mpss3.3.tar.bz2
rpm2cpio mpss-3.3/k1om/kernel-dev-2.6.38+mpss3.3-1.knightscorner.rpm | cpio -idmv
rpm2cpio /root/phi/mpss-3.3/ofed/modules/ofed-driver-devel-2.6.32-220.el6.x86_64-3.3-1.x86_64.rpm | cpio -idmv
cp boot/Module.symvers-2.6.38.8+mpss3.3 linux-2.6.38+mpss3.3/Module.symvers
cp boot/config-2.6.38.8+mpss3.3 linux-2.6.38+mpss3.3/.config
cd linux-2.6.38+mpss3.3/
make ARCH=k1om silentoldconfig modules_prepare
source /opt/mpss/3.3/environment-setup-k1om-mpss-linux
export LD=k1om-mpss-linux-ld
cd /root/phi/lustre-release
git checkout 2.5.2
sh autogen.sh
./configure --with-linux=/root/phi/linux-2.6.38+mpss3.3 --disable-server --disable-tests --disable-doc --with-o2ib=/usr/src/ofed-driver --host=k1om-mpss-linux --build=x86_64-pc-linux
rpm -ivvh /root/rpmbuild/RPMS/x86_64/lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm --nodeps
rpm -ivh /root/rpmbuild/RPMS/x86_64/lustre-client-mic-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm --nodeps

Thank You

Atul Yadav

[Frances_R_Intel]

Just to be sure I understand - you are using the Mellanox* OFED 2.1.x?

There appear to be a couple instructions missing here and there (probably just a cut-and-paste error) but what it looks like you are doing is following the directions in section 2.5 of the MPSS User's Guide to install OFED support for the MPSS using Mellanox* OFED 2.1, then following the instructions on https://wiki.hpdd.intel.com/display/~dmiter/Lustre+on+Intel+Xeon+Phi to build Lustre for the coprocessor.

Is this what you are doing or am I lost?

If this is what you are doing, then the problem is with where lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm and lustre-client-mic-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm are installing their files.

The files are getting installed in /opt/lustre/2.5.54/k1om-mpss-linux. From there, they still need to find their way to the coprocessor 

If you look at /usr/share/doc/ofed-driver-3.3/lustre-phi.txt on the machine where you have installed this software, you will see a suggestion for how you could do this using a tar file. I will need to check but I think it might be possible, instead, to install the files on the coprocessor using an overlay directive. (You can find directions for this in the MPSS User's Guide.) In any event, you will also need to set up your configuration files and set it up so that Lustre starts.

One last thing - don't forget to install Lustre for your host, recompiling it to match the version of OFED you are using.

[Frances_R_Intel]

Ah - it has been pointed out to me that there is a copy of those same instructions on another forum topic - https://software.intel.com/en-us/forums/topic/475315 - which you have been posting to. That makes sense. Now I am not lost. 

But in any event, it is still necessary to install those files on the coprocessor. If I find a better way than the two I already pointed out (using a tar file and copying it over to the coprocessor or creating an overlay using micctrl) I will come back and post it here.