- Intel Community

Wendy__C_ · ‎09-30-2013

Do folks have instructions of howto install/run Lustre on Xeon Phi (as client) ? I'm on a relatively old MPSS (mpss_gold_update_3, mic-2.1.6720). My server runs 1.8.8 version of Lustre.

Thanks, Wendy

Frances_R_Intel · ‎10-07-2013

Have you checked out: http://software.intel.com/en-us/articles/configuring-intel-xeon-phi-coprocessors-inside-a-cluster?

With the next release of the MPSS, some of the information in here will be out of date (in particular, file locations and some configuration information) but for the version you are using, this document should help. There is a section on compiling the Lustre software and in the section discussing the boot script he wrote, Michael goes into starting the Lustre mods.

Wendy__C_ · ‎10-10-2013

Frances, Thanks for the pointer. I did check out the document but building a GNU build environment inside Xeon Phi is too involved for my purpose. With the help from Dmitry Eremin (dmitry.eremin@intel.com), the Lustre client is now up and running by doing:

Download the SOURCE package from http://software.intel.com/en-us/articles/intel-manycore-platform-software-stack-mpss
Unpack it and find the package-full_src-k1om.tar.bz2 file (in gpl directory). Unpack (again) this archive at /tmp/src. At /tmp/src, execute the "make defconfig-miclinux", followed by "make -C card/kernel ARCH=k1om modules_prepare"
Make sure "intel-mic-gpl-<version>.x86_64" package installed.
export PATH=/usr/linux-k1om-4.7/bin:$PATH
Obtain luster source (git clone git://git.whamcloud.com/fs/lustre-release.git)
Goto top of lustre tree (lustre-release) : do a "sh autogen.sh", followed by "./configure --with-linux=/tmp/src/card/kernel --disable-server --disable-tests --disable-doc --without-o2ib --host=x86_64-k1om-linux --build=x86_64-pc-linux", then "make rpms"
Look for lustre-client-mic RPMs in the usual rpmbuild directory (in RHEL case, it is /root/rpmbuild) , install them.
Restart mpss
All set to go :)

-- Wendy

Frances_R_Intel · ‎10-10-2013

Thanks for posting the directions. Glad it is working.

Pierre_Lagier · ‎04-04-2014

Hello everyone,

We just gave a try with mpss 3.2 and we faced some trouble to build lustre client for the phi, starting with missing or renamed packages from previous mpss releases - well, we moved from 2.x to 3.2 straight, kind of a big jump...

Any update/clues/advices will be more than welcome !

Thanks in advance,

Pierre.

BelindaLiviero · ‎04-07-2014

Hi Pierre,

here I adapted some instructions I saw from a colleague, but have not verified in full. I am hoping these will contain enough information to help unblock you -- please let us know if this worked

Assumptions/notes:

Only Lustre* version 2.5 and above of the Lustre client can actually be cross-compiled with MPSS 3.2 and OFED
It is required to have OFED version 1.5.4.1 Infiniband drivers installed

Steps for installation:

1. Download the SOURCE mpss-3.x-k1om.tar and OS specific files from the MPSS download page

2. Unpack from them "kernel-dev-*.rpm", "ofed-driver-*-devel-*.rpm" and "linux-*.tar.bz2" files

3. Prepare Intel MPSS sources for Lustre build:

rpm2cpio kernel-dev-*.rpm | cpio -idm
rpm2cpio ofed-driver-*-devel-*.rpm | cpio -idm
tar xjvf linux-*.tar.bz2 && cd linux-*
cp -f ../boot/config-* .config
make ARCH=k1om silentoldconfig modules_prepare
cp -f ../boot/Module.symvers-* Module.symvers

4. Now build Lustre:

./opt/mpss/3.x/environment-setup-k1om-mpss-linux
sh autogen.sh
./configure --with-linux=<path>/linux-2.6.38+mpss3.x --with-o2ib=<path>/usr/src/ofed-driver-*.el6.x86_64 --host=k1om-mpss-linux --build=x86_64-pc-linux
make rpms

Only two Lustre* RPMs should be installed on the host:

lustre-client-mic-<version>.x86_64.rpm
lustre-client-mic-modules-<version>.x86_64.prm

Additional step: ssh mic? "echo 'options Inet networks=\"o2ib0(ib0)\"' > /etc/modprobe.d/lustre.conf"

The Host configuration in /etc/modprobe.d/lustre.conf should look like this: options Inet networks="o2ib0(ib0)" and mounting the Lustre filesystem should be something like this: mount -t lustre 12.12.12.1@o2ib:/lustrefs/mnt/lustrefs

The coprocessor's /etc/modprobe.d/lustre.conf should have the options Inet networks="o2ib0(ib0)" entry and the mount command will look like this: mount.lustre 12.12.12.1@o2ib:/lustrefs/mnt/lustrefs

Atul_Y_ · ‎08-05-2014

Hi Team,

As per the above mention steps we are able to build the lustre mic module with o2ib.

But at the time of mounting lustre file system we are getting at lustre server side

Aug 5 23:57:54 IO1 kernel: Lustre: Skipped 1 previous similar message
Aug 5 23:59:32 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept 192.168.2.88@o2ib on 192.168.2.101@o2ib (ib1:1:192.168.3.101): bad dst nid 192.168.3.101@o2ib
Aug 5 23:59:57 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept 192.168.2.88@o2ib on 192.168.2.101@o2ib (ib1:1:192.168.3.101): bad dst nid 192.168.3.101@o2ib
Aug 6 00:00:22 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept 192.168.2.88@o2ib on 192.168.2.101@o2ib (ib1:1:192.168.3.101): bad dst nid 192.168.3.101@o2ib
Aug 6 00:11:01 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2923:kiblnd_cm_callback()) 192.168.2.88@o2ib: REJECTED 28
Aug 6 00:11:26 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2923:kiblnd_cm_callback()) 192.168.3.88@o2ib1: REJECTED 28

MIC details are given below:-

[root@phi3 ~]# ibv_devices
device node GUID
------ ----------------
scif0 4c79bafffe340459
mlx4_0 f452140300831c60
[root@phi3 ~]#

Created Wed Aug 6 00:36:48 2014

System Info
HOST OS : Linux
OS Version : 2.6.32-431.el6.x86_64
Driver Version : 3.3-1
MPSS Version : 3.3
Host Physical Memory : 65920 MB

Device No: 0, Device Name: mic0

Version
Flash Version : 2.1.02.0386
SMC Firmware Version : 1.14.4616
SMC Boot Loader Version : 1.8.4326
uOS Version : 2.6.38.8+mpss3.3
Device Serial Number : ADKC32600556

please guide to resolve the problem

THank YOu

Atul Yadav