mounting GPFS on Xeon Phi

[Miah__Wadud]

Hello, what is the best way to mount GPFS on an Intel Xeon Phi card? Do I have to export GPFS as NFS and then mount NFS on the Phi? Thanks in advance,

[Jess]

We're using the export as NFS option, and I have to say it is very unstable! Only one of the GPFS file systems can be mounted most often, and MPSS spouts strange warnings about the file handles being the same as /, which they aren't really. The portability layer won't build for the MIC architecture at the moment. We've asked IBM about it, but they seem uninterested in doing anything.

[Miah__Wadud]

Hi Zaniyah, thanks for the reply. I think the only way forward for now is to use NFS with its instabilities. With the release of Nights Landing imminent, I can see why IBM aren't very interested.

Wadud.

[BelindaLiviero]

is it at all possible to mount the GPFS filesystem as GPFS on the host, and export it as NFS (for the coprocessor to mount from the host)?  -- is this a workable scenario that is more stable and performant?

[Miah__Wadud]

Hi Belinda, that is a very good point. Will give it a go but this will mean that Phi cards on different nodes cannot change a file as the locks on files will be prohibitively expensive. However, our use of the Phi isn't that sophisticated.

[Andrey_Vladimirov]

Hi Wadud,

are you aware of performance limitations of NFS on Xeon Phi? At this point in time, with NFS you are stuck at 20 MB/s for streaming I/O. Unless you have minimal requirements for I/O on Xeon Phi, any workaround invonving NFS will be unsatisfactory. The only distributed file system that works well with Xeon Phi as of today is Lustre (see http://research.colfaxinternational.com/post/2014/07/28/io.aspx ). Of course, it does not solve your problem if you already have GPFS in production.

Andrey

[Jess]

Belinda, this is what we are doing, but NFS on Phi is not behaving in a very stable manner. It may be that NFS exporting is incompatible with GPFS. The message the Phis give when mounting say /home (which is a GPFS mount on the host, exported via NFS to the Phis) says that the software cannot tell the difference between / and /home, and believes they are the same file system. It then refuses to mount sometimes, or randomly drops the connection.

[Belinda_L_Intel1]

Hi, We don’t have GPFS nor a way to reproduce this environment. 

So, thinking out loud:  I don’t know if certain MTU settings, or forcing TCP usage, or block sizes are going to make a difference.   Perhaps you can try to NFS mount the GPFS filesystem on the host (and see how reliable is that first, and work out the kinks) before trying to make things work well on the coprocessor.

A cursory google search for GPFS and NFS turned up several guides and redbooks which may point out gotchas (if there are any).   It is possible you could contact IBM and talk about this, they may have some advice -- regardless of whether the client is 'supported' or not I am sure they deal with many customers who have a diverse ecosystem where not everything is capable of running GPFS natively

[Jess]

We have tried many times to export GPFS from the host to the phis, which is what I am describing above. It is unstable and unreliable, regardless of what options you pick. The only way to fix this is to make GPFS support available from the phi operating system itself, which will require IBM's involvement.

[Michael_R_]

I know this thread is a little old, but I just came across it. We are using GPFS on our cluster and have 2 nodes with Phi cards and we have been using the NFS export option for quite awhile now. We have not noticed any problems. I'm wondering if I'm just not seeing the problems because our cards don't get used much or if we're using some options that you haven't tried yet. Here is what our setup currently looks like.

CentOS 6.6

MPSS 3.4.2

GPFS 3.5.0-22

We export from our management server which is GPFS client only. (Not an NSD)

/etc/exports  looks like this:

/gpfs/sb 172.17.0.0/16(rw,no_root_squash,sync,fsid=201)
/gpfs/lb 172.17.0.0/16(rw,no_root_squash,sync,fsid=202)

All the Phi cards have IPs in the 172.17.0.0/16 subnet on their mic0 interface.

Their /etc/fstab has these entries:

mgmt:/gpfs/sb     /gpfs/sb        nfs             nolock          1 1
mgmt:/gpfs/lb     /gpfs/lb        nfs             nolock          1 1

Where the mgmt hostname resolves to an IP on the cluster facing interface on the management server.

This is what /proc/mounts looks like on the cards:

mgmt:/gpfs/sb /gpfs/sb nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,port=65535,timeo=70,retrans=3,sec=sys,local_lock=all,addr=172.17.0.1 0 0
mgmt:/gpfs/lb /gpfs/lb nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,port=65535,timeo=70,retrans=3,sec=sys,local_lock=all,addr=172.17.0.1 0 0

Does anybody having problems with GPFS exported as NFS see anything in there that isn't in your setups?

Also, If you're having problems with GPFS exported as NFS to Phi are you also having the same problems if you try to export to another x86 host? That should be a fully supported configuration so if that doesn't work you can get support from IBM and  the solution may carry over to your Phi setup.

I hope this helps somebody.

Mike Robbert