NFS configuration

[YH]

Hi,

I am trying to the NFS mounting to work and followed the instruction on the user guide section 7.4. 

However, when I got to micctrl --addnfs=172.31.1.254:/micNfs --dir=micNfs

I got the following warning

[Warning] mic0: Server 172.31.1.254 may not be reachable if the interface is not routed out of the host
[Warning] Modified existing NFS entry for MIC card path '/micNfs'
[Warning] mic1: Server 172.31.1.254 may not be reachable if the interface is not routed out of the host
[Warning] Modified existing NFS entry for MIC card path '/micNfs'
[Warning] mic2: Server 172.31.1.254 may not be reachable if the interface is not routed out of the host
[Warning] Modified existing NFS entry for MIC card path '/micNfs'

and once I log into mic0, and try mount -a and I got the following error

mount: RPC: Remote system error - No route to host

mount:mounting 172.31.1.254:/micNfs on /micNfs failed: Bad file descriptor

I am using CentOS 7.0. 

Thanks

[Sunny_G_Intel]

Hi,

I would like to verify if you followed NFS configuration steps as given in the Intel® Xeon Phi™ Coprocessor system admin guide section 5.6 Configuring NFS mount on  Intel® Xeon Phi™ Coprocessors.

Let me know if you still have trouble configuring NFS.

Thanks

[YH]

Yes, I followed these instructions. But, didn't get it working.

[Sunny_G_Intel]

Hi YH,

Looking at the your first post I am assuming that you are trying to mount an NFS share running on the same host to which coprocessor is attached (static pair network). i.e. If you follow this latest version of Intel® Xeon Phi™ Coprocessors sysadmin guide section 5.6.3 NFS mounting a host export then your setup falls into Case 1.

Can you please verify if you have the host firewall or iptables configured to allow NFS to work, 

tcp/udp port 111 - RPC 4.0 portmapper
tcp/udp port 2049 - nfs server

Can you ping the host(172.31.1.254) from the coprocessor? If not there can be some issues with network configuration. 

I would also like to verify if you updated the following files as given before executing addnfs using micctrl:

//Append /etc/exports on the NFS server with the line
/micNfs <NFS Client IP Address or Range of IP Address>(rw, no_root_squash)


//Add in /etc/hosts.allow on the NFS server
ALL: <NFS Client IP Address or Range of IP Address>

//Let NFS know the files have changed by executing
$ exportfs -a

Also in order to better assist you, can you please let me know which version of MPSS you are using.

Thanks

[YH]

I got it now. I missed the step for /etc/hosts.allow

Will the procedure be the same if I did a internal bridge for to connect all the mic cards?

[Sunny_G_Intel]

Hi YH,

I am glad you got it working. 

Yes, the steps for NFS mounting a host export will be similar to what you have right now. However, It is recommended that you get the internal bridge configured correctly before trying NFS mount over internal bridge. For internal bridge, you should verify if you have connectivity between two or more coprocessor cards and also between each coprocessor and the host. Once you have internal bridge configured you should be able to complete NFS mounting by using proper IP addresses.

Thanks

[YH]

Hi Sunny, 

It failed again after a reboot. 

I tried cleanconfig, then initdefaults still couldn't work.

Anyone had this problem before?

Thanks,

[Sunny_G_Intel]

Hi,

Can you please explain what is failing? Is the mount failing ? If yes, Did you run "exportfs -a" ? Are your firewall and network settings still correct? 

I would like to refer you to the section: 6.1.3 NFS mounting issues for Intel® Xeon Phi™ coprocessors  from the System Administration guide for troubleshooting NFS mounting related issues for Intel® Xeon Phi™ coprocessors.

Thanks

[YH]

Sunny,

Yesterday, I got it working for both without the internal bridge and internal bridge. Then, I moved on to getting the wrf code working. While, doing so, somehow I couldn't access mic0 anymore and have to reboot. 

Then things started not working. After I did the service mpss start, the nfs mount no longer see the hosts directory. 

So, I went back to clean all the config and re initial everything. Below is my setting on the host

cat /etc/hosts.allow 

ALL:172.31.1.1
ALL:172.31.2.1
ALL:172.31.3.1

cat /etc/exports
/micNfs 172.31.1.1(rw,no_root_squash)
/micNfs    172.31.2.1(rw,no_root_squash)
/micNfs    172.31.3.1(rw,no_root_squash)

cat /etc/sysconfig/iptables

# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
-A INPUT -m state --state NEW -m udp -p udp --dport 2049 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 111 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 4045 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 4045 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 1110 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 1110 -j ACCEPT
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Then I did the following to reset things

service mpss stop

service iptables restart

sleep 5 

micctrl --cleanconfig

micctrl --initdefaults

exportfs -a

service nfs restart

micctrl --addnfs=172.31.1.254:/micNfs --dir=/micNfs mic0

([Warning] mic0: Server 172.31.1.254 may not be reachable if the interface is not routed out of the host)

micctrl    --addnfs=172.31.2.254:/micNfs --dir=/micNfs mic1

[Warning] mic1: Server 172.31.2.254 may not be reachable if the interface is not routed out of the host

micctrl    --addnfs=172.31.3.254:/micNfs --dir=/micNfs mic2

[Warning] mic2: Server 172.31.3.254 may not be reachable if the interface is not routed out of the host

service mpss start

ssh mic0

mount -a

mount: RPC: Remote system error - No route to host
mount: mounting 172.31.1.254:/micNfs on /micNfs failed: Bad file descriptor

Thanks.

[Sunny_G_Intel]

Hello YH,

The command"micctrl --cleanconfig" removes the NFS export directory and removes configuration files in the /etc/mpss directory. 

Can you please check if /etc/fstab file on coprocessor has a NFS entry like this:

172.31.1.254:/mnt/mic0Share     /mic0NFS        nfs             defaults                1 1

If the entry does not exist then, On host run

//Your micctrl addnfs commands
service nfs restart  and/or  exportfs -a  //exportfs -a just lets the nfs server know about the changes in exports file, if there are no changes it will not affect anything
ssh mic0
//Check /etc/fstab again for nfs entry
//If entry exists run
mount -a

If the entry exists in /etc/fstab file on coprocessor then

On host run,
service nfs restart
ssh mic0
mount -a

Thanks

[YH]

Yes the file fstab is available at host /var/mpss/etc/mic0/etc/fstab and in mic0 /etc/fstab.

The line 172.31.1.254:/micNfs /micNfs nfs nolock 1 1 is also there. 

But, again 

mount -a

mount: RPC: Remote system error - No route to host
mount: mounting 172.31.1.254:/micNfs on /micNfs failed: Bad file descriptor

I think it's have something to do with the virtual network. But don't know how to fix this.

[Sunny_G_Intel]

Hi YH,

Can you please share with me output of

micctrl --config mic0

 I see the following error when either I have no connectivity from MIC to Host or when the Host NFS daemon is not running. 

mount: RPC: Remote system error - No route to host
mount: mounting 172.31.1.254:/micNfs on /micNfs failed: Bad file descriptor

So at this point (without running initdefaults or clean config) can you again verify /etc/exports and /etc/hosts.allow file. Once the files look correct can you execute the following commands:

//On host //assuming firewall settings are already done
exportfs -a
service nfs restart
ssh mic0

//On Coprocessor
ping "host IP address"    //To verify if you can connect to host
mount -a  

Mounting NFS export on coprocessor is verify similar to mounting any NFS export on Linux. By using micctrl --addnfs you just add entry for /etc/fstab automatically. 

I hope this helps

[YH]

Sunny, 

It worked again. I shutdown the firewalld daemon and redo everything then it worked. 

I was puzzled, because when I first install mpss, I couldn't do any of the firewall thing, I still be able to mount.

Hopefully, this is really the problem. 

Thank you. 

by the way

micctrl --config mic0 gives the following, if you see something weird please let me know.

mic0:
=============================================================
    Config Version: 1.1

    Linux Kernel:   /usr/share/mpss/boot/bzImage-knightscorner
    Map File:       /usr/share/mpss/boot/System.map-knightscorner
    BootOnStart:    Enabled
    Shutdowntimeout: 300 seconds

    ExtraCommandLine: highres=off
    PowerManagment: cpufreq_on;corec6_off;pc3_on;pc6_off

    Root Device:   Dynamic Ram Filesystem /var/mpss/mic0.image.gz from:
    Base:      CPIO /usr/share/mpss/boot/initramfs-knightscorner.cpio.gz
    Overlay:   Filelist /var/mpss/sep /var/mpss/sep/sep3.15/sep.filelist on
    Overlay:   Filelist /var/mpss/itt_lib /var/mpss/itt_lib/itt.filelist on
    Overlay:   Filelist /opt/intel/mic//amplxe/vtune_amplifier_xe_2015.1.1.380310 /opt/intel/mic//amplxe/vtune_amplifier_xe_2015.1.1.380310/_amplxe_vtune_amplifier_xe_2015.1.1.380310.filelist on
    Overlay:   Filelist /opt/intel/mic//amplxe/vtune_amplifier_xe_2015.1.1.380310 /opt/intel/mic//amplxe/vtune_amplifier_xe_2015.1.1.380310/vtsspp.filelist on
    CommonDir: Directory /var/mpss/common
    Micdir:    Directory /var/mpss/mic0

    Network:       Static Pair
        Hostname:  eagle-mic0
        MIC IP:    172.31.1.1
        Host IP:   172.31.1.254
        Net Bits:  24
        NetMask:   255.255.255.0
        MtuSize:   64512
        MIC MAC:   4c:79:ba:1c:17:40
        Host MAC:  4c:79:ba:1c:17:41

    LDAP:          Disabled
     NIS:          Disabled

    Cgroup:
        Memory:    Disabled

    Console:        hvc0
    VerboseLogging: Disabled
    CrashDump:      /var/crash/mic 16GB