topic Marek, I belive that MTU in Software Archive

Problem with setting up NFS on Phi cards

SDT — Mon, 12 Aug 2013 14:56:19 GMT

Hello,

I am using mpss_gold_update_3-2.1.6720-15-rhel-6.2 and 2 Xeon Phi cards (mic0 and mic1). Here is my configuration of NFS:

[root@phi ~]# cat /usr/local/mic/filesystem/common/etc/fstab
devpts /dev/pts devpts defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
172.16.28.40:/usr/intel /intel nfs rsize=8192,wsize=8192,nolock,intr 0 0
172.16.28.30:/home /home-hydra nfs rsize=8192,wsize=8192,nolock,intr 0 0
172.16.28.30:/opt /opt nfs rsize=8192,wsize=8192,nolock,intr 0 0

[root@phi ~]# cat /usr/local/mic/filesystem/common.filelist
dir /intel 755 0 0
dir /home-hydra 755 0 0
dir /opt 755 0 0
file /etc/fstab etc/fstab 664 0 0

At the begining I thought that there is something wrong with network configuration. I set up network by executing following commands:

micctrl --addbridge=br0 --type=External --ip=172.16.28.40 --netbits=23 --mtu=9000 mic0 mic1
micctrl --network=static --bridge=br0 --ip=172.16.28.41:172.16.28.42 mic0 mic1

"micctrl --config" shows configurations:

mic0:

=============================================================
Linux Kernel: /lib/firmware/mic/uos.img
BootOnStart: Enabled
Shutdowntimeout: 300 seconds

...

Network: Static bridge br0

MIC IP: 172.16.28.41
Host IP: 172.16.28.40
Net Bits: 23
NetMask: 255.255.254.0
MtuSize: 9000
Hostname: phi-mic0
MIC MAC: xx:xx:xx:xx:xx:xx
Host MAC: xx:xx:xx:xx:xx:xx

because ifconfig on mic0 and mic1 was showing me wrong value of MTU and NETMASK.

After that I added info about MTU and NETMASK to ifcfg-mic0:
echo 'MTU=9000' >> /usr/local/mic/filesystem/mic0/etc/sysconfig/network/ifcfg-mic0
echo 'NETMASKBITS=23' >> /usr/local/mic/filesystem/mic0/etc/sysconfig/network/ifcfg-mic0

what solved the problem. Unfortunately mic0 and mic1 still seems to hang again. It seems to work for a 10-20 seconds after first log in, but as soon as I start accessing files on network file system is hangs and all subsequent logins also hang immediately on trying to cd to network file system.

Do you have any ideas what can be wrong?

Maciek,

marek_kaletka — Mon, 12 Aug 2013 15:27:39 GMT

Maciek,

If you happen to have different MTU sizes on the host and mic's side (check with ifconfig or ip a) - check my "MTU mismatch size in external bridge configuration" thread for workaround.
There is a bug in current MPSS stack, so you don't always get what you set in config files.

Marek, I belive that MTU

SDT — Mon, 12 Aug 2013 15:44:56 GMT

Marek, I belive that MTU sizes on the host and mic's side are set up propelry.
MTU on host's bridge br0 = MTU on mic's interface mic0 (= 9000).
Please see the output of my ifconfig:

[root@phi ~]# ifconfig
br0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:172.16.28.40 Bcast:172.16.29.255 Mask:255.255.254.0
inet6 addr: fe80::225:90ff:fe96:248c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:4477863 errors:0 dropped:0 overruns:0 frame:0
TX packets:1118405 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:707396290 (674.6 MiB) TX bytes:8060051641 (7.5 GiB)

eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet6 addr: fe80::225:90ff:fe96:248c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:4810454 errors:0 dropped:0 overruns:0 frame:0
TX packets:5819300 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1096282693 (1.0 GiB) TX bytes:7956089741 (7.4 GiB)
Memory:fbe20000-fbe40000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:489597 errors:0 dropped:0 overruns:0 frame:0
TX packets:489597 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7533911903 (7.0 GiB) TX bytes:7533911903 (7.0 GiB)

mic0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet6 addr: fe80::4e79:baff:fe33:ad9/64 Scope:Link
UP BROADCAST RUNNING MTU:64512 Metric:1
RX packets:92 errors:0 dropped:0 overruns:0 frame:0
TX packets:38502 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10969 (10.7 KiB) TX bytes:5400729 (5.1 MiB)

mic1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet6 addr: fe80::4e79:baff:fe33:aeb/64 Scope:Link
UP BROADCAST RUNNING MTU:64512 Metric:1
RX packets:6637 errors:0 dropped:0 overruns:0 frame:0
TX packets:50733 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:422311 (412.4 KiB) TX bytes:116081991 (110.7 MiB)

[root@phi ~]# ssh mic0
[root@phi-mic0 /root]# ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

mic0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:172.16.28.41 Bcast:0.0.0.0 Mask:255.255.254.0
inet6 addr: fe80::4e79:baff:fe33:ad8/64 Scope:Link
UP BROADCAST RUNNING MTU:9000 Metric:1
RX packets:40341 errors:0 dropped:521 overruns:0 frame:0
TX packets:118 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5647817 (5.3 MiB) TX bytes:15254 (14.8 KiB)

Maciek,

marek_kaletka — Fri, 16 Aug 2013 13:34:19 GMT

Maciek,

on the host, br0's MTU=9000, but mic0 & mic1 MTU's are 64512.

This problem should no longer

Frances_R_Intel — Thu, 13 Aug 2015 23:38:46 GMT

This problem should no longer exist on MPSS 3.x and later.