MTU missmatch size in external bridge configuration

[marek_kaletka]

Host system running Scientific Linux 6.4, 2 5110p MICs with MPSS 2.1.6720-13.

Configured external bridge with MTU 1500 (our switches don't like anything bigger than that).

Each time we do micctrl --resetconfig or just reboot host system, both host's and mic's virtual mic0 interface is configured with MTU of 64k, which obviously doesn't work well when connected directly to bridge interface with MTU set to 1500. Even plain lsiting of files on NFS mount takes ages, transfers in few KB/s range.

Manual setting of MTU=1500 on host and in /opt/mic/filesystem/micX/etc/sysconfig/network/ifcfg-mic0, followed by service network restart and micctrl -R works as workaround, but i'm looking for more convenient and persistent solution.

Any ideas ?

[Frances_R_Intel]

In your /etc/sysconfig/mic/micN.conf file, you have the external bridge defined with either an MTU size of 1500 (or no MTU size specified, since 1500 is the default for an external bridge) and you do not have Network StaticPair defined?

[marek_kaletka]

I have currently (manualy added) MTU=1500 line in both /etc/sysconfig/mic/micN.conf files, but this is wiped out after every micctrl --resetconfig or system reboot. The same for /opt/mic/filesystem/micN/etc/sysconfig/network/ifcfg-mic0.

Should result in default MTU of 1500, but doesn't - both virtual interfaces are set (probably by mpss) to 64k.

[Frances_R_Intel]

From the Intel(R) Manycore Platform Software Stack (Intel(R) MPSS) Boot Configuration Guide for the latest MPSS (mpss_gold_update_3-2.1.6720-13)

<blockquote> The MTU parameter has been deprecated and is now a part of the Bridge and Network parameters. Running micctrl --initdefaults will convert this parameter to be included in the new ones. </blockquote> 

Can you try running micctrl --initdefaults and then checking the micN.conf to make sure you have a Bridge entry for your external bridge that uses 1500 for the MTU size and that you have no Network StaticPair entry? (FYI, the NetBits parameter has also been deprecated.) Hopefully this is the solution. If not, we'll dig a little deeper.

[marek_kaletka]

Apologies for late answer, was quite busy and without access to our MIC machines.

I've did our configuration from scratch, using only micctrl to configure things. Network config made by micctrl:

cat /etc/sysconfig/mic/ | grep Network
   Network StaticBridge eth0br1 10.26.0.42 yes

cat /etc/sysconfig/network-scripts/ifcfg-mic0
   DEVICE=mic0
   ONBOOT=yes
   NM_CONTROLLED="no"
   BRIDGE=eth0br1

 cat /opt/intel/mic/filesystem/mic0/etc/sysconfig/network/ifcfg-mic0
  IPADDR=10.26.0.42
  GATEWAY=10.26.0.1

on host:

ip a | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
4: mic0: <BROADCAST,UP,LOWER_UP> mtu 64512 qdisc pfifo_fast state UP qlen 1000
5: mic1: <BROADCAST,UP,LOWER_UP> mtu 64512 qdisc pfifo_fast state UP qlen 1000
6: eth0br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

on mic:

[root@compute000-mic0 /root]# ip a | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
2: sit0: <NOARP> mtu 1480 qdisc noop state DOWN
3: mic0: <BROADCAST,UP,LOWER_UP> mtu 64512 qdisc pfifo_fast state UNKNOWN qlen 100

As one can see, there is no MTU size defined in any of conf files, so it sholud default to 1500 on all interfaces.
But it doesn't - both mic's and hosts's micX interfaces are 64512. 

[Rene_Oertel]

Try using

MTUsize 1500

in /etc/sysconfig/mic/micN.conf and then a micctrl --resetconfig

I think it is a bug in mpss_gold_update_3-2.1.6720-13 (still downloading mpss_gold_update_3-2.1.6720-15) ...

[marek_kaletka]

Rene,

thanks for the tip - it works for host's part, unfortunatelly not for mic's - MTU still at 64512. Must be a bug, so waiting for new mpss (6720-15) doesn't solve this one.
 

[Rene_Oertel]

Hmm mysterious, after setting the value in the mentioned configuration file and restarting MPSS, there is no MTU value in the config file /etc/sysconfig/network/ifcfg-mic0 on the MIC?

Indeed, 6720-15 doesn't solve the standard problem, but should work with the manual setting in the mic0.conf and so on.

[marek_kaletka]

can confirm: manual add of MTU size now survives mpss restart (but not micctrl --resetconfig). Can live with this until solved in one of future mpss releases. Thanks all for help.