External Bridge configuration

HM_Li · ‎09-08-2014

The system is CentOS 6.5 and the origin host network is:

------------------------------------------------------------------------
eth0      Link encap:Ethernet  HWaddr 00:25:90:E8:29:34  
          inet addr:192.168.1.45  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fee8:2934/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:471869 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31936 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:72666918 (69.3 MiB)  TX bytes:4467116 (4.2 MiB)
          Memory:dfb20000-dfb40000

Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
ib0       Link encap:InfiniBand  HWaddr A0:00:01:00:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:10.0.1.45  Bcast:255.255.255.0  Mask:255.255.255.0
          inet6 addr: fe80::f652:1403:85:7721/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1024
          RX bytes:168 (168.0 b)  TX bytes:2704 (2.6 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:17635 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17635 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:917332 (895.8 KiB)  TX bytes:917332 (895.8 KiB)
------------------------------------------------------------------------

The steps are:

service mpss stop
micctrl --addbridge=br0 --type=external --ip=192.168.1.45
micctrl --network=static --bridge=br0 --ip=192.168.1.201
service netwok restart
service mpss start

But even after rebooting the system, it can't connect to mic0(192.168.1.201).

The iptables is empty, and the ifconfig output is:

------------------------------------------------------------------------

br0       Link encap:Ethernet  HWaddr 02:85:BA:B2:7E:AB  
          inet addr:192.168.1.45  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::85:baff:feb2:7eab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:636 (636.0 b)

eth0      Link encap:Ethernet  HWaddr 00:25:90:E8:29:34  
          inet addr:192.168.1.45  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fee8:2934/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:522314 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34160 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:79215086 (75.5 MiB)  TX bytes:4742132 (4.5 MiB)
          Memory:dfb20000-dfb40000

Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
Ifconfig is obsolete! For replacement check ip.
ib0       Link encap:InfiniBand  HWaddr A0:00:01:00:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:10.0.1.45  Bcast:255.255.255.0  Mask:255.255.255.0
          inet6 addr: fe80::f652:1403:85:7721/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1024
          RX bytes:224 (224.0 b)  TX bytes:2704 (2.6 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:17733 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17733 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:922604 (900.9 KiB)  TX bytes:922604 (900.9 KiB)

mic0      Link encap:Ethernet  HWaddr 02:85:BA:B2:7E:AB  
          inet6 addr: fe80::85:baff:feb2:7eab/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:468 (468.0 b)  TX bytes:1512 (1.4 KiB)

mic1      Link encap:Ethernet  HWaddr 3A:43:32:3A:1E:F8  
          inet6 addr: fe80::3843:32ff:fe3a:1ef8/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:468 (468.0 b)  TX bytes:1470 (1.4 KiB)

------------------------------------------------------------------------

Can you help me? Thank you very much.

HM_Li · ‎09-08-2014

If changing the subnet for mic from 192.168.1.0/24(the host subnet) to other subnet 192.168.0.0/24, it works.

But how can let them in the same subnet?

JJK · ‎09-08-2014

your ethernet bridge br0 looks a bit odd; normally the ethernet device associated with the bridge does not have its own IP address anymore. How did you create the bridge br0?

Here's what I normally use

brctl addbr br0
brctl addif br0 eth0
ifconfig eth0 0.0.0.0 up
ifconfig br0 <IP> netmask <MASK>

Also, the document "Configuring Intel® Xeon PhiTM coprocessors inside a cluster" lists

After installing the network bridge, eth0 now does not have an IP address anymore, and the address has moved to br0. br0 and eth0 share a hardware address:

$ ifconfig
br0 Link encap:Ethernet HWaddr 00:1E:67:0C:B7:4C
inet addr:36.101.15.36 Bcast:36.101.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:636853 errors:0 dropped:0 overruns:0 frame:0
TX packets:37529 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:65024686 (62.0 MiB) TX bytes:65690500 (62.6 MiB)

eth0 Link encap:Ethernet HWaddr 00:1E:67:0C:B7:4C
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:1036186 errors:0 dropped:0 overruns:0 frame:0
TX packets:1633683 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:146064064 (139.2 MiB)

what does 'brctl show br0' return on your host?

I've configured a bridge on my Xeon Phi box using the commands listed above and I can reach the Xeon Phi just fine using it's "bridge address" .