E1G44ET2 server adapter: port order changes on reboot

[KLanz]

I'm posting here in the hope that someone has seen symptoms like this and can

offer some insight as to what's going on.

Executive summary: We have a newly-installed E1G44ET2BLK 4-port server adapter;

following a server reboot, we found that the ethernet jacks on that card had

been reordered! How did that happen, and how do we prevent it?

Here's how ifconfig saw this card after it was installed, with an ethernet

cable plugged into the bottom jack:

igb0: flags=1000802 mtu 1500 index 13

inet 0.0.0.0 netmask 0

igb1: flags=1000802 mtu 1500 index 14

inet 0.0.0.0 netmask 0

igb2: flags=1000802 mtu 1500 index 12

inet 0.0.0.0 netmask 0

igb3: flags=1001000843 mtu 1500 index 5

inet 171.64.175.54 netmask fffff800 broadcast 171.64.175.255

We checked every port on the card one at a time by plugging in an ethernet

cable and running ifconfig to see which network interface showed valid link

status (the RUNNING bit). We labeled the jacks b0 through b3 on the back panel; no

surprise, they were ordered top-to-bottom, b0 to b3.

After a server reboot, ifconfig showed the following:

igb0: flags=1000802 mtu 1500 index 13

inet 0.0.0.0 netmask 0

igb1: flags=1000802 mtu 1500 index 14

inet 0.0.0.0 netmask 0

igb2: flags=1000802 mtu 1500 index 12

inet 0.0.0.0 netmask 0

igb3: flags=1001000843 mtu 1500 index 5

inet 171.64.175.54 netmask fffff800 broadcast 171.64.175.255

Notice that igb3 is no longer RUNNING; igb1 shows as RUNNING, instead. Yet the

cable had not been moved! It's still plugged in to the bottom jack. What's

changed is the mapping from network device name to physical port: the bottom

jack, which used to be igb3, is now igb1. And igb1, which used to be the 2nd

jack down from the top, is now the bottom jack. Our carefully-applied labels

are now wrong!

To restore network connectivity, I had to move the cable from the bottom jack

to the 2nd jack down. This restored RUNNING status to igb3.

We can't live with network jacks that move around on our server every time

we reboot. How is this remapping happening? Is this a feature of the E1G44ET2

card? An issue with the card's firmware? A device driver or OS problem?

The OS is NexentaStor v3.1.3 Enterprise (SunOS Release 5.11, OpenSolaris

Build 134). Server node is based on the Xyratex OneStorHS 1235T chassis.

Driver version is apparently 1.1.18:

# modinfo | grep igb

179 fffffffff8084000 1e178 284 1 igb (Intel 1Gb Ethernet)

# strings /kernel/drv/amd64/igb | grep igb

igb 1.1.18

-- Kai Lanz Stanford University School of Earth Sciences