Help! Stymied by ixgbe broken interaction with iproute2...

[idata]

When loading the module ixgbe with sr-iov enabled (max_vfs), I cannot see the relevant interfaces with the iproute utility. This is required functionality in order to assign a static MAC address to each VF which are otherwise randomly generated on reboot.

Any suggestions would be greatly appreciated!

Some data points:

• Server Information
  • Manufacturer: HP
  • Product Name: ProLiant DL165 G7
• Ethernet Card Information:
  • Intel Corporation 82599EB 10-Gigabit SFI/SFP+
• uname:
  • Linux 3.0.0-16-server x86_64 GNU/Linux
• OS Release:
  • Ubuntu 11.10

Steps to reproduce:

1. Determine the ethernet interface assignments:

# dmesg |grep ixgbe\ 0000:05:00.[0-9].*detected|head -n 2

[ 8.448031] ixgbe 0000:05:00.0: eth4: detected SFP+: 5[ 8.816035] ixgbe 0000:05:00.1: eth5: detected SFP+: 6

2. Load the module without sr-iov enabled (i.e. no max_vfs) and see that the interfaces are visible:

# modprobe ixgbe

# ip link show|grep eth[45]

14: eth4: mtu 1500 qdisc noop state DOWN qlen 1000 15: eth5: mtu 1500 qdisc noop state DOWN qlen 1000

3. Unload the module and reload with sr-iov enabled

# rmmod ixgbe

# modprobe ixgbe max_vfs=63# ip link show|grep eth[45] <<null output!>>Troubleshooting Steps:

• /proc/net/dev shows my devices exist at all times

# cat /proc/net/dev|grep eth[45]| cut -d : -f 1

eth4 eth5

• configuring the interface via ifconfig is consistent at all times

# ifconfig eth5 10.0.0.1/24

# ifconfig eth5eth5 Link encap:Ethernet HWaddr 00:1b:21:b9:a5:e5 inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0

• Tested with the built in as well as self-compiled latest stable releases of http://sourceforge.net/projects/e1000/files/ixgbe%20stable/ ixgbe and http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 iproute2

# modinfo ixgbe

filename: /lib/modules/3.0.0-16-server/kernel/drivers/net/ixgbe/ixgbe.ko version: 3.8.21-NAPI license: GPL description: Intel(R) 10 Gigabit PCI Express Network Driver author: Intel Corporation, <</span>mailto:linux.nics@intel.com linux.nics@intel.com> srcversion: 06E123F39C1AAB9CE66E411

# ip -V

ip utility, iproute2-ss110317

[idata]

Additional note:

I also tested this on a seperate, identical DL165G7 with an new 82599EB card and got the same results, so it's very unlikely to be a defective hardware issue.

[Patrick_K_Intel1]

I'm consulting with my experts on a theory.

In the meantime, if you only create say 10 or 20 VF's, does it work correctly?

I believe this has to do with the large ammount of text that needs to be displayed when you run this command. It will show all ethernet devices - if you only have a single Intel 82599 in your system and nothing else, this will still be 128 Ethernet devices that needs to be dispalyed.

The way these kernel utilities display text is pretty confusing to me, however I do recall in the past that there was challenges associated with them trying to output large amounts of text due to some paging issues or something along those lines.

I'll try to get more information.

[Patrick_K_Intel1]

My experts tell me that if you get the latest iproute2, this problem should go away. Patches were made for the reasons I explained above.

Please give this a try and report back to the forum on your hopeful success)

thanx,

Patrick

[idata]

Hey Patrick, your suggestion was spot on!

# modprobe ixgbe max_vfs=20

# ip link show .....14: eth4: mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 00:1b:21:b9:a5:e4 brd ff:ff:ff:ff:ff:ff vf 0 MAC fe:77:26:31:b2:c0 vf 1 MAC 86:b7:4b:4e:38:61 vf 2 MAC 7e:9a:ea:ef:ed:8a vf 3 MAC ce:e4:e1:b5:3d:75 vf 4 MAC 12:fd:b6:78:5c:8e vf 5 MAC 1e:21:8b:90:5d:a4 vf 6 MAC 36:e5:6a:2d:7b:35 vf 7 MAC 62:bf:66:99:f9:9c vf 8 MAC c6:15:70:86:87:6d vf 9 MAC f2:53:a4:32:cd:c3 vf 10 MAC 06:fe:90:5f:cf:a1 vf 11 MAC e2:f1:d3:00:b5:7c vf 12 MAC de:c3:57:36:e0:06 vf 13 MAC 8e:cd:c3:ca:3c:fd vf 14 MAC f6:a7:22:ce:29:57 vf 15 MAC fa:85:d5:df:a5:fb vf 16 MAC 3e:ef:26:e7:6e:ee vf 17 MAC 9a:28:59:72:c7:3f vf 18 MAC 4e:84:49:60:1a:51 vf 19 MAC b6:c0:40:f4:8e:b7

I found that the maximum number I can specify is 37 using either the built-in OS's or self-compled versions of iproute2

I was using the iproute2 source from git so I presume that's the latest. I also grabbed http://kernel.org/pub/linux/utils/net/iproute2/ iproute2-3.3.0 which behaves the same. Do you know where the patches live? I searched around on http://marc.info/?l=linux-netdev&r=1&w=2 linux-netdev with no success.

Thanks again!

[Patrick_K_Intel1]

I'll follow-up with the engineer who did the patches - may not have been for this problem.

While not as elegant - I believe you can specify a specific PF & VF combination to get data on. Doing this you could, via a script loop through all of the VF's for a PF.

[idata]

While the folowing does work, like you said it is very kludgy. While the PF's MAC is viewable via /sys/class/net/eth4/address, the VF's MAC is not available anywhere except dmesg (that I know of).

# ip link set eth4 vf 62 mac 0A:0A:0A:0A:0A:0A

# dmesg |grep ixgbe |grep 05.*VF\ 62[97507.953387] ixgbe 0000:05:00.0: eth4: IOV: VF 62 is enabled mac 7E:A7:93:3E:0E:DC[100288.404401] ixgbe 0000:05:00.0: setting MAC 0a:0a:0a:0a:0a:0a on VF 62

Thanks, and I'll be looking forward to seeing that patch!

Cheers,

Scott

[idata]

I found out the patches that are needed from Greg Rose via the http://marc.info/?l=linux-netdev&m=133278809815315&w=2 net-dev mailing list. The patches that are needed are in the following commits:

• http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=a4b64fbe482c7766f7925f03067fc637716bfa3f a4b64fbe482c7766f7925f03067fc637716bfa3f
• http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=115c9b81928360d769a76c632bae62d15206a94a 115c9b81928360d769a76c632bae62d15206a94a

[Patrick_K_Intel1]

Thanks for passing that along. Much appreciated.