Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
5227 Discussions

Help! Stymied by ixgbe broken interaction with iproute2...

idata
Employee
2,583 Views

When loading the module ixgbe with sr-iov enabled (max_vfs), I cannot see the relevant interfaces with the iproute utility. This is required functionality in order to assign a static MAC address to each VF which are otherwise randomly generated on reboot.

Any suggestions would be greatly appreciated!

Some data points:
  • Server Information
    • Manufacturer: HP
    • Product Name: ProLiant DL165 G7
  • Ethernet Card Information:
    • Intel Corporation 82599EB 10-Gigabit SFI/SFP+
  • uname:
    • Linux 3.0.0-16-server x86_64 GNU/Linux
  • OS Release:
    • Ubuntu 11.10

Steps to reproduce:

1. Determine the ethernet interface assignments:

# dmesg |grep ixgbe\ 0000:05:00.[0-9].*detected|head -n 2

 

[ 8.448031] ixgbe 0000:05:00.0: eth4: detected SFP+: 5

 

[ 8.816035] ixgbe 0000:05:00.1: eth5: detected SFP+: 6

2. Load the module without sr-iov enabled (i.e. no max_vfs) and see that the interfaces are visible:

# modprobe ixgbe

 

# ip link show|grep eth[45]

 

14: eth4: mtu 1500 qdisc noop state DOWN qlen 1000

 

15: eth5: mtu 1500 qdisc noop state DOWN qlen 1000

3. Unload the module and reload with sr-iov enabled

# rmmod ixgbe

 

# modprobe ixgbe max_vfs=63

 

# ip link show|grep eth[45]

 

<<null output!>>

Troubleshooting Steps:

  • /proc/net/dev shows my devices exist at all times

# cat /proc/net/dev|grep eth[45]| cut -d : -f 1

 

eth4

 

eth5

  • configuring the interface via ifconfig is consistent at all times

# ifconfig eth5 10.0.0.1/24

 

# ifconfig eth5

 

eth5 Link encap:Ethernet HWaddr 00:1b:21:b9:a5:e5

 

inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0

# modinfo ixgbe

 

filename: /lib/modules/3.0.0-16-server/kernel/drivers/net/ixgbe/ixgbe.ko

 

version: 3.8.21-NAPI

 

license: GPL

 

description: Intel(R) 10 Gigabit PCI Express Network Driver

 

author: Intel Corporation, <</span>mailto:linux.nics@intel.com linux.nics@intel.com>

 

srcversion: 06E123F39C1AAB9CE66E411

 

# ip -V

 

ip utility, iproute2-ss110317
0 Kudos
8 Replies
idata
Employee
1,197 Views

Additional note:

I also tested this on a seperate, identical DL165G7 with an new 82599EB card and got the same results, so it's very unlikely to be a defective hardware issue.

0 Kudos
Patrick_K_Intel1
Employee
1,197 Views

I'm consulting with my experts on a theory.

In the meantime, if you only create say 10 or 20 VF's, does it work correctly?

I believe this has to do with the large ammount of text that needs to be displayed when you run this command. It will show all ethernet devices - if you only have a single Intel 82599 in your system and nothing else, this will still be 128 Ethernet devices that needs to be dispalyed.

The way these kernel utilities display text is pretty confusing to me, however I do recall in the past that there was challenges associated with them trying to output large amounts of text due to some paging issues or something along those lines.

I'll try to get more information.

0 Kudos
Patrick_K_Intel1
Employee
1,197 Views

My experts tell me that if you get the latest iproute2, this problem should go away. Patches were made for the reasons I explained above.

Please give this a try and report back to the forum on your hopeful success)

thanx,

Patrick

0 Kudos
idata
Employee
1,197 Views

Hey Patrick, your suggestion was spot on!

# modprobe ixgbe max_vfs=20

 

# ip link show

 

.....

 

14: eth4: mtu 1500 qdisc noop state DOWN qlen 1000

 

link/ether 00:1b:21:b9:a5:e4 brd ff:ff:ff:ff:ff:ff

 

vf 0 MAC fe:77:26:31:b2:c0

 

vf 1 MAC 86:b7:4b:4e:38:61

 

vf 2 MAC 7e:9a:ea:ef:ed:8a

 

vf 3 MAC ce:e4:e1:b5:3d:75

 

vf 4 MAC 12:fd:b6:78:5c:8e

 

vf 5 MAC 1e:21:8b:90:5d:a4

 

vf 6 MAC 36:e5:6a:2d:7b:35

 

vf 7 MAC 62:bf:66:99:f9:9c

 

vf 8 MAC c6:15:70:86:87:6d

 

vf 9 MAC f2:53:a4:32:cd:c3

 

vf 10 MAC 06:fe:90:5f:cf:a1

 

vf 11 MAC e2:f1:d3:00:b5:7c

 

vf 12 MAC de:c3:57:36:e0:06

 

vf 13 MAC 8e:cd:c3:ca:3c:fd

 

vf 14 MAC f6:a7:22:ce:29:57

 

vf 15 MAC fa:85:d5:df:a5:fb

 

vf 16 MAC 3e:ef:26:e7:6e:ee

 

vf 17 MAC 9a:28:59:72:c7:3f

 

vf 18 MAC 4e:84:49:60:1a:51

 

vf 19 MAC b6:c0:40:f4:8e:b7

I found that the maximum number I can specify is 37 using either the built-in OS's or self-compled versions of iproute2

I was using the iproute2 source from git so I presume that's the latest. I also grabbed http://kernel.org/pub/linux/utils/net/iproute2/ iproute2-3.3.0 which behaves the same. Do you know where the patches live? I searched around on http://marc.info/?l=linux-netdev&r=1&w=2 linux-netdev with no success.

Thanks again!

0 Kudos
Patrick_K_Intel1
Employee
1,197 Views

I'll follow-up with the engineer who did the patches - may not have been for this problem.

While not as elegant - I believe you can specify a specific PF & VF combination to get data on. Doing this you could, via a script loop through all of the VF's for a PF.

0 Kudos
idata
Employee
1,197 Views

While the folowing does work, like you said it is very kludgy. While the PF's MAC is viewable via /sys/class/net/eth4/address, the VF's MAC is not available anywhere except dmesg (that I know of).

# ip link set eth4 vf 62 mac 0A:0A:0A:0A:0A:0A

 

# dmesg |grep ixgbe |grep 05.*VF\ 62

 

[97507.953387] ixgbe 0000:05:00.0: eth4: IOV: VF 62 is enabled mac 7E:A7:93:3E:0E:DC

 

[100288.404401] ixgbe 0000:05:00.0: setting MAC 0a:0a:0a:0a:0a:0a on VF 62

Thanks, and I'll be looking forward to seeing that patch!

Cheers,

Scott

0 Kudos
idata
Employee
1,197 Views
0 Kudos
Patrick_K_Intel1
Employee
1,197 Views

Thanks for passing that along. Much appreciated.

0 Kudos
Reply