Community
cancel
Showing results for 
Search instead for 
Did you mean: 
idata
Community Manager
2,484 Views

a problem with 82599 using SR-IOV in linux

Hi,

Now I am doing test about sr-iov. I can setup vfs using 82576 and pass it to VMs with xen 3.2 and kernel 18 in CENTOS 5.4. However, when I change it to 82599, no VF appears any more. If I use "dmesg" to see what happens, it displays "not enough MMIO resource for SR-IOV" . First ,we think that it may be caused by the OS, but we have tried several OS including Centos 5.4(i386 and X64),5.6, REDHAT 5.5,5.6 and the problem remains.Now we really don't know what's the problem.Thanks very much!

If anyone happen to know what to do with it, I hope you can give me a hand.

0 Kudos
44 Replies
idata
Community Manager
10 Views

Hi ,

Another question about 82599 chip...

There are two 10G ports in 82599, each one has one PF and maximum 63 VFs. I know in one port, the PF and VF can communicate with each other by internal switch in 82588, how about the VF between different port? Can they communicate each other by the internal switch in 82599?

Thanks!

Mark

 

Patrick_K_Intel1
Employee
10 Views

You are correct that VF's on the same PF can communicate with each other through the internal switch.

However VF's on different PF's must communicate via an extneral switch.

Thanks for visiting the blog and I hope this was helpful.

- Patrick

idata
Community Manager
10 Views

Hello,

I ended up buying an I-350 T2 so I could use the VFs in both Windows and Linux VMs. It has been working great for a month, with traffic mainly going to the rest of the network (via gigabit switch).

I am now having an issue communicating between VMs. Both PFs are connected to a gigabit switch.

The host can communicate with both Windows and Linux VMs. The issue is in the communication between VMs. It fails using VFs on the same PF or across 2 PFs (and the switch).

Any advise on where to look for the issues?

I am running Centos 6.2 and KVM. Windows VM is Win 2008 R2. Linux VM is Centos 6.2. Both VMs are fully up to date.

Thank you for your help.

idata
Community Manager
10 Views

Hello,

A little more info:

- Communication works between Windows VMs

- Communication works from Windows VM to Linux VM (ping, shares, etc)

- Communication does not work between Linux VMs

- Communication does not work from Linux VM to Windows VM

Linux VM igbvf driver version 2.0.0-k. Is that correct?

The latest driver from the Intel website is 1.1.5

Linux Host igb driver version 3.4.7

Thank you

idata
Community Manager
10 Views

Hello,

I installed igbvf driver version 1.1.5 from SourceForge, rebooted, and had the same issues.

Still can not see other VMs that use a VF of the same I350-T2 NIC.

Thank you very much for your help.

Patrick_K_Intel1
Employee
10 Views

My guess is the firewall configuration on your linux VM's. I always have to configure them for communication. Please take a look at that. First disable the firewall completely and see if you can communicate as you desire and go from there.

Let us know what you find.

thanx,

Patrick

idata
Community Manager
10 Views

Patrick,

thank you for your help.

The firewall does not seem to be the issue. I get the same behavior. I have 4 VMs: 2 Windows (let's call them W1 and W2) and 2 Linux (L1 and L2). I also have a Linux host (Host) that connect thru a gigabit switch.

W1 ping to W2: OK

W2 ping to W1: OK

W1 ping to either L1 or L2: OK

W2 ping to either L1 or L2: OK

L1 or L2 ping to W1 or W2 : Fails

L1 ping to L2: Fails

L2 ping to L1: Fails

Host ping to L1, L2, W1, W2: OK

L1, L2, W1, W2 ping to Host: OK

Seems like packets going from the Linux VM that should reach another VF get blocked.

By the way, all VMs use 2 VFs each. As far as I can tell, I pass a VF from each PF to each VM.

Does the rule that VFs with even numbers belong to one PF and the ones with odd numbers to the other still apply to the I350?

For example:

05:10.0 -> PF 0

05:10.1 -> PF 1

05:10.4 -> PF 0

05:10.5 -> PF 1

Thanks again for your help.

Patrick_K_Intel1
Employee
10 Views

If you are able to ping from W1 and W2 to L1 and L2, then this means that L1 and L2 are successfully sending back the ping response to the Windows VM's.

This tells me that the VF's (and corrosponding drivers) are working correctly.

How do you have the two VF's configured?

idata
Community Manager
10 Views

Patrick,

Thank you for your help.

You have a good point.

The VF in the Linux VMs are configured with static IP. I also configured the VF in the Win VM with static IPs.

OS NIC IP PF:VF

Win 2k8R2 # 0 .40 00:00

Win 2k8R2 # 1 .41 01:00

Linux # 0 .50 00:01

Linux # 1 .51 01:01

.50 ping to .40 = dead

.50 ping to .41 = OK

.51 ping to .40 = OK

.51 ping to .41 = Dead

It looks like there is no response to ping when both VF are on the same PF. Pings across the switch work.

Is there any way to verify that the internal switch is enabled?

Thank you for your help

Patrick_K_Intel1
Employee
10 Views

The internal switch is definately enabled, or else no VF would be able to communicate.

Can the PF's communicate with each other? Are they configured with VLAN's or different subnets?

Are the MAC addresses all unique?

Earlier you said that each VM has 2 VF's - do you configure both? Different IP's, or are they teamed?

idata
Community Manager
10 Views

Patrick,

Thank you for your help.

All these VMs are at home in a very boring network: no VLANs, single subnet 192.168.1.0.

Each VM has 2 VF (one from each PF). They are configured with static consecutive IP numbers: 40&41, 50&51, 55&56, 118&119.

I program the MACs for the VFs with the script that launches the VMs in order to have consistent MAC between reboots. I use MACs ending in PF:VF with PF from 0 to 1 and VF from 0 to 5. The driver in the host is set to enable 6 VF per PF.

Except for the fact that pings fail between some combinations of VM, the rest of the communication seems to work well from the rest of the network. Independent physical PCs can see all the VM without issues at all their IPs. I just pinged all the IPs listed above and all respond immediately (less than 1 ms).

I will need to test if the PFs talk to each other. I can not run those tests from the office without risking leaving myself disconnected from the host.I will test it tonight.

Thanks again for your help.

idata
Community Manager
10 Views

Patrick,

Beginners ignorance: I can specify what NIC to ping from. I did not know that I do not need to disable the other NICs to select the source of the ping...

The 2 PFs with IP .32 & .33 talk to each other in both directions.

Patrick_K_Intel1
Employee
10 Views

Can you do an ip link show on each PF and provide the output.

thanx,

Patrick

idata
Community Manager
10 Views

Patrick,

After I rebooted the server, I lost half of the VFs.

Is it an issue that 2 NICs are sharing the same IRQ?

03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

Subsystem: Super Micro Computer Inc Device 10d3

Flags: bus master, fast devsel, latency 0, IRQ 17

Memory at fdce0000 (32-bit, non-prefetchable) [size=128K]

I/O ports at d800 [size=32]

Memory at fdcdc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [c8] Power Management version 2

Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+

Capabilities: [e0] Express Endpoint, MSI 00

Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-

Capabilities: [100] Advanced Error Reporting

Capabilities: [140] Device Serial Number 00-25-90-ff-ff-4b-6d-18

Kernel driver in use: e1000e

Kernel modules: e1000e

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

Subsystem: Super Micro Computer Inc Device 10d3

Flags: bus master, fast devsel, latency 0, IRQ 18

Memory at fdde0000 (32-bit, non-prefetchable) [size=128K]

I/O ports at e800 [size=32]

Memory at fdddc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [c8] Power Management version 2

Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+

Capabilities: [e0] Express Endpoint, MSI 00

Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-

Capabilities: [100] Advanced Error Reporting

Capabilities: [140] Device Serial Number 00-25-90-ff-ff-4b-6d-19

Kernel driver in use: e1000e

Kernel modules: e1000e

05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Subsystem: Intel Corporation Ethernet Server Adapter I350-T2

Flags: bus master, fast devsel, latency 0, IRQ 16

Memory at fe800000 (32-bit, non-prefetchable) [size=1M]

Memory at fde7c000 (32-bit, non-prefetchable) [size=16K]

Expansion ROM at fde80000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Capabilities: [70] MSI-X: Enable+ Count=10 Masked-

Capabilities: [a0] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-03-ee-86

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

Capabilities: [1a0] Transaction Processing Hints

Capabilities: [1c0] Latency Tolerance Reporting

Capabilities: [1d0] Access Control Services

Kernel driver in use: igb

Kernel modules: igb

05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Subsystem: Intel Corporation Ethernet Server Adapter I350-T2

Flags: bus master, fast devsel, latency 0, IRQ 17

Memory at fea00000 (32-bit, non-prefetchable) [size=1M]

Memory at fe9fc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Capabilities: [70] MSI-X: Enable+ Count=10 Masked-

Capabilities: [a0] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-03-ee-86

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

Capabilities: [1a0] Transaction Processing Hints

Capabilities: [1d0] Access Control Services

Kernel driver in use: igb

Kernel modules: igb

The output of ip link show is:

1: lo: mtu 16436 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:25:90:4b:6d:18 brd ff:ff:ff:ff:ff:ff

3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:25:90:4b:6d:19 brd ff:ff:ff:ff:ff:ff

4: eth2: mtu 1500 qdisc mq state UP qlen 1000

link/ether a0:36:9f:03:ee:86 brd ff:ff:ff:ff:ff:ff

vf 0 MAC da:96:f0:e3:d5:a8

vf 1 MAC ee:b3:6e:8f:e1:9a

vf 2 MAC 7e:87:4e:d9:5e:9f

vf 3 MAC ae:6c:e9:5e:56:16

vf 4 MAC 06:a2:2d:c1:f7:53

vf 5 MAC 1a:35:b5:81:44:71

5: eth3: mtu 1500 qdisc mq state UP qlen 1000

link/ether a0:36:9f:03:ee:87 brd ff:ff:ff:ff:ff:ff

6: eth4: mtu 1500 qdisc noop state DOWN qlen 1000

link/ether ae:ea:c6:0d:f0:44 brd ff:ff:ff:ff:ff:ff

7: eth5: mtu 1500 qdisc noop state DOWN qlen 1000

link/ether 46:5e:11:bb:b1:8b brd ff:ff:ff:ff:ff:ff

8: eth6: mtu 1500 qdisc noop state DOWN qlen 1000

link/ether 46:dc:23:45:4a:a3 brd ff:ff:ff:ff:ff:ff

9: eth7: mtu 1500 qdisc noop state DOWN qlen 1000

link/ether 3a:ff:e5:67:49:5e brd ff:ff:ff:ff:ff:ff

10: eth8: mtu 1500 qdisc noop state DOWN qlen 1000

link/ether 3e:8a:d7:00:a1:69 brd ff:ff:ff:ff:ff:ff

11: eth9: mtu 1500 qdisc noop state DOWN qlen 1000

link/ether e6:40:38:06:8d:8e brd ff:ff:ff:ff:ff:ff

12: br0: mtu 1500 qdisc noqueue state UNKNOWN

link/ether 00:25:90:4b:6d:18 brd ff:ff:ff:ff:ff:ff

As far as I can tell the PF that is sharing IRQ 17 with the other NIC is the one that is not generating VFs.

The system has the latest igb (3.4.7) and igbvf (1.1.5)

Any idea where to go next?

Thank you for your help !!!

Patrick_K_Intel1
Employee
10 Views

when you specify max_vfs, do you use just 1 number or 2?

The kernel driver (the 'inbox' driver) only takes a single parameter and makes that many VF's for all the devices that use that driver for SR-IOV.

The one you downloaded allows you to specify the # of VF's per port, separated by a comma.

My guess is that you have max_vfs=6. If you have max_vfs=6,6 you should be back to where you were.

I do not know where all those extra eth devices are coming from - they do not have the same MAC address as the VF's listed for eth 2.

- Patrick

idata
Community Manager
10 Views

Patrick,

you guessed right. I was passing max_vfs=6 to the driver. Now I have 6 VFs per PF by passing max_vfs=6,6

I will check if the VMs can see each other later (possibly tomorrow).

I am also trying to solve the issue that every time that I restart the host I get different MACs for the VFs. I am currently changing the MACs with a script.

Do you know any wait to make the VF MACs permanent? Some igb driver parameter?

Thank you very much for your help.

idata
Community Manager
10 Views

Patrick,

Thank you very much for your help and patience.

Finally everything works now.

The changes from when I had problems to now is the use of the driver directly from SourceForge. The system was rebooted a lot of times when I was not able to get VFs from one of the PFs. As you pointed out, the Intel driver takes 2 numbers for the max_vfs parameter (I assume it will take 4 numbers in a I350-T4, right?). My mistake.

Again, thank you very much for your help.

Patrick_K_Intel1
Employee
10 Views

Phew! And Great!

Very happy all is well now. Yes, you can do 4 numbers for a I350-T4, and 6 numbers if you have a T4 and a T6 :-) Remember - this option is only for the SourceForge driver, the kernel driver only takes a single parameter and it is used for all ports.

The static MAC addresses is indeed a problem. One that we can't really control. Only solution I am aware of is exactly what you are doing, which is to set the MAC for each VF yourself.

Come back anytime!

- Patrick

idata
Community Manager
10 Views

Patrick,

One last questions (for now):

Is there a good guide for the parameters that I can pass to the igb and igbvf drivers?

Thank you very much for your help.

Patrick_K_Intel1
Employee
10 Views