Guest OS enumeration for VF PCIe device for 82599

[idata]

I am struk with enabling the MSIx in the Guest OS for 82599.

In hardware MSIx capability is set it has "70" PF driver is loaded to enable the Virtual function in the host then "lspci" command is used to check whether VF is enabled or not .

VF configuration space for msix "capability [70] MSIx " it is showing correct in the host.

But if i map the VF to Guest OS then it is showing "capability [40] MSIx".SO MSIx is enabled is not detect by the hardware and the VF interrupt is not functioning properly

LSPCI output for VFs in Guest

04:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

Subsystem: Intel Corporation Device 0003

Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0

Region 0: [virtual] Memory at df100000 (64-bit, non-prefetchable) [size=16K]

Region 3: [virtual] Memory at df200000 (64-bit, non-prefetchable) [size=16K]

Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express (v0) Endpoint, MSI 00

DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-

MaxPayload 128 bytes, MaxReadReq 128 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port # 0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us

ClockPM- Surprise- LLActRep- BwNot-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

Capabilities: [100 v1] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 0

ARICtl: MFVC- ACS-, Function Group: 0

Kernel driver in use: ixgbevf

Kernel modules: ixgbevf

MAPPED VFS to a GUEST OS -lspci output

00:05.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

00: 86 80 ed 10 07 04 10 00 01 00 00 02 00 00 00 00

10: 00 00 04 f2 00 00 00 00 00 00 00 00 00 40 04 f2

20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 03 00

30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00

40: 11 00 02 80 03 00 00 00 03 20 00 00 00 00 00 00

50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

70: 11 a0 02 00 03 00 00 00 03 20 00 00 00 00 00 00

80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

a0: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

　

00:05.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

Subsystem: Intel Corporation Device 0003

Physical Slot: 5

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0

Region 0: Memory at f2040000 (32-bit, non-prefetchable) [size=16K]

Region 3: Memory at f2044000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] MSI-X: Enable+ Count=3 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Kernel driver in use: ixgbevf

Kernel modules: ixgbevf

Fedora 14 is used in both host and Guest.  

CPU : Intel xeon Hypervisor : QEMU/KVM VMM : Fedora 14 default Kernel Version : 2.6.35

[Mark_H_Intel]

I asked around about your post and heard back from a few developers. In our labs MSI-X capability is displayed in the correct offset of [70]. We do not know why you are seeing something different. Of course there could be something differentwith QEMU or the guest on your setup.

You might consider using RHEL 6.x. The tech brief at http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html covers the configuration and this is a configuration that is known to work. Maybe going through the paper you can find something that will help you in your setup. On the other hand, if the issue is related to differences with any part of the distribution, then this paper might not help you.

When you say, "the VF interrupt is not functioning properly," what is happening? Is the VF able to pass traffic? Are you expecting a different number of interrupts to be used? Please explain further.

What is in the output of the 'cat /proc/interrupts` for both the PF and VF?

Mark H

[idata]

Thanks for the support i will try to load the RHEL 6.x an d thanks for the document

Mean time the information on interrupts

Interrupts for PF

[root@veena /]# lspci | grep 82599

83:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

83:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

[root@veena /]# modprobe ixgbe max_vfs=1

[root@veena /]# lspci | grep 82599

83:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

83:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

83:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

83:10.1 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

[root@veena /]# cat /proc/interrupts

115: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-rx-0

116: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-tx-0

117: 9 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4:lsc

118: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5-rx-0

119: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5-tx-0

120: 8 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5:lsc

121: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-0

122: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-0

123: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6:mbx

124: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-0

125: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-0

126: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7:mbx

I have assigned 83:10.0 device to the guest OS

Interrupts on VF

[root@vm1 VM1]# lspci | grep 82599

00:06.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

[root@vm1 VM1]# cat /proc/interrupts

CPU0

0: 278 IO-APIC-edge timer

1: 199 IO-APIC-edge i8042

2: 0 XT-PIC-XT cascade

4: 2 IO-APIC-edge

8: 0 IO-APIC-edge rtc0

12: 130 IO-APIC-edge i8042

14: 8818 IO-APIC-edge ata_piix

15: 878 IO-APIC-edge ata_piix

34: 459 IO-APIC-fasteoi virtio1

35: 1104 IO-APIC-fasteoi uhci_hcd:usb1, Intel 82801AA-ICH

40: 0 PCI-MSI-edge virtio0-config

41: 8 PCI-MSI-edge virtio0-input

42: 1 PCI-MSI-edge virtio0-output

43: 0 PCI-MSI-edge eth1-rx-0

44: 0 PCI-MSI-edge eth1-tx-0

45: 0 PCI-MSI-edge eth1:mbx

NMI: 0 Non-maskable interrupts

LOC: 36375 Local timer interrupts

SPU: 0 Spurious interrupts

PMI: 0 Performance monitoring interrupts

PND: 0 Performance pending work

RES: 0 Rescheduling interrupts

CAL: 0 Function call interrupts

TLB: 0 TLB shootdowns

TRM: 0 Thermal event interrupts

THR: 0 Threshold APIC interrupts

MCE: 0 Machine check exceptions

MCP: 1 Machine check polls

ERR: 0

MIS: 0

[root@vm1 VM1]# lspci

00:06.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

Subsystem: Intel Corporation Device 0003

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0

Region 0: Memory at f2040000 (32-bit, non-prefetchable) [size=16K]

Region 3: Memory at f2044000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] MSI-X: Enable+ Count=3 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Kernel driver in use: ixgbevf

Kernel modules: ixgbevf

[Patrick_K_Intel1]

My SR-IOV Guru's believe this is likely a Fedora 14 problem. We would like to suggest using the latest Fedora release, or try to reproduce the problem with RHEL 6.2.

It takes a several components to make SR-IOV work, BIOS, platform,end-point device (the Intel 82599) and of course the OS. There are lots and lots of patches to the linux kernel for SR-IOV support. Sometimes not all of those patches make it into a distro release due to any number of reasons.

Please update us on your efforts.

thanks,

Patrick

[idata]

Hi Patrick,

I tried on RHEL 6.2 the previous issue was not there,new issue has been arrived.

Configuration:

1. CONFIG_DMAR_TABLE=y in kernel Config menu,
2. intel_iommu=on in grub.config
3. VT-d enabled in bios.

Scenarios 1:

• Loaded the ixgbe max_vfs=0, and started the testing.Below are dmesg from the kernel. DRHD: handling fault status reg 2DMAR:[DMA Read] Request device [03:00.0] fault addr 619c00000 DMAR:[fault reason 01] Present bit in root entry is clear

Scenarios 2:

• loaded the ixgbe max_vfs=2, and started the testing.
• Assigned the Vfs to the GuestOs.VFs are working fine.

Didn't receive any DMAR error.

If intel_iommu=off or CONFIG_DMAR_TABLE=n,the above issue is not seen and ethernet driver is working fine(only PF driver ).But VFs cannot be assigned to the guest OS.

Can u please tell where it went worng ......................

[Patrick_K_Intel1]

Thanks for trying that and reporting your findings. It would appear to be some kind of DMAR issue - we are preparing to investigate. It makes sense if there is a DMAR problem that VF's assigned to a VM would no work.

To aid in the investigation, our engineers have asked if you are using the inbox (kernel) drivers, or the latest from sourceforge.

thanks,

Patrick

[idata]

I am using the kernel version 3.2.13 rt23.

[Patrick_K_Intel1]

That is a very old driver. I would recommend getting the latest versions (PF and VF) - which is available here:

http://sourceforge.net/projects/e1000/files/ixgbe%20stable/ http://sourceforge.net/projects/e1000/files/ixgbe%20stable/

Please give this a try and let us know how it works.

thanks,

Patrick

[idata]

Hi Patrick,

Thanks for the support................

I will download and test the lastest driver and let u know the result.

Veena

[idata]

It is working with the latest source code Thnaks for the suppot

I am curious to know what is reason for the DMAR issue????

DRHD: handling fault status reg 2

DMAR:[DMA Read] Request device [03:00.0] fault addr 619c00000 DMAR:[fault reason 01] Present bit in root entry is clear

[Patrick_K_Intel1]

Great, I am happy it working and to be able to help.

I do not know what is happening with this DMAR issue you have described. We suspect it may be a BIOS issue, however I will pass along the information you provided to our SR-IOV experts and relay back anything they may offer by way of explaination.