Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

82599EB,not enough MMIO resources for SR-IOV,Centos6.3

idata
Employee
2,588 Views

Dear All,

I'm using latest version of ixgbe , ixgbevf and want to use SR-IOV function with xt-520 T2(dualport),but when I try to load ixgbe driver with this command:

# modprobe ixgbe max_vfs=2,2

It seems that only first port of the card's VFs works,the problem is second port of the card's VFs didn't work, and I found some errors in /var/log messages:

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12

Any help would be greatly appreciated.

below is the related information:

Server: R710

OS: centos6.3

NIC: X520-T2(dualport)

kernel: 2.6.32-279.14.1.el6.x86_64

BIOSVersion: 6.3.0(latest)

BIOS:Inter VT-d/SR-IOV(enabled)

ixgbe:3.11.33(latest)

ixgbevf:2.7.12(latest)

kernel boot parameters: kernel /vmlinuz-2.6.32-279.14.1.el6.x86_64 ro root=UUID=c81b679a-e223-4490-91c0-114af927f28c rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM biosdevname=0 intel_iommu=on

/var/log/messages:

Nov 8 14:56:54 12 kernel: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.11.33

Nov 8 14:56:54 12 kernel: Copyright (c) 1999-2012 Intel Corporation.

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: PCI INT B -> GSI 50 (level, low) -> IRQ 50

Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2

Nov 8 14:56:54 12 kernel: ixgbe: 0000:07:00.0: ixgbe_check_options: FCoE Offload feature enabled

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (unregistered net_device): SR-IOV enabled with 2 VFs

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: FCoE offload feature is not available. Disabling FCoE offload feature

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e2

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: MAC: 2, PHY: 2, PBA No: G21371-003

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Enabled Features: RxQ: 1 TxQ: 1 LRO

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 0 is enabled mac 0E:15:B9:26:B3:7D

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 1 is enabled mac 32:69:2D:16:B9:40

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Intel(R) 10 Gigabit Network Connection

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: PCI INT A -> GSI 40 (level, low) -> IRQ 40

Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2

Nov 8 14:56:54 12 kernel: ixgbe: 0000:07:00.1: ixgbe_check_options: FCoE Offload feature enabled

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: FCoE offload feature is not available. Disabling FCoE offload feature

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e3

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: MAC: 2, PHY: 2, PBA No: G21371-003

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC

Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Intel(R) 10 Gigabit Network Connection

# dmesg |grep -E 'DMA|IOMMU'

ACPI: DMAR 00000000bf3b3668 001C0 (v01 DELL PE_SC3 00000001 DELL 00000001)

DMA 0x00000001 -> 0x00001000

DMA32 0x00001000 -> 0x00100000

DMA zone: 56 pages used for memmap

DMA zone: 102 pages reserved

DMA zone: 3839 pages, LIFO batch:0

DMA32 zone: 14280 pages used for memmap

DMA32 zone: 764849 pages, LIFO batch:31

Intel-IOMMU: enabled

DMAR: Host address width 40

DMAR: DRHD base: 0x000000fed90000 flags: 0x1

IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020fe

DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff

DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff

DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff

DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff

DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff

DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff

DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff

DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff

DMAR: ATSR flags: 0x0

DMAR: Device scope device [0000:00:1a.02] not found

DMAR: Device scope device [0000:00:1a.02] not found

DMAR: Device scope device [0000:00:1d.02] not found

DMAR: Device scope device [0000:00:1d.02] not found

IOMMU 0xfed90000: using Queued invalidation

IOMMU: Setting RMRR:

IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c2000 - 0xbf4c3000]

IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c0000 - 0xbf4c1000]

IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4a7000 - 0xbf4a8000]

IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4a5000 - 0xbf4a6000]

IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4a3000 - 0xbf4a4000]

IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a1000 - 0xbf4a2000]

IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4b1000 - 0xbf4c0000]

IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4b1000 - 0xbf4c0000]

IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4b1000 - 0xbf4c0000]

IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4b1000 - 0xbf4c0000]

IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c8000 - 0xbf4e0000]

IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c8000 - 0xbf4e0000]

IOMMU: Prepare 0-16MiB unity mapping for LPC

IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]

PCI-DMA: Intel(R) Virtualization Technology for Directed I/O

ata1: SATA max UDMA/13...

0 Kudos
1 Reply
Patrick_K_Intel1
Employee
1,173 Views

Thanx for coming to our forum and using Inel Ethernet.

Generally whenever we hear about lack of MMIO space, it is almost always the BIOS. I've done lots of SR-IOV on R710's and Intel 82599 10GbE devices.

You seem to have all the latest goodies - PF and VF driver.

The only variable in here that I've not used is your OS. I've never touched centos. Can you try install RHEL 6.2 and give it a try? Let's remove all unknown variable and see if we can figure it out.

- Patrick

0 Kudos
Reply