How to enable SR-IOV in I350-T2 NIC

[IS2]

Hello,

I purchased from local reseller Intel I350-T2 NIC but it is missing SR-IOV capability.

According to I350-T2 specification SR-IOV capability should be on-board. How can I enable SR-IOV?

Is there an option to enable it in EEPROM or replace EEPROM?

pci id of interface:

8086:1521 (rev 01) Subsystem: 8086:5002

Thank you

e0thtool -i eth7

driver: igb

version: 3.2.10-k

firmware-version: 1.5-2

bus-info: 0000:04:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

$ sudo lspci -s 04:00.0 -v

04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Subsystem: Intel Corporation Ethernet Server Adapter I350-T2

Flags: bus master, fast devsel, latency 0, IRQ 38

Memory at df200000 (32-bit, non-prefetchable) [size=1M]

Memory at df1f8000 (32-bit, non-prefetchable) [size=16K]

Expansion ROM at df100000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Capabilities: [70] MSI-X: Enable+ Count=10 Masked-

Capabilities: [a0] Express Endpoint, MSI 00

Capabilities: [e0] Vital Product Data

Capabilities: [100] Advanced Error Reporting

Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-04-80-70

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

Capabilities: [1a0] Transaction Processing Hints

Capabilities: [1c0] Latency Tolerance Reporting

Capabilities: [1d0] Access Control Services

Kernel driver in use: igb

Kernel modules: igb

More detailed output:

$ sudo lspci -s 04:00.0 -vvv -n

04:00.0 0200: 8086:1521 (rev 01)

Subsystem: 8086:5002

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 38

Region 0: Memory at df200000 (32-bit, non-prefetchable) [size=1M]

Region 3: Memory at df1f8000 (32-bit, non-prefetchable) [size=16K]

Expansion ROM at df100000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Count=10 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-

LnkCap: Port # 0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <32us

ClockPM- Surprise- LLActRep- BwNot-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-

LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -6dB

Capabilities: [e0] Vital Product Data

Product Name: I350 GbE Controller

Read-only fields:

[V0] Vendor specific: FFV13.0.0

[PN] Part number: G15136

[MN] Manufacture ID: 31 30 32 38

[V1] Vendor specific: DSV1028VPDR.VER1.0

[V3] Vendor specific: DTINIC

[V4] Vendor specific: DCM1001008FC52101008FC53201008FC54301008FC5

[V5] Vendor specific: NPY4

[V6] Vendor specific: PMT13

[V7] Vendor specific: NMVIntel Corp

[RV] Reserved: checksum good, 0 byte(s) reserved

End

Capabilities: [100 v2] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+

AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

Capabilities: [140 v1] Device Serial Number a0-36-9f-ff-ff-04-80-70

Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 1

ARICtl: MFVC- ACS-, Function Group: 0

Capabilities: [1a0 v1] Transaction Processing Hints

Device specific mode supported

Steering table in TPH capability structure

Capabilities: [1c0 v1] Latency Tolerance Reporting

Max snoop latency: 0ns

Max no snoop latency: 0ns

Capabilities: [1d0 v1] Access Control Services

ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

Kernel driver in use: igb

Kernel modules: igb

[Patrick_K_Intel1]

Thanx for posting your question.

That device definitely supports SR-IOV - I've used it myself .

For SR-IOV to function, it requires a number of components to all work together, including the platform BIOS, Chipset, Operating System and the SR-IOV device itself.

If the BIOS or OS doesn't support SR-IOV it will never show up. Both must suppor the feature for even lspci to show the functionality. I'd recommend that you check to see if all the other pieces support SR-IOV.

thanx,

Patrick

[IS2]

Hello Patrick,

I used Intel 10GB interfaces with SR-IOV capability on same machine running Ubntu 12.04.3 server.

Now I evaluating i350-T2 on same machine and I double checked BIOS -- SR-IOV global setting is "enable".

I boot system with "intel_iommu=on" kernel parameter.

Any idea why i350-T2 device does not report its SR-IOV capability?

Can I check it in EEPROM?

Thanks for your help,

Igor

[Patrick_K_Intel1]

There is an additional requirment, one that is not obvous and is very frustrating...

The device - in your case the I350-T2, must be in a slot that support SR-IOV. Some server designs have some PCIe connecors not connected directly to the Chipset and did not use a PCIe switch that supports SR-IOV.

This doc is for 10G, but explains some of these issues:

https://www-ssl.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html Intel® 10 GbE, PCI-SIG* SR-IOV, and Sharing Spec on Linux*

Try putting the device in a different slot (try the one you used for 10G). If that doesn't work then please provide more information on the plat form you are using, uncluding Model, Bios Version, driver versions and kernel log and we'll see what we can do.

- Patrick

[IS2]

Hello Patrick,

I have checked other slot -- same result. Both slots are PCIe v 2, x8.

Platform is Dell PowerEdge R610, BIOS version 6.1.0.

It is running Ubuntu server 12.04.3 LTS x86_64 with kernel 3.2.0-54-generic.

Driver: igb version: 3.2.10-k firmware-version: 1.5-2

Please see attached link to log and debug files.

http://temp-share.com/show/3Yg8b3X2x Temp Share : debug.tgz | Free temporary file hosting

Thanks for your help,

Igor

[Patrick_K_Intel1]

Hi Igor,

I am assuming you purchased this NIC from Dell? I've other posts on this. The bottom-line is that currently (I hear rumors this may be changing) the Intel I350 NICs that Dell sells have SR-IOV disabled.

We (Intel) has no control over this. We make custom versions of our devices for several OEMs, who determine which features they wish to use. In this case the OEM decided not to initially support SR-IOV on Intel's I350 devices.

If you were to purchase a retail version of an I350 you would find that SR-IOV would work fine. Though that might void your system warranty with you server manufacturer since it wouldn't be officially blessed hardware.

I would recommend you contact Dell for advice and instructions.

- Patrick

[IS2]

I thought OEM have to change the subsystem vendor ID like in other posts on same issue.

Ok, I will check with my local reseller regarding this NIC OEM..

Thanks a lot Patrick for your help

Igor