Why LnkSta and LnkCtl show different with X710?

[JMa11]

Hi,

My lspci -vvv output for the X710DA4 Ethernet Adapter shows LnkSta to be 8GT/s but LnkCtl is 2.5GT/s.

Motherboard used is the S2600CW with dual E5-2697Av4 CPU.

Thanks....

[idata]

Hi JacksonM,

Thank you for posting your query in Wired Ethernet Communities.X710-DA4 should work according to specs which is PCI-E Gen 3 x8 (8GT/s), as to why the LnkCtl shows 2.5GT/s. Kindly share additional information below for us to further investigate.1. ethtool - i output to show the driver version and firmware version of your X710-DA42. BIOS and FW version of your S2600CW3. Have you tried to install the X710-DA4 on other PCI-E Gen 3 x8 or x16 slots?4. Have you tried installing other PCI-E Gen3 x8 on the same slot on where the X710-DA4 was installed?Regards,Vince

[JMa11]

Thanks, Vince. I captured a few more output for you to see.

The driver loaded is the i40e.

DPDK output: PMD:

eth_i40e_dev_init(): FW5.0 API 1.5 NVM 05.00.04 eetrack 800024cd

(But we have also disabled the DPDK and put driver to kernel. Output from below is without DPDK).

Yes, we have tried plugging into other slots as well.

Motherboard BIOS (Note: I am using the S2600CO board here. And also upgraded my BIOS for the board. The same results with Cottonwood Pass board)

SE5C600.86B.02.06.0005.080320162159

Ethtool -I output

driver: i40e

version: 1.5.16

firmware-version: 5.04 0x800024cd 0.0.0

bus-info: 0000:04:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

Here's the lspci output (LnkSta and LnkCtl2)

LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 4s to 13s, TimeoutDis-, LTR-, OBFF Disabled

LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

Kernel driver in use: i40e

d0: 01 00 1e 00 00 00 00 00 00 00 00 00 00 00 00 00

It seems that the motherboard and X710 card cannot negotiate the speed. On another system (Advantech motherboard), where I was able to see 8GT/s on LnkCtl2, the first byte of the d0 register is 03.

Hope it is clear. Let me know if you need more details.

Appreciate your help

Jackson

[idata]

Hi Jackson, thanks for sharing the information, i'm currently checking your concern and will update this thread as soon as possible.

regards,Vince

[JMa11]

Hi, Just a bit more info to update:

I have another system (Advantech ASMB-913) which came up with the Gen3 on LnkCtl2. It's the same Ubuntu OS, same batch of 10G NIC (though not the exact same X710 card).

Any leads?

Thanks,

Jackson

[idata]

Hi Jackson,

Thanks for the additional details, we're still investigating your issue. Please share which PCIe slot the X710 was installed on S2600CO.Regards,Vince

[JMa11]

Hi, Vince:

On the S2600CO, it was connected to Slot 5 on the motherboard.

Thanks,

Jackson

[idata]

Thanks Jackson, have you tried connecting to other slots such as slot 4 for further isolation? are there other add-in cards installed on the S2600CO?

regards,Vince

[JMa11]

Yes, Vince, I have installed the X710 card on other slots as well. It was the only card I plugged into the system.

[idata]

Thanks Jackson, have you tried to install one of the supported PCIe* Gen 3 add-in adapters mentioned in the site below to further isolate the issue?

http://www.intel.com/content/www/us/en/support/boards-and-kits/000007457.htmlUnder your system's BIOS, Advanced, Processor Configuration, the processor has label of "V1" or "V2"? Example - E52640 V2 @ 2.00GHzRegards,Vince

[JMa11]

Thanks for the reply, Vince.

No, I was not aware of this list of PCI-e peripherals.

The processors are the v4 Type. It's the E5-2697Av4 with S2600CW-R motherboard. Since the X710DA4 card is also from Intel, I just assume all will come up as Gen3 without issue. They should work as advertised...

Also, I have plugged in a Coletto Creek (Intel 8950 QAT) card and it showed up on lspci as Gen2 (5.0GT/s). Again, it was detected as Gen3 x16.

This leads me to think maybe it's a motherboard problem? How does the motherboard affect the speed of the PCIe?

Thanks,

Jackson

[JMa11]

Here's the Coleto Creek output: (Sorry about the misspell earlier)

83:00.0 Co-processor: Intel Corporation Coleto Creek PCIe Endpoint

Subsystem: Intel Corporation Device 35c8

Physical Slot: 0-1

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0

Interrupt: pin A routed to IRQ 10

Region 0: Memory at 38007fe00000 (64-bit, prefetchable) [size=512K]

Region 2: Memory at c8000000 (64-bit, non-prefetchable) [size=256K]

Region 4: Memory at c8040000 (64-bit, non-prefetchable) [size=256K]

Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [60] MSI-X: Enable- Count=33 Masked-

Vector table: BAR=2 offset=0003b000

PBA: BAR=2 offset=0003b800

Capabilities: [6c] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [74] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-

LnkCap: Port # 0, Speed 5GT/s, Width x16, ASPM L0s, Exit Latency L0s <512ns, L1 unlimited

ClockPM- Surprise- LLActRep- BwNot-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

[idata]

Thanks Jackson, kindly confirm if I've captured the correct details from your testing.

1. S2600CO and X710 - LnkSta = 8GT/s, LnkCtl2 = 2.5GT/s2. S2600CW and X710 - LnkSta = 8GT/s, LnkCtl2 = 2.5GT/s3. S2600CW and QAT 8950 - LnkSta = 5GT/s, LnkCtl2 = 5GT/s4. Advantech ASMB-913 and X710 - LnkSta = 8GT/s, LnkCtl2 = 8GT/sregards,Vince

[JMa11]

Hi, Vince:

Yes, you have captured the details perfectly.

Thanks,

Jackson

[JMa11]

Hi, Vince:

Just to keep updated, I have the PCIe Parameters on the setup page set to both and . Neither change how the "lspci" reports the state.

Thanks.

[JMa11]

Hi, Vince:

Seems like we may have an answer (or a closing to one).The Link Status Register (LnkSta) is the indication of the negotiated speed. While the Link Control Register is not directly related.

Thanks...

[idata]

Hi Jackson, we've also confirmed from our end that the link control register does not represent the speed of the link.

Please let us know if you still have other assistance needed.regards,Vince

[JMa11]

Thank you, Vince.