x710-da4 negotiating lower PCIe LnkSta and LnkCap

[ChaiTea]

Hello,

Installed a new x710-da4 in an Ubuntu 16.04 chassis. Updated the firmware. the lspci -vv shows the LnkSta shows  'Speed 2.5GT/s, Width x1'. I have changed slots and machines but the LnkSta and LnkCap show the same info. 1st machine has PCIe 2.0 slots 2nd machine tried has PCie 3.0 slots.

Nics from other vendor show correct LnkSta.

~# lspci -s 0000:10:00 -vv
10:00.0 Ethernet controller: Intel Corporation Device 37d0 (rev 04)
Subsystem: Intel Corporation Device 0004
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at c8000000 (64-bit, prefetchable) [size=16M]
Region 3: Memory at cd200000 (64-bit, prefetchable) [size=32K]
Expansion ROM at cc000000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00001000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Vital Product Data
Product Name: Example VPD
Read-only fields:
[V0] Vendor specific:
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number bc-74-b5-ff-ff-1e-f2-f8
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 32, Total VFs: 32, Number of VFs: 0, Function Dependency Link: 00
VF offset: 16, stride: 1, Device ID: 37cd
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000cc200000 (64-bit, prefetchable)
Region 3: Memory at 00000000cd220000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
No steering table available
Capabilities: [1b0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: i40e
Kernel modules: i40e

[ChaiTea]

Other attachements:

[ChaiTea]

I had this in my earlier email, showing the run of nvmupdate below. The card in question still shows up to date, the others show 'update not available'. the executable stops without any update option.

root@SP2callgen01:~/Intel/NVMU_new/700Series/Linux_x64# ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.42.7
Copyright (C) 2013 - 2020 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [*-........]

Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) 82576 Gigabit Dual Port 1.2(1.2) 10C9 00:001 Update not
Network Connection available
02) Intel(R) 82576 Gigabit Dual Port 2.1(2.1) 10E7 00:003 Update not
Server ExpressModule available
03) Intel(R) Ethernet Network Adapter 5.21(5.15) 37D0 00:016 Up to date
X722-4
04) Intel(R) Ethernet Server Adapter 1.99(1.63) 1521 00:020 Update not
I350-T4 available
05) Intel(R) Ethernet Server Adapter 0.147(0.93) 10FB 00:022 Update not
X520-1 available

Tool execution completed with the following status: All operations completed successfully.
Press any key to exit.

[Mike_Intel]

Hello ChaiTea,

Thank you for providing the pictures of the markings of the card and checking that there are 3 PCIe cards installed. Just a question, since there are 3 cards, have you checked if you are updating the firmware of the correct card?

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[Mike_Intel]

Hello ChaiTea,

Thank for providing these details. Let me check this again on why you are getting an error during firmware update.

Please give us 2 to 3 days to provide an update.

However, while checking the firmware of the card, please check and provide the brand and model of the motherboard and kindly update the firmware also.

Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[AlfredoS_Intel]

Hi Chaitea,

Thank you for waiting for our update.

While we were closely checking your concern, we find the need to get a more extensive look on your network adapter.

If possible, please provide us with the front and back pictures of the Ethernet adapter of the X722-DA4. Kindly provide pictures focusing on the markings as well (SN and PBA). The reason why we need this is the markings are not detected on the SSU logs that was provided and you are unable to update the FW of the card. We need to confirm the markings of the card installed in his system.

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[ChaiTea]

The pictures were already uploaded to this thread. They show all markings and writing. I am not sure what other angle I can take pictures from. I even had pictures of the box uploaded earlier. 

[ChaiTea]

BTW, The serial number is in the SSU logs provided earlier =>

Capabilities:"[140] Device Serial Number bc-74-b5-ff-ff-1e-f2-f8"

It matches with the one in the picture, attaching here again. 

[AlfredoS_Intel]

Hi Chaitea,

Thank you for your response and providing more details regarding the issue,

We will continue checking this and the information that provided is invaluable to us.

Please allow us some time to check on this. We hope for your understanding regarding this

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support 

[AlfredoS_Intel]

Hi Chaitea,

Thank you for waiting for our update.

Here are our engineer’s findings regarding your concern:

After further validation, we would to like inform you and in the same way apologize to you that we have confirmed that you are using the latest firmware for X722. The latest NVM version for the X722 is 5.15 which you currently have.

This is based on the nvmupdate.cfg file from Version: 8.15. The X722 uses a different version than all the other 700 series adapters.

https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series

Here is a snippet from the config file that shows this:

BEGIN DEVICE

DEVICENAME: X722

VENDOR: 8086

DEVICE: 37D0

NVM IMAGE: LBG_B2_5p15_CF_FH4x10G.bin

EEPID: 8000275E

REPLACES: 8000156F 80001A3D 80001DF0 80002080 800023C4 8000265B

OROM IMAGE: BootIMG.FLB

EEPROM MAP: iSCSI.txt

RESET TYPE: REBOOT

END DEVICE

The 5p15 portion of the bin file name is the version - 5.15

As for your issue, we suspect that there may be some incompatibility with the motherboard. Please see our questions regarding this:

1. Have you already checked with the Motherboard (MB) manufacturer to check for compatibility?

2. May we ask if you have already tried to update the MB BIOS?

3. We noticed in the SSU that you BIOS was in "Legacy" mode, may we ask if you have already tried changing that to UEFI?

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[Mike_Intel]

Hello ChaiTea,

I hope you had a great weekend. We just want to make a follow up on the information that we asked to further check the issue.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[ChaiTea]

Hello,
I won't have access to system till next year. I do have a follow up questions though.
Are you suggesting that the card is incompatible with the motherboard? Where can I find compatible matrix? I have tried this in two chassis so far. Both show same issue.

[Mike_Intel]

Hello ChaiTea,

We are still in the process of investigating that's why we are also checking the board. The compatibility is always dictated by the board. Is you want to know if a specific card is supported, it is always recommended to check if the board supports it.

Since you don't have access to the server, kindly tell us when can you provide an update for me to know when to make another follow up post.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[Mike_Intel]

Hello ChaiTea,

Just sending another follow up to check when can we make a follow up since you are close during the holidays.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[ChaiTea]

I think we are going round and round with this issue. you are not answering with any specifics.

Compatibility of motherboard is going to be hard to determine. This is an older chassis which only supports pcie gen2 slots. Most definitely this chassis was designed before the  722 card was introduced.

1. I have used ethernet cards from other manufacturers on this chassis. They all negotiate down to gen2 speeds,  why not this one?

2. Do you have a matrix of compatible motherboards?

[Mike_Intel]

Hello ChaiTea,

I am really sorry if you feel that we are going around in circles, we are trying to investigate the cause of the issue so that we can provide recommendations later on.

Here are my answer to your questions.

1. X722-DA4 is a Gen 3.0 PCIe card and your system is a Gen 2.0 PCIe. Not all Gen 3 PCIe cards can do gown to PCIe 2.0.
2. We only have the list below which are Intel boards. Kindly open the link and click on Compatible products. The reason why we only have Intel boards there is because motherboards are dictating the list of components that it can support and the the cards in the market. They do this during the testing stage of the board and they determine if the BIOS and other components works well with the PCIe cards.

https://ark.intel.com/content/www/us/en/ark/products/139351/intel-ethernet-network-adapter-x722-da4.html

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[Mike_Intel]

Hello ChaiTea,

How are you today. I just want to check if you have an update or if you have clarifications regarding this thread.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[Mike_Intel]

Hello ChaiTea,

I am just sending another follow up on the information that I requested about the motherboard.

Since we have not heard back from you, I need to close this inquiry. 

If you need further assistance, please post a new question. 

Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians