E810-XXVDA2 unable to establish link at 25G with sfp28 optical transceivers

[kishkaru]

Hello, I have a brand new E810-XXVDA2 network adapter (Intel branded, not OEM), and I'm setting it up for the first time. I want to connect it to my switch, which is a Ubiquiti usw-pro-aggregation, on an SFP28 port.

The client is Windows Server 2022. I have the latest Intel driver for Windows Server 2022 installed, v 1.12.164.0, which came from the Intel v28 release package.

If I use a SFP28 DAC cable between the switch and the E810 nic, then the connection establishes at 25G immediately. However, when I use SFP28 optic transceivers, the link does not establish.

I have tried a loopback test using the SFP28 optics from the switch port-to-port and that works at 25G. I have also tried a loopback on the E810 nic port-to-port and that works too at 25G. So this tells me both the optic transceivers and cable is fine.

If I use SFP+ (10G) optics, or force the connection to 10G using SFP28 optics, then the link establishes at 10G. So the issue seems to be isolated to the 25G connection.

I'm guessing there is a failure to auto-negotiate the connection between the switch and the nic. I have tried forcing 25G link speed on both sides, but no luck. I also played around with the FEC (Forward Error Correction) settings on both sides, but no luck. The Ubiquiti switch port uses FC-FEC (BASE-R) for FEC. It can also be configured to disable FEC.

Any ideas?

[Irwan_Intel]

Hi kishkaru,

Thank you for your post on Intel Ethernet Communities.

I have forwarded your issue to our next level for further investigation and troubleshooting.

In the meantime, I wanted to confirm some details that could assist us in resolving your problem:

1)Could you please let us know the model of your SFP28 Optics transceiver?

2)Have you attempted to disable auto-negotiation on both ends and manually set the speed and duplex to 25G full duplex?

We look forward to hearing back from you with this information.

Best regards,

Irwan_Intel

[kishkaru]

1) Transceiver: Ubiquiti UACC-OM-SFP-28-SR . I have also tried a generic SFP28 transceiver that is coded for Intel. Neither one is working at 25G speeds, but works at 10G speeds. Both passes loopback test at 25G speeds on both the Ubiquiti switch and on the E810 nic.

2) Yes, I have tried forcing 25G link speed on both ends, but no luck.

[Irwan_Intel]

Hi kishkaru,

Thank you for the informations provided. This sound like some compatibility issue between the switch and the network card. We will dig further into this matter and see if we have any information that can help to resolve this problem.

Have you also tried reaching out to Ubiquity support? They might have some good advice on how to deal with this problem as well.

We appreciate your patience as we work to get everything sorted out.

Regards,

Irwan_Intel

[kishkaru]

Yes, I have reached out to Ubiquiti support. Unfortunately they have not responded to me yet. They typically take a long time to respond.

I have another machine that is running Linux, and with the same nic model (also a brand new E810XXV-DA2). That one also has this same issue with unable to establish a link using SFP28 optics.

The system is running on Linux kernel 6.2.10. Device info:

$ lspci -vv

11:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
        Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-2
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 24
        IOMMU group: 21
        Region 0: Memory at 850000000 (64-bit, prefetchable) [size=32M]
        Region 3: Memory at 856000000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at b0200000 [disabled] [size=1M]
        Capabilities: <access denied>
        Kernel driver in use: ice
        Kernel modules: ice

$ inxi -Fxz

  Device-3: Intel Ethernet E810-XXV for SFP driver: ice v: kernel port: N/A
    bus-ID: 11:00.0
  IF: enp17s0f0np0 state: down mac: <filter>
  Device-4: Intel Ethernet E810-XXV for SFP driver: ice v: kernel port: N/A
    bus-ID: 11:00.1
  IF: enp17s0f1np1 state: down mac: <filter>

Device is using ice driver:

$ modinfo ice

filename:       /lib/modules/6.2.10-arch1-1/kernel/drivers/net/ethernet/intel/ice/ice.ko.zst
firmware:       intel/ice/ddp/ice.pkg
license:        GPL v2
description:    Intel(R) Ethernet Connection E800 Series Linux Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     0CBF14DA5BAAC5EFC0EC9EB
alias:          pci:v00008086d00001888sv*sd*bc*sc*i*
...
depends:        
retpoline:      Y
intree:         Y
name:           ice
vermagic:       6.2.10-arch1-1 SMP preempt mod_unload 
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        0A:22:1F:E6:B0:12:DD:EC:05:08:BA:27:E4:E9:98:7F:6CA:0C:1F
sig_hashalgo:   sha512
signature:      30:64:02:30:04:79:39:CB:BF:74:A2:84:A3:17:07:DA:85:10:F5:D0:
                00:DB:46:0A:14A:79:5C:22:B3:84:24:B5:0C:26:6F:87:DC:78:6F:
                DC:89:15:FD:2F:B9:E1:67:0C:75:97:0A:02:30:28:7C:33:5A0:BF:
                94:B2:E2:5A:9B:55:65:D1:C1:F0:F5:61:62:C5:06:54:E5:A9:8A:06:
                D8:76:5C:3B:07:66:C6:75:D5:07:6C:85:2B:DF:3C:88:48:1A:2F:A5:
                13:7D
parm:           debug:netif level (0=none,...,16=all) (int)

 Port info:

$ sudo ethtool enp17s0f0np0

Settings for enp17s0f0np0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
                                25000baseCR/Full
                                25000baseSR/Full
                                1000baseX/Full
                                10000baseSR/Full
                                10000baseLR/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: None
        Advertised link modes:  25000baseSR/Full
                                10000baseSR/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: None
        Speed: 25000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no 

[Irwan_Intel]

Hi kishkaru,

Thanks for giving us the information. Unfortunately, we have not done testing with that switch.

As suggested by our next level, Can the switch be configured for RS-FEC rather than FC-FEC? Our NIC will try RS-FEC first.

Please let us know if you have any questions or concerns.

Regards,

Irwan_Intel

[kishkaru]

No, unfortunately the switch port can only be configured for "FEC on" or "FEC off". The "FEC on" setting is FC-FEC (BASE-R).

[Irwan_Intel]

Hi kishkaru,

We talked to our next level and concluded that we don't have any more ideas to suggest since we haven't tested the switch and only verified that the connection are working when connecting port to port. Our suggestion is to contact the manufacturer of the switch and ask if there are any specific settings on the ports that might help.

Please let us know if you have further questions or concerns.

Regards,

Irwan_Intel

[Irwan_Intel]

Hi kishkaru,

I hope this message finds you in good health.

I would like to follow up with you regarding your issue. Have you had the chance to reach out to the switch manufacturer and share their findings with us?

Please let me know if you have any further questions or concerns that require our attention.Otherwise, I will proceed to close this ticket.

Thank you for your time and have a great day!

Regards,

Irwan_Intel

[kishkaru]

Hello Irwan,

Thanks for checking in. Unfortunately I am still stuck with where I was earlier.

I reached out to Ubiquiti support a month ago, creating a support request, but they are not replying. I am truly disappointed in their lack of support. I've been falling back by running at 10GBe mode in the meantime, which defeats the purpose of having the 25GBe switch and the new E810XXV nics. If you have any other ideas, definitely let me know.

[kishkaru]

Ubiquiti support has finally replied to me. Their conclusion is that since the loopback test works on the usw-pro-aggregation switch with their SFP28 transceiver, there is nothing wrong with the switch or the transceiver. They are blaming the client nic (the E810XXV) for the problem of compatibility. Now we're at a classic case of one side blaming the other for the problem, and neither side is willing to help diagnose the issue.

Their solution to me is to simply use DAC cables, since that works. Using a DAC cable is not a solution. DACs cannot be used in long runs or used with patch panels. DAC vs optics use cases are different.

[Irwan_Intel]

Hi kishkaru,

If the switch is compliant with IEEE standards, it is required to support RS-FEC with optics. Therefore, if the lack of RS-FEC support is the issue, it can be considered an issue on Ubiquiti's side.

To address this, a potential workaround is to configure the E810 by setting the link mode to 25G-AUI-AOC/ACC with FEC disabled, instead of using 25GBASE-SR. This effectively instructs the E810 to establish a link as if an AOC is connected rather than an SR module. If you are customizing your own optical transceivers, you have the option to modify the transceiver's EEPROM to indicate that it is an AOC rather than a 25GBASE-SR module. In such cases, make sure to set the module's compliance code to the following value:

Please let me know if you have further questions or concerns.

Regards,

Irwan_Intel

[kishkaru]

Thanks for the followup. Where do I set the link mode to "25G-AUI-AOC/ACC" on the E810? If you've attached an image, it isn't displaying for me.

Does this "trick" require modifying EEPROM on the SFP28 transceiver to indicate AOC on both ends of the link?

And finally regarding the Unifi USW-Pro-Aggregation, Ubiquiti doesn't say it is IEEE compliant. It simply says "Certifications: CE, FCC, IC".

Which specific IEEE standard requires a SFP28 switch to support RS-FEC with optics?

[Irwan_Intel]

Hi kishkaru,

According to our next level, the specific IEEE standard that requires a SFP28 switch to support RS-FEC with optics is IEEE 802.3.

We have found some articles that you might find useful for reference.

https://blogs.cisco.com/sp/dont-mix-up-your-fecs

Regarding your question about setting the link mode to "25G-AUI-AOC/ACC" on the E810, unfortunately, we cannot provide assistance with transceiver customization. The suggestion was proposed by an engineer as a potential workaround, but we have not tested it ourselves.

If you have any further questions or concerns, please feel free to let us know.

Regards,

Irwan_Intel

[Irwan_Intel]

Hi kishkaru,

I'm just checking in to see if you've had a chance to review our previous post. Please inform us if you have any additional questions or concerns.

Regards,

Irwan_Intel

[kishkaru]

Hey Irwan, sorry for the late response.

I asked Ubiquiti support regarding IEEE 802.3 compliance for the usw-aggregation-pro switch and am still waiting for a reply.

Also thanks for the helpful post you linked.

I'm currently hunting for SFP28 transceivers that state explicitly that they support FC-FEC or no FEC. Hopefully if I can get a pair of those, I can configure both ends of the link to communicate at FC-FEC. I have a feeling that the generic SFP28 transceiver that I have, and the Ubiquiti's own UACC-OM-SFP-28-SR   does not support FC-FEC. Which is ironic because their switch does not support RS-FEC. Although the loopback test working does confuse matters.

[Irwan_Intel]

Hi kishkaru,

I appreciate the information you provided. I wanted to follow up and check if you have received a response from Ubiquiti. Additionally, if you have any further questions or concerns, please let us know.

Regards,

Irwan_Intel

[kishkaru]

Yes, the official response from Ubiquiti support is:

1. It is not possible to set RS-FEC for the switch port. They completely dodged my question about IEEE 802.3 compatibility.

2. It is not possible to configure the FEC at all, officially. Using the switch CLI is not supported by Ubiquiti, they only allow config changes from the Unifi Network application.

It seems like they really don't want end users messing with the FEC settings. Unfortunately, it doesn't seem like I'm going to get any more information out of Ubiquiti support.

----

In other news, I procured some SFP28 transceivers that state explicitly that they support FC-FEC or no FEC. Unfortunately, those also don't work either, even when both sides of the link are explicitly set to FC-FEC or No FEC.

Furthermore, after having them plugged into the E810 NIC and the switch for about a day, it caused a Windows kernel panic. Checking the Windows Event Log, under System, I see hundreds of these error messages, emitted continuously multiple times a second, from "icea" with "PROBLEM: A parallel fault was detected.". See attached screenshot:

Eventually there was a kernel panic, which caused a hard system reboot. I don't know if the two are related. Unplugging the SFP28 module at either end of the link stopped these messages from being emitted.

Are these errors thrown by the ice driver when it's unable to establish a 25G link with the switch? Does this error at all help diagnose why the link with the switch is not establishing?

EDIT: looks like the error is thrown from port #2 on the E810 nic. That port is connected via a SFP28 DAC to the switch, so is likely unrelated?

[kishkaru]

Another update: As suggested earlier, I have tried an AOC (Active Optical Cable) with SFP28. But unfortunately still no luck at 25G link. 10G works as usual.

One thing I noticed that at least on the switch side, the cable is being detected as "25GBase-SR", rather than "25GBASE-AOC". I don't know how to retrieve the cable stats on the E810 nic side.

Earlier you mentioned:

---

configure the E810 by setting the link mode to 25G-AUI-AOC/ACC with FEC disabled, instead of using 25GBASE-SR.

---

How do I configure this link mode on the E810? Now that I have a real SFP28 AOC, there shouldn't be any need to reprogram any EEPROM on the transceivers.

[kishkaru · Accepted solution]

Update: I got the link working at 25G!

Solution = update the firmware (NVM) of your Intel E810 NIC to 4.20 or higher.

-----

I saw several other people talking about firmware (NVM) update for their nics  in this forums. That got me thinking to searching to see if there's an NVM update for my E810XXV-DA2, and I noticed there was one (4.20) released on 2/17/2023.

I installed the update:

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.39.32.6
Copyright(C) 2013 - 2023 Intel Corporation.


WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [***.......]


Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Network Adapter   3.32(3.20)   159B 00:017 Update        
    E810-XXV-2                                                   available     

Options: Adapter Index List (comma-separated), [A]ll, e[X]it
Enter selection: A
Would you like to back up the NVM images? [Y]es/[N]o: Y
Update in progress. This operation may take several minutes.
[**........]


Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Network Adapter   4.32(4.20)   159B 00:017 Update        
    E810-XXV-2                                                   successful    


Tool execution completed with the following status: All operations completed successfully.
Press any key to exit.

 which updated the NVM of the nic from 3.20 to 4.20.

Immediately after the update, the link established at 25G! No configuration needed. Both link speed and FEC was set to "auto".

I tried all sorts of SFP28 transceivers, including the Ubiquiti one (UACC-OM-SFP-28-SR), the Intel-coded one, the SFP28 transceiver supporting FC-FEC, and the SFP28 AOC. They all worked after the NVM update!

------

So it turned out to be an issue on Intel's side after all, the Ubiquiti switch wasn't to blame.

Intel support should've noticed that my NVM (3.20) was outdated and suggested to upgrade to the latest version. If we had done this from the start, the problem would've been solved instead of dragging this out 2+ months.

There seems to be several other bugfixes. The "Media Type" now shows as "Fiber-SR", and the "Negotiated Link Speed" shows as "16.0 GT/s". These were not being populated earlier.

[Irwan_Intel]

Hi kishkaru,

Thank you for sharing your experience and findings with the firmware update for your E810-XXVDA2 NIC in the forums. We are glad to hear that the NVM update (4.20) resolved the issue and established the link at 25G without any additional configuration needed. It's great to know that the update made various SFP28 transceivers work seamlessly.

Apologies for any inconveniences, please rest assured that we will take your feedback into consideration to improve our support process in the future. If you have any further questions or need additional assistance, feel free to reach out to us.

Regards,

Irwan_Intel