PTP not working for 10GBASE-KR interface with E823-L Ethernet Controller and ice 1.9.11 driver

[kelly_8613]

I am trying to run PTP tests with HW timestamping but receiving the errors below.  The test is running on the 10GBASE-KR control plane interface from the embedded NAC Quad 1 on the Xeon D 1700 Intel Ice Lake SoC processor.

Information on the adapter and driver:

# lspci -v | grep "Ethernet" -A 1
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
DeviceName: Onboard LAN
--
f4:00.0 Ethernet controller: Intel Corporation Ethernet Connection E823-L for backplane
DeviceName: Onboard LAN
--
f4:00.1 Ethernet controller: Intel Corporation Ethernet Connection E823-L for backplane
DeviceName: Onboard LAN

# ethtool -i eno3
driver: cw_ice
version: 1.9.11
firmware-version: 2.28 0x8000fd37 1.3200.0
expansion-rom-version:
bus-info: 0000:f4:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -i eno3
driver: ice
version: 4.18.0-372.9.1.el8.x86_64
firmware-version: 2.28 0x8000fd37 1.3200.0
expansion-rom-version:
bus-info: 0000:f4:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

* We have modified the ice 1.9.11 driver and renamed it to cw_ice.  I also tried testing with the native ice driver from Almalinux 8.6 and saw the same results.

Failing PTP log:

# ptp4l -f /etc/ptp4l.conf -i eno3 -m
ptp4l[220.937]: selected /dev/ptp0 as PTP clock
ptp4l[220.942]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[220.942]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[220.942]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[227.492]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[227.492]: selected local clock 000000.fffe.000101 as best master
ptp4l[227.492]: port 1: assuming the grand master role
ptp4l[238.502]: timed out while polling for tx timestamp
ptp4l[238.502]: increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
ptp4l[238.502]: port 1: send sync failed
ptp4l[238.502]: port 1: MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[254.502]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[261.299]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[261.299]: port 1: assuming the grand master role
ptp4l[272.309]: timed out while polling for tx timestamp
ptp4l[272.309]: increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
ptp4l[272.309]: port 1: send sync failed
ptp4l[272.309]: port 1: MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[288.309]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[295.591]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[295.591]: port 1: assuming the grand master role
ptp4l[306.601]: timed out while polling for tx timestamp
ptp4l[306.601]: increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
ptp4l[306.601]: port 1: send sync failed
ptp4l[306.601]: port 1: MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[322.602]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[330.247]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[330.247]: port 1: assuming the grand master role

I have additional test logs and tshark logs if needed. Thanks.

[Diego_INTEL]

Hello @kelly_8613,

Thank you for contacting Intel Embedded Community.

Have you tried transmitting with less speed?

Also, you can check this document for know issues in section 3.1.2 Intel® Ethernet 820 Series, at page 21, looking at the E823-L.

https://cdrdv2-public.intel.com/730842/730842_Release_27_4_EPG_v1.0.pdf

Best regards,

@Diego_INTEL 

[kelly_8613]

Hi Diego,

Sorry for the delayed response. I've tried PTP testing with the interface running at 1G instead of 10G, is that what you meant?  I believe I saw similar failures. 

As for the known issues link that you sent, I didn't find any useful information that would give insight to why PTP is failing.  Do you know if this interface/driver/or firmware supports PTPv2?

Thank you,

Kelly

[Diego_INTEL]

Hello @kelly_8613,

Don't worry and yes, that helps to know that the issue is not only tied to 10 G speed.

I have been investigating and maybe an update in the firmware may help.

https://www.intel.com/content/www/us/en/content-details/768093/intel-ethernet-controller-products-release-28-0-release-notes.html?wapkw=e823%20firmware&DocID=768093

There is a newer NVM version: 3.09.

I have sent an email to the address registered in your account.

Best regards,

@Diego_INTEL

[Eric35]

Hello,

I am having similar connection issues with my e823-l, how can I download the latest 3.09 NVM?

I can only find the 4.2 NVM for the e810, which cleared up all of my issues with that card.

Thanks,

Eric

[Diego_INTEL]

Hello @Eric35,

Thank you for contacting Intel Embedded Community.

In order to access the latest NVM for the E823-L, your account must be upgraded to Premier, this will give you access to the document.

https://www.intel.com/content/www/us/en/secure/content-details/769201/intel-ice-lake-d-icx-d-lcc-pc-lan-enabling-kit-lek.html?DocID=769201

Best regards,

@Diego_INTEL 

[Eric35]

How does one obtain a premier account? 

[Diego_INTEL]

Hello @Eric35,

You may check this link:

https://www.intel.com/content/www/us/en/support/articles/000058073/programs/resource-and-documentation-center.html

Best regards,

@Diego_INTEL 

[kelly_8613]

Hi Diego,

Thank you for your response. I received your e-mail regarding the new firmware and will try that out shortly.  I will get back to you with my results.

Thank you,

Kelly

[Diego_INTEL]

Hello @kelly_8613,

Good, I hope it works. 

Best regards,

@Diego_INTEL

[kelly_8613]

Hi Diego,

My apologies for the late update but I have verified that the new firmware works and I'm able to PTP successfully.  Thank you for the help!!

- Kelly 

[Diego_INTEL]

Hello @kelly_8613,

Don't worry, that's great to know. You are welcome.

Best regards,

@Diego_INTEL