XXV 710 HW timestamp issue

dmikand · ‎07-09-2021

I've met with ptp4l timesync issue. Master send SYNC packets without timestamp and client receive these packets without RX timestamp when twoStepFlag set to 1. Such behavior doesn't seem to conform to the standard. Switching to 1-step ptp is not possible because NIC driver returns "out of range" ioctl result when ptp4l tries to set RX filters up.

Do you know are there misconfigurations, hardware or software issues?

Note:

Two NICs are connected directly.

Master cmd line:
./ptp4l -i enp2s0f0 -m2EH -f ./default.cfg

Slave cmd line:

./ptp4l -i enp2s0f0 -m2sEH -f ./default.cfg

Master default.cfg:

[global]
#
# Default Data Set
#
twoStepFlag 1
clientOnly 0
socket_priority 0
priority1 128
priority2 128
domainNumber 0
#utc_offset 37
clockClass 248
clockAccuracy 0xFE
offsetScaledLogVariance 0xFFFF
free_running 0
freq_est_interval 1
dscp_event 0
dscp_general 0
dataset_comparison ieee1588
G.8275.defaultDS.localPriority 128
maxStepsRemoved 255
#
# Port Data Set
#
logAnnounceInterval 1
logSyncInterval 0
operLogSyncInterval 0
logMinDelayReqInterval 0
logMinPdelayReqInterval 0
operLogPdelayReqInterval 0
announceReceiptTimeout 3
syncReceiptTimeout 0
delayAsymmetry 0
fault_reset_interval 4
neighborPropDelayThresh 20000000
serverOnly 0
G.8275.portDS.localPriority 128
asCapable auto
BMCA ptp
inhibit_announce 0
inhibit_delay_req 0
ignore_source_id 0
#
# Run time options
#
assume_two_step 0
logging_level 6
path_trace_enabled 0
follow_up_info 0
hybrid_e2e 0
inhibit_multicast_service 0
net_sync_monitor 0
tc_spanning_tree 0
tx_timestamp_timeout 1

unicast_listen 0
unicast_master_table 0
unicast_req_duration 3600
use_syslog 1
verbose 0
summary_interval 0
kernel_leap 1
check_fup_sync 0
clock_class_threshold 248
#
# Servo Options
#
pi_proportional_const 0.0
pi_integral_const 0.0
pi_proportional_scale 0.0
pi_proportional_exponent -0.3
pi_proportional_norm_max 0.7
pi_integral_scale 0.0
pi_integral_exponent 0.4
pi_integral_norm_max 0.3
step_threshold 0.0
first_step_threshold 0.00002
max_frequency 900000000
clock_servo pi
sanity_freq_limit 200000000
ntpshm_segment 0
msg_interval_request 0
servo_num_offset_values 10
servo_offset_threshold 0
write_phase_mode 0
#
# Transport options
#
transportSpecific 0x0
ptp_dst_mac 01:1B:19:00:00:00
p2p_dst_mac 01:80:C2:00:00:0E
udp_ttl 1
udp6_scope 0x0E
uds_address /var/run/ptp4l
uds_ro_address /var/run/ptp4lro
#
# Default interface options
#
clock_type OC
network_transport UDPv4
delay_mechanism E2E
time_stamping hardware
tsproc_mode filter
delay_filter moving_median
delay_filter_length 10
egressLatency 0
ingressLatency 0
boundary_clock_jbod 0
#
# Clock description

#
productDescription ;;
revisionData ;;
manufacturerIdentity 00:00:00
userDescription ;
timeSource 0xA0

Slave default.cfg:

[global]
#
# Default Data Set
#
twoStepFlag 1
clientOnly 0
socket_priority 0
priority1 128
priority2 128
domainNumber 0
#utc_offset 37
clockClass 248
clockAccuracy 0xFE
offsetScaledLogVariance 0xFFFF
free_running 0
freq_est_interval 1
dscp_event 0
dscp_general 0
dataset_comparison ieee1588
G.8275.defaultDS.localPriority 128
maxStepsRemoved 255
#
# Port Data Set
#
logAnnounceInterval 1
logSyncInterval 0
operLogSyncInterval 0
logMinDelayReqInterval 0
logMinPdelayReqInterval 0
operLogPdelayReqInterval 0
announceReceiptTimeout 3
syncReceiptTimeout 0
delayAsymmetry 0
fault_reset_interval 4
neighborPropDelayThresh 20000000
serverOnly 0
G.8275.portDS.localPriority 128
asCapable auto
BMCA ptp
inhibit_announce 0
inhibit_delay_req 0
ignore_source_id 0
#
# Run time options
#
assume_two_step 0
logging_level 6
path_trace_enabled 0
follow_up_info 0
hybrid_e2e 0
inhibit_multicast_service 0
net_sync_monitor 0
tc_spanning_tree 0
tx_timestamp_timeout 1

unicast_listen 0
unicast_master_table 0
unicast_req_duration 3600
use_syslog 1
verbose 0
summary_interval 0
kernel_leap 1
check_fup_sync 0
clock_class_threshold 248
#
# Servo Options
#
pi_proportional_const 0.0
pi_integral_const 0.0
pi_proportional_scale 0.0
pi_proportional_exponent -0.3
pi_proportional_norm_max 0.7
pi_integral_scale 0.0
pi_integral_exponent 0.4
pi_integral_norm_max 0.3
step_threshold 0.0
first_step_threshold 0.00002
max_frequency 900000000
clock_servo pi
sanity_freq_limit 200000000
ntpshm_segment 0
msg_interval_request 0
servo_num_offset_values 10
servo_offset_threshold 0
write_phase_mode 0
#
# Transport options
#
transportSpecific 0x0
ptp_dst_mac 01:1B:19:00:00:00
p2p_dst_mac 01:80:C2:00:00:0E
udp_ttl 1
udp6_scope 0x0E
uds_address /var/run/ptp4l
uds_ro_address /var/run/ptp4lro
#
# Default interface options
#
clock_type OC
network_transport UDPv4
delay_mechanism E2E
time_stamping hardware
tsproc_mode filter
delay_filter moving_median
delay_filter_length 10
egressLatency 0
ingressLatency 0
boundary_clock_jbod 0
#

# Clock description
#
productDescription ;;
revisionData ;;
manufacturerIdentity 00:00:00
userDescription ;
timeSource 0xA0

NIC configuration:

Master:

# ip link show dev enp2s0f0
6: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:a6:33:30 brd ff:ff:ff:ff:ff:ff

Slave:

# ip link show dev enp2s0f0
4: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:a6:33:c0 brd ff:ff:ff:ff:ff:ff

02:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
Subsystem: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at f3000000 (64-bit, prefetchable) [size=16M]
Memory at f5000000 (64-bit, prefetchable) [size=32K]
Expansion ROM at f7c80000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 30-33-a6-ff-ff-91-96-b4
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Capabilities: [1d0] #19
Kernel driver in use: i40e
Kernel modules: i40e

Software versions:
# ethtool -i enp2s0f0
    driver: i40e
    version: 2.8.20-k
    firmware-version: 8.30 0x8000a483 1.2926.0
    expansion-rom-version:
    bus-info: 0000:02:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes

Also tried with 4.15.0-142-generic kernel version and this NIC driver version:

    driver: i40e
    version: 2.15.9
    firmware-version: 8.30 0x8000a483 1.2926.0
    expansion-rom-version:
    bus-info: 0000:02:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes

supports-priv-flags: yes

# ethtool -T enp2s0f0

Time stamping parameters for enp2s0f0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 2
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
none (HWTSTAMP_FILTER_NONE)
ptpv1-l4-sync (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
ptpv1-l4-delay-req (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
ptpv2-l4-event (HWTSTAMP_FILTER_PTP_V2_L4_EVENT)
ptpv2-l4-sync (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
ptpv2-l4-delay-req (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
ptpv2-l2-event (HWTSTAMP_FILTER_PTP_V2_L2_EVENT)
ptpv2-l2-sync (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
ptpv2-l2-delay-req (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
ptpv2-event (HWTSTAMP_FILTER_PTP_V2_EVENT)
ptpv2-sync (HWTSTAMP_FILTER_PTP_V2_SYNC)
ptpv2-delay-req (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)

linuxptp commit id: f3290eee8567f957b8c61c2d39ee618bc7ccffb1

linux kernel version: 5.4.0-77-generic
Also tried with 4.15.0-142-generic

linux kernel PTP config:

    # grep "CONFIG_PTP" /boot/config-4.15.0-142-generic
    CONFIG_PTP_1588_CLOCK=m
    CONFIG_PTP_1588_CLOCK_KVM=m

loaded kernel modules:
    # lsmod | grep ptp
    ptp       20480 2 i40e,igb
    pps_core 20480 1 ptp

linux distrib version:
    # cat /etc/lsb-release
    DISTRIB_ID=Ubuntu
    DISTRIB_RELEASE=18.04
    DISTRIB_CODENAME=bionic
    DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

cheyert · ‎07-28-2021

Have you managed to resolve this?

We're getting a similar issue where our DELAY_REQ messages are not timestamped.

The odd thing is that when we reverse the role of PTP master and slave, that the Rx HW timestamping seems to work for SYNC messages and PDELAY_REQ messages.

Just the DELAY_REQ messages are failing with the messaes "Delay Req received without timestamp" once ptp4l is running.

Configuration is E2E, HW timestamping, L2 transport) and the RX filter is set to

RX timestamp filter mode 12 (HWTSTAMP_FILTER_PTP_V2_EVENT).

and it seems to be the only possible way to configure it.

kharo22 · ‎12-20-2023

Hello

I met the same issue within E810.

but on my side

ethtool -T  enp8sf0
Time stamping parameters for enp8sf0:
Capabilities:
	hardware-transmit
	software-transmit
	hardware-receive
	software-receive
	software-system-clock
	hardware-raw-clock
PTP Hardware Clock: 1
Hardware Transmit Timestamp Modes:
	off
	on
Hardware Receive Filter Modes:
	none
	all

and also

Private flags for enp8sf0:
link-down-on-close           : off
fw-lldp-agent                : off
channel-inline-fd-mark       : off
channel-pkt-inspect-optimize : on
channel-pkt-clean-bp-stop    : off
channel-pkt-clean-bp-stop-cfg: off
vf-true-promisc-support      : off
mdd-auto-reset-vf            : off
vf-vlan-pruning              : off
legacy-rx                    : off
cgu_fast_lock                : off
dpll_monitor                 : on
extts_filter                 : on
itu_g8262_filter_used        : off
allow-no-fec-modules-in-auto : off

kernel is currently 6.2, I got same issue within 5.11 and kernel 6.5, last ice drivers 1.12.18

XXV 710 HW timestamp issue

700 Series Network Adapters (up to 40GbE)