Low Latency Ethernet 10G MAC Debug Checklist

[#1]

Low Latency Ethernet 10G MAC Debug Checklist

December 18, 2014

Last Major Update 

Objective

The objective of the wiki page is to provide a useful debug guidelines and checklist which help to debug and identify issue related to Altera Low Latency Ethernet 10G MAC Megacore and resolve it effectively.

Introduction of Low latency Ethernet 10G MAC

■ Supports MAC operating speed of 10M/100M/1G/10Gbps 

■ 32-bits Avalon Streaming at 312.5MHz with conversion logic to support 64-bit 

■ Supports PHY interfaces of XGMII (32/64-bits) at 312.5/156.25MHz, GMII (8-bits) at 125MHz and MII (4-bits) at 125MHz with clock enable for 2.5/25MHz 

■ Optional IEEE 1588v2 features 

■ Optional statistic collections for transmit and receive data paths 

■ Optional ECC correction and detection 

■ Compliant to IEEE 802.3 – 2008 

■ Unidirectional feature (14.0 onwards)

Debug Checklist

Things to do before start debugging:

1. Understand the problem statement/requirement. 

2. ACDS Version/IP Version/Target Device/LL MAC IP variants (Speed variant, adapters enable/disable, etc) 

3. Check Fogbugz/KDB/Errata.

4. Crosscheck with the IEEE 802.3 Specification and Avalon Specification 

5. Configuration Register Settings

6. Timing Closure – hardware issue

7. Debug Information

-Hardware vs. Simulation

-Failing symptom: Intermittent or consistent

-Transmitter issue vs Receiver issue

8. Simplify design or test case if necessary

9. Ensure all signals connected to the MAC IP appropriately

10. Ensure all clocks are operating at their specified frequencies

11. Ensure all resets working correctly.

Clock Frequency for LL 10G Ethernet System

Clock_freq_requirement_LL_MAC.png (Click here for image)

Recommended Reset Handling

■ Ensure all tx_rst_n, rx_rst_n and csr_rst_n have been released properly.

■ Make sure the de-assertion of the resets happen after all the tx_clk, rx_clk and csr_clk are stable. 

■ Tie tx_rst_n , rx_rst_n and csr_rst_n together.

■ From v14.0 and onwards, after asserting tx_rst_n/rx_rst_n signal, wait at least 150 ns to completely reset all the logic inside the TX/RX datapath of the MAC IP core.

Essential signals for signal tap

Top level signals:

■ ../alt_em10g32:*_mac_inst|

Clock: tx/rx_312_5_clk, tx/rx_156_25_clk, tx/rx_rst_n, tx/rx_312_5_clk, csr_clk

Reset: tx/rx_rst_n, csr_rst_n

csr : csr_*

Datapath: avalon_st_tx/rx_* , xgmii_tx/rx_* , gmii_tx/rx_*, mii_tx/rx_*

status : link_fault_status_xgmii_rx_data, speed_sel

Internal Adapters signals (with Adapters enabled)

■ ../alt_em1:0g32:*_mac_inst|altera_eth_avalon_st_adapter:st_adpt.avalon_st_adpt_inst| altera_dc_fifo:tx_156_to_312|

in_*, out_*

■ ../alt_em1:0g32:*_mac_inst|altera_eth_avalon_st_adapter:st_adpt.avalon_st_adpt_inst| altera_dc_fifo:rx_312_to_156|

in_*, out_*

■ ../alt_em1:0g32:*_mac_inst|altera_eth_avalon_mm_adapter:csr_adpt.avalon_mm_adapter|

sl_*, ms_*

■ ../alt_em1:0g32:*_mac_inst|alt_em10g_32_64_xgmii_conversion:xgmii_adpt.xgmii_conv_inst|

xgmii_rx, xgmii_rx_data_out, xgmii_rx_control_out, xgmii_tx,xgmii_tx_control_in,xgmii_tx_data_in

Case Studies

Case Study 1

• Problem Statement:

- Intermittent packet corruptions seen at the output of Low Latency 10G Ethernet MAC connected to the User Logic

Try1.png  (Click here for image)

• Failure symptoms:

- Incoming packets from PHY to LL Ethernet 10G MAC has no CRC errors

- Failure is observed during 1Gbps speed mode

- Packet corruptions happen in multiple channels of the LL MAC

- Corruptions happen at random locations within a packet

- Failure is observed in different seeds of Quartus compilation

- No timing violations are reported in TimeQuest

- Unable to replicate issue with Design Example

• Debug Steps:

1. Identify the instantiated LL Ethernet 10G MAC variant

2. Capture Signal Tap on below signals to narrow down the failing location

-GMII RX and TX signals

-Avalon ST TX and RX input/output signals to Avalon ST TX/RX 64 bit adapter

3. Compare data input and output of both locations and identify for bit flip or error bits.

-Compare input signals from GMII interface and output signals from Avalon ST interface

Case_study_1_debug.png ‎(Click here for image)

4. Identify the failing paths

-In this case the failing paths is on data path from GMII RX to Avalon ST TX (Avalon ST loopback)

5. Check all SDC files to spot for false path constraints on failing paths

-Report out all false paths in TimeQuest to verify if any of the false paths are related to GMII RX to Avalon ST RX/TX

6. At this point, if unsure, issue should to be escalated to Factory Apps for further timing verification.

• Root Cause:

Customer has set false path for data paths crossing 2 different clock domains (125MHz to 312.5MHz). However, these constraints are invalid because some paths using these clocks are required to be synchronous to each other.

Case Study 2

• Problem Statement:

- Error packets seen at the GMII TX of LL Ethernet 10G MAC intermittently

Case_study_2_latest.png (Click here for image)

• Failure symptoms:

-Failure is observed during 1Gbps speed mode

-No error signal asserted from the Avalon ST TX on LL MAC

-Failure happen in multiple channels of the LL MAC

-Failure is observed in different seeds of Quartus compilation

-No timing violations are reported in TimeQuest

-Unable to replicate issue with Design Example

• Debug steps:

1.Identify the instantiated LL Ethernet 10G MAC variant.

2.Capture Signal Tap on below signals to narrow down the failing location.

- GMII TX signals

- Avalon ST TX input signals to the Avalon ST TX 64 bit adapter

3.Compare data input and output of both locations to identify if data starts to differ before the input of the MAC or after the input of the MAC.

4. Analyze all the control signals of the GMII TX interface and Avalon ST TX interface.

-Input of LL MAC at Avalon ST TX interface: No errors observed in the packet

Case_study_2_debug1.png (Click here for image)

5. At this point, further probing to internal signals are required. Below are the signals captured at the output of DC FIFO within LL Ethernet 10G MAC IP

-Altera_avalon_dc_fifo:tx_156_to_312|out_ready

-Altera_avalon_dc_fifo:tx_156_to_312|out_startofpacket

-Altera_avalon_dc_fifo:tx_156_to_312|out_endofpacket

-Altera_avalon_dc_fifo:tx_156_to_312|out_error

-Altera_avalon_dc_fifo:tx_156_to_312|out_valid

-Altera_avalon_dc_fifo:tx_156_to_312|out_data

-Altera_avalon_dc_fifo:tx_156_to_312|out_empty

Case_study_2_debug2.png  (Click here for image)

• Root Cause:

-False path was set in between 156MHz and 312.5MHz clock domains for read/write pointers in DC FIFO in the embedded altera_avalon_dc_fifo.sdc. However, there is a timing requirement for DC FIFO pointers to meet for LL Ethernet 10G MAC IP.

-With the false path being set, 1 clock cycle for DC FIFO write point value gets updated late 1 clock cycle causing MAC to underflow and generate gmii tx err.

Disclaimer

© [2013] Altera Corporation. The material in this wiki page or document is provided AS-IS and is not supported by Altera Corporation. Use the material in this document at your own risk; it might be, for example, objectionable, misleading or inaccurate.