1G/10Gb Ethernet FPGA IP uses XGMII instead of GMII in 1Gb mode

[DmitryKo]

Hello, all.

I attempt to launch the Ethernet 1Gb link through the 1G/10Gbe and 10GBASE-KR PHY Intel Arria 10 FPGA IP.
1Gb patch-cord connects SFP+ port from board with Arria10  with SFP+ port on Mikrotik router.
On Mikrotik SPF+ port  Auto-Negotiation is disabled and only speed 1Gb is enabled.

Configuration of 1G/10Gbe and 10GBASE-KR PHY IP is:
IP variant: 1G/10Gb Ethernet
Initial datapath: 1G
tab 10M/100M/1Gb Ethernet -> Enable 1Gb Ethernet protocol = enabled
tab Speed detection -> Enable automatic speed detection = disabled

1G/10Gbe and 10GBASE-KR PHY IP asserts rx_syncstatus and led_link, because accepts BCh 50h pairs.
Router also established link (see attachment).
I see outgoing traffic on the router port.

1. I see ingoing packets (LLDP) on RX on 1G/10Gbe and 10GBASE-KR PHY IP.
I see them on XGMII_RX_DATA[7:0] although I would have expected them on gmii_rx_data since the speed 1Gb is set.
Why used XGMII interface instead of GMII?

2. 1G/10Gbe and 10GBASE-KR PHY IP provides 8bit GMII interface in accordance with documentation.
But I see some changes in XGMII_RX_DATA bits from 8 up to 15.
I know XGMII_RX_DATA[71:0] interface includes data and control for 8 lanes. Each lane consists of 8 bits of data and 1 bit of control.
There are not in the Arria 10 Transceiver PHY documentation the information about this mode of using XGMII interface (when data expected by 8bit GMII interface really transferred by 72bit XGMII interface). Could someone tell me where such usage is described?

3. I performed the following test. Connect the PHY.XGMII_RX_DC to PHY.XGMII_TX_DC (loopback).
But I didn't see the incoming traffic on router port. Why?

Code was started from Intel example:

    eth10g_kr_phy phy (

        .tx_serial_clk_10g        (tx_serial_clk_10g), // 5.156 GHz

        .tx_serial_clk_1g         (tx_serial_clk_1g), // 625 MHz

        .rx_cdr_ref_clk_10g       (clk_322),  

        .rx_cdr_ref_clk_1g        (clk_125),   

  .tx_pma_div_clkout        (tx_div_to_xgmii_clk),

    .rx_pma_div_clkout        (rx_div_to_xgmii_clk),

        .xgmii_tx_clk             (clk_125), //(clk_156_25), (tx_div_to_xgmii_clk),                                                                          

        .xgmii_rx_clk             (clk_125), //(clk_156_25), (tx_div_to_xgmii_clk),                                                                           

  .mode_1g_10gbar (1'b1), // 1 = 1G, 0 = 10G  if automatic speed detection is disabled

        .tx_clkout            (phy_tx_recovered_clk),                                                             

        .rx_pma_clkout            (phy_rx_recovered_clk),      

        .tx_analogreset           (xcvr_reset_control_tx_analogreset[0]),                                             

        .tx_digitalreset          (xcvr_reset_control_tx_digitalreset[0]),                                            

        .rx_analogreset           (xcvr_reset_control_rx_analogreset[0]),                                             

        .rx_digitalreset          (xcvr_reset_control_rx_digitalreset[0]),                                            

//        .usr_seq_reset            (phy_usr_seq_reset),                                                                                                                          

        .mgmt_clk                 (csr_clk),                                                                           

        .mgmt_clk_reset           (reset),                                                                         

        .mgmt_address             (phy_mgmt_address[12:2]),    

        .mgmt_read                (phy_mgmt_read),                                                                    

        .mgmt_readdata            (phy_mgmt_readdata),                                                                

        .mgmt_waitrequest         (phy_mgmt_waitrequest),                                                             

        .mgmt_write               (phy_mgmt_write),                                                                   

        .mgmt_writedata           (phy_mgmt_writedata),                                                               

        // GMII 

        .gmii_tx_d                (generated_tx_d),  //(phy_xgmii_rx_data), //(mac_gmii_tx_d),                                                                    

        .gmii_tx_en               (generated_tx_c), //phy_rx_syncstatus), //(mac_gmii_tx_en),                                                                   

        .gmii_tx_err              (mac_gmii_tx_err),                                                                  

        .gmii_rx_d                (phy_gmii_rx_d),                                                                    

        .gmii_rx_err              (phy_gmii_rx_err),                                                                  

        .gmii_rx_dv               (phy_gmii_rx_dv),                                                                   

        .tx_pcfifo_error_1g       (phy_tx_pcfifo_error_1g),                                                           

        .rx_pcfifo_error_1g       (phy_rx_pcfifo_error_1g),                                                           

        .rx_clkslip               (phy_rx_clkslip),                                                                   

        // XGMII

        .xgmii_tx_dc              ({generated_tx_c, generated_tx_d}), //(phy_xgmii_rx_data), // (mac_xgmii_tx_data), loopback RX->TX                                                               

        .xgmii_rx_dc              (phy_xgmii_rx_data),                                                                

        .rx_is_lockedtodata       (xcvr_reset_control_rx_is_lockedtodata[0]),                                         

        .tx_cal_busy              (xcvr_reset_control_tx_cal_busy[0]),                                                

        .rx_cal_busy              (xcvr_reset_control_rx_cal_busy[0]),                                                

        .rx_data_ready            (phy_rx_data_ready),                                                                

        // PHY Status

        .rx_block_lock            (rx_block_lock),  

        .rx_syncstatus            (phy_rx_syncstatus),   

        .led_an                   (led_an),                                                                       

        .led_char_err             (led_char_err),                                                                 

        .led_disp_err             (led_disp_err),                                                                 

        .led_link                 (led_link),                                                                     

        .rx_hi_ber                (phy_rx_hi_ber),                                                                    

        // Transceiver Serial Interface

        .tx_serial_data           (tx_serial_data),                                                               

        .rx_serial_data           (rx_serial_data),

        .rx_clkout                (),

        .tx_pma_clkout            (),

        .led_panel_link           ()

    );      

    // Core PLL

    eth10g_core_pll core_pll (

        .pll_refclk0        (clk_322),

        .pll_powerdown      (reset),

        .outclk0            (clk_156_25),

        .outclk1            (clk_312_5),

        .pll_locked         (core_pll_locked),

        .pll_cal_busy       ()

    );

    // TX PLL for 10G

    eth10g_atx_pll atx_pll_10g (

        .pll_powerdown   (xcvr_reset_control_pll_powerdown), // from reset controller

        .pll_refclk0     (clk_322),

        .tx_serial_clk   (),

        .pll_locked      (xcvr_reset_control_pll_locked),

        .pll_cal_busy    (txpll_cal_busy), // to eth10g_channel

        .mcgb_rst        (xcvr_reset_control_pll_powerdown), // from reset controller 

        .mcgb_serial_clk (tx_serial_clk_10g) // 5.156 GHz to phy

    );

fpll_625 pll_625 (

     .pll_refclk0 (clk_125),

  .pll_powerdown (reset),

  .pll_locked (fpll_625_locked),

  .pll_cal_busy (),

     .tx_serial_clk (tx_serial_clk_1g) // 625 MHz to phy

);

//=============PHY Reset Controller==========================================

    altera_xcvr_reset_controller xcvr_reset_controller (

        .clock              (csr_clk),                                              

        .reset              (reset_controller_reset_out),                        

        .pll_powerdown      (xcvr_reset_control_pll_powerdown),          

        .tx_analogreset     (xcvr_reset_control_tx_analogreset),        

        .tx_digitalreset    (xcvr_reset_control_tx_digitalreset),      

        .tx_ready           (xcvr_reset_control_tx_ready),                    

        .pll_locked         (xcvr_reset_control_pll_locked),                

        .pll_select         (1'b0),               

        .tx_cal_busy        (xcvr_reset_control_tx_cal_busy[0] & txpll_cal_busy),              

        .rx_analogreset     (xcvr_reset_control_rx_analogreset),         

        .rx_digitalreset    (xcvr_reset_control_rx_digitalreset),      

        .rx_ready           (xcvr_reset_control_rx_ready),                   

        .rx_is_lockedtodata (xcvr_reset_control_rx_is_lockedtodata), 

        .rx_cal_busy        (xcvr_reset_control_rx_cal_busy)               

    );

Thank in advance for help.
Dmitry.

[ZiYing_Intel]

Hi Dmitry,

Q1. Can you check back whether you got connect to the correct port?

Q2. In the documentation, it only shows about the xgmii_rx_data[63:0]/[31:0].

The width is:

• 64 bits for 1G/2.5G/10G configurations.

• 32 bits for 10M/100M/1G/2.5G/5G/10G (USXGMII) configurations.

For further information, you may refer to link below https://www.intel.com/content/www/us/en/docs/programmable/683617/21-1/1g-2-5g-5g-10g-multi-rate-ethernet-phy.html

Best regards,

zying

[DmitryKo]

Hi, zying.

Q1. I checked. I connected to correct port.

Q2. I use 1G/10Gbe and 10GBASE-KR PHY Intel Arria 10 FPGA IP, not 1G/2.5G/5G/10G Multi-rate Ethernet PHY.

In https://www.intel.com/content/www/us/en/docs/programmable/683426/18-1-18-1/ll-ethernet-10g-mac-operating-modes.html 
in chapter 1.3 I read:
Speed Mode: 1G/10G
MAC Interface: 

• 8-bit GMII
• 32-bit XGMII
• 64-bit XGMII (Use legacy Ethernet 10G MAC XGMII interface enabled

Intel® PHY Compliance: 1G/10GbE Ethernet PHY IP

How I can manage which interface will be used? I think the interface is selected by CORE automatically. But I didn't find this information (which interface used on which speed) in documentation.  It was my question 1.

Best regards, 

Dmitry

[ZiYing_Intel]

Hi Dimistry,

Do you only using the Start of Frame control byte only?

XGMII is required to have the Start Of Frame control byte aligned to lane/byte 0 and the SFD aligned to lane 3 with the first byte of the packet on lane 0 but GMII (being a byte interface) is only required to align the Start of Frame control byte to lane 0.

Best regards,

zying

[ZiYing_Intel]

Hi Dimistry,

Is there any update from your side?

Best regards,

zying

[DmitryKo]

Hi, Zying

I use only 8 lowest bits (lane 0) in XGMII tx data bus and 1 next bit (8th) as control bit.
I see the same behavior on xgmii_rx_data: DCh/50h as Idle command is transferred by 8 [7:0] lowest bits, by lane 0 only 

But I see some activity on xgmii_rx_data on next 8 bits [16:9]: 95h/04h. In the same time on xgmii_rx_control 1 is present on 0 lane only. From this I can make assumption than these bytes are control bytes. But their values are not understandable.

About this not expected behavior on xgmii bus was may question.  

I sent FBh/55h 6 times/D5h before starting packet transfer.

Best regards, Dmitry.

[ZiYing_Intel]

Hi Dimistry,

For Q1, received data may be seen on XGMII port, but it should be ignored while in 1GbE mode.
For Q3, you should use GMII interface when you are using 1GbE mode.

Best regards,
zying

[ZiYing_Intel]

Hi Dimistry,

I now transition this thread to community support. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread.

Best regards,

zying

[DmitryKo]

Hi, Zying,

Q1.  My question was: I see ingoing packets (LLDP) on RX on 1G/10Gbe and 10GBASE-KR PHY IP.
I see them on XGMII_RX_DATA[7:0] although I would have expected them on gmii_rx_data since the speed 1Gb is set.

Your answer: Received data may be seen on XGMII port, but it should be ignored while in 1GbE mode

If I will ignore the data on the XGMII port and I will not get any data on GMII port I will be in situation when I can't use 1G/10Gbe and 10GBASE-KR PHY Core IP. I don't understand how I can use your answer, sorry.

Q3. I am ready to use GMII interface, but I don't understand how to follow your recommendation, if the IP CORE itself uses XGMII in not documented mode. Could you provide more concrete recommendation what I need to change in CORE configuration to use GMII interface, please?

Best regards, Dmitry.