Low latency Ethernet 10R Reference Design. How to modify the behaviour of the testbench?

PVanL · ‎03-18-2020

I understand the "avalon_if_param_pkg" is used to configure the behaviour of the Avalon Interfaces. Eth_register_map_params_pkg to configure the inside of the packet. I want to change the gaps between the 32 ethernet packets (it is now 8 to 12 bytes). Is there a way to modify this gap, and if so, how? I also want to change the length of the payload of the packets. How?

Deshi_Intel · ‎03-20-2020

HI,

Can you be more specific on which example design that you are referring to here ?

Do you have the link to the example design ?
Or if this is example design generated from certain Intel FPGA IP, then pls share with us which IP and using what setting ?

Thanks.

Regards,

dlim

PVanL · ‎03-20-2020

Hi Dlim,

Thanks for responding to my question.

The example is "Scalable Low Latency Ethernet 10G MAC Intel FPGAIP",

see https://fpgacloud.intel.com/devstore/platform/14.0.0/Standard/scalable-ll-ethernet-10g-mac-with-10g-baser-phy-design-example/?wapkw=LL%20Ethernet%2010G%20MAC,

and see amongst others "ug-20162_LL_MAC_cyclone10GX" manual.

Or the UG-01144, see https://www.intel.com/content/www/us/en/programmable/documentation/bhc1395127830032.html

I am not sure if i changed any settings, a least not intentionally, except for some (irrelevant) timing in the testbench (to make it faster), What i am sure about is that i replaced the INTEL PHY with my own PHY, with some specific features. I got that working.

The configuration I use is 10 Gbps, 1 channel, xgmii 64 + 8 bit.

Now i want to change the testbench (tb_top.sv in the example), such that that there are not exclusively packets of 75 bytes (including preamble, CRC, closing byte "FD")

The packets always end with last2 valid bytes (when the packet starts in hte second half of the 64 bit dataword, or with last6 valid bytes (when the packet starts on the first half of the 64 bit dataword.

I also want to check my PHY on last1, last2...last8 valid byte at the end of packet.

The packets are very tight to each other: 8 byte or 12 idle byte, see examples below (where "07" is an indication of "idle")

time(nsec) data                               cntr
3789 istin A3208DD2E6FB626E 00000000 
3796 istin 0707070707FD2B8F 11111100 
3802 istin 555555FB07070707 00011111 
3809 istin 00000000D5555555 00000000 
3815 istin EEAACC88CCEE0000 00000000

(example 8 byte from MAC), or

3853 istin CCBCF5E2A39F4168
3860 istin 07FD2B8FA320C1C1
3866 istin 0707070707070707
3873 istin 555555FB07070707
3879 istin 00000000D5555555
3885 istin EEAACC88CCEE0000

(example 12 byte from MAC)

I want to be able to change this gap of 8 or 12 byte to any multiple of N * 4 bytes. (exclusive of the 0, 1, 2, 3 last idle byte at the end of a packet)

Now i am building a fifo between eth_generator and the MAC to create those gaps, but that is not a very elegant way ).

I believe the clue is somewhere in:

*  "avalon_if_parameter_pkg.sv"  : for packet length
* "default_test_params_pkg.sv" : i thought for a moment change "INSERT PAD" from 1 to 5, but this does not result in a visible change anywhere
* "eth_mac_frame"  : there is a function "virtual function void insert_payload(ref bit [7:0] frame_data[], input bit [7:0] payload[])" (i am more VHDL then Verilog, what functionality is this virtual void?)
* "eth_register_map_params_pkg.sv" seems to be address map

I attached a part of the transcript, from vlog command onwards.

line 3 - 100 : compiling

line 101 - 314 : elaboration

line 315 - 369 : warnings

line 377 - 443 : starting of testbench

line 446 - 626 : warnings during runtime (not so relevant, those appear in my PHY, not in INTEL's), mainly searching for blocklock and the processing of 'x' s

line 626 -677 : receipt of packets

line 678 - 683 : processing compare

line 688 - 756 : reporting compare.

Looking forward to your advice.

Regards, Pieter

Deshi_Intel · ‎03-23-2020

Hi Pieter,

I looked at the LL 10G example design and I can see that the Ethernet traffic generator is controlled by a BFM (avalon_driver.sv) which is then further controlled by Altera Avalon BFM (csr, st Tx and st Rx)

alt_em10g32_0_EXAMPLE_DESIGN\LL10G_10GBASER\simulation\ed_sim\models\avalon_driver.sv
/ip/altera/sopc_builder_ip/verification/altera_avalon_mm_master_bfm/altera_avalon_mm_master_bfm.sv
/ip/altera/sopc_builder_ip/verification/altera_avalon_st_sink_bfm/altera_avalon_st_sink_bfm.sv
/ip/altera/sopc_builder_ip/verification/altera_avalon_st_source_bfm/altera_avalon_st_source_bfm.sv

To be honest, it's complicated and not easy to decode these BFM. I am sorry. I can point you some direction but I won't be able to do much help for you.

Perhaps you can also consider to re-write your own test bench as well

Else If you look at the testbench top level file again (tb_top.v), look like from line 640 onwards in function (if (id == 0),

Looks like user is able to tune some setting provided you spend time to understand the BFM address mapping.
Feel free to explore it

Thanks.

Regards,

dlim

PVanL · ‎03-23-2020

Hi Deshil, Thanks for looking at this, it is indeed not easy, I am afraid I have to study Verilog manuals, some parts of the code are hard to read (in the US you probably feel the same about vhdl). There are four workarounds: 1/ study the BFM models as you suggested, 2/ rewrite the testbench (that is a hard one, as the compare function is essential that I do not yet fully understand), 3/ use the atlanta protocol, (used in Avalon driver, with the start/end of packet indicator, I assume I can raise the ‘ready’ indicator to ‘0’ to create an artificial idle period), and 4/ finish the fifo in which I create also idles. I am almost done with 4/, but 3/ seems the more simple approach that I though of halfway building nbr 4/. Regards, Pieter

Deshi_Intel · ‎03-23-2020

Hi,

I am sorry. Indeed a painful process for you.

There is one thing that I would like to clarify on your option 3 - regarding tx_ready signal.

May I know are you referring to "avalon_st_tx_ready" as mentioned in user guide doc (page 101, table 40) ?
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_32b_10g_ethernet_mac.pdf
the tx_ready is actually a output status signal from MAC to user indicating condition of MAC. If MAC is busy then it will back pressure user data traffic by de-asserting tx_ready else if tx_ready is high, that's mean MAC is ok to accept user data.
I believed if you want to stall user traffic, then the input control signal should be "avalon_st_tx_valid"

Thanks.

Regards,

dlim

PVanL · ‎04-07-2020

Hi Deshil, Thanks, that is indeed the parameter I am using. However, there is something odd. I first built in altera_eth_10g_mac_base_r a nice fifo buffer between Avalon_st_adapter and the MAC (in the altera_eth_10g_mac_r_wrap ). SC_FIFO => AVALON_ST_ADAPTER => MAC was modified into: SC_FIFO => AVALON_ST_ADAPTER => FIFO => MAC Then I ran the testbench: passed! Below you will see the signal from end of packet 1, packet 2, and start of packet 3 Val = valid, sop = start of packet, eop = end of packet, mty = empty, dav = the ready signal. You see that the signal is nicely hold during dav = 0, and continues again after dav is made 1 again (by the MAC). So far, so good. val 1 sop 0 eop 0 mty 0 dav 1 E6FB626E 6E62FBE6 val 1 sop 0 eop 1 mty 2 dav 1 00008DD2 D28D0000 val 1 sop 1 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 01003000 00300001 val 1 sop 0 eop 0 mty 0 dav 1 D138F5BA BAF538D1 val 1 sop 0 eop 0 mty 0 dav 1 D8D2FF84 84FFD2D8 val 1 sop 0 eop 0 mty 0 dav 1 8A497C15 157C498A val 1 sop 0 eop 0 mty 0 dav 1 2716DDDA DADD1627 val 1 sop 0 eop 0 mty 0 dav 1 F3B7CD8C 8CCDB7F3 val 1 sop 0 eop 0 mty 0 dav 0 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 0 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 0 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 0 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 0 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 1 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 1 1F34C835 35C8341F val 1 sop 0 eop 0 mty 0 dav 1 DF6B7265 65726BDF val 1 sop 0 eop 0 mty 0 dav 1 523E4CBC BC4C3E52 val 1 sop 0 eop 0 mty 0 dav 1 A39F4168 68419FA3 val 1 sop 0 eop 0 mty 0 dav 1 CCBCF5E2 E2F5BCCC val 1 sop 0 eop 1 mty 2 dav 1 0000C1C1 C1C10000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 1 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 1 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 02003000 00300002 Then I built, without using this extra buffer, based on your suggestion, a simple statemachine, generating idles by making ‘avalon_st_tx_ready = ‘0’ back to the Avalon_st_adapter to indicate ‘hold your signal’. So, without using the extra fifo. After some iterations it worked. Simulation passed. SC_FIFO => AVALON_ST_ADAPTER => IDLE_GENERATOR => MAC (the idle generator has no fifo, it just holds tx_ready towards the Avalon st adapter, when a idle should be made, and forwards tx_ready from the MAC when not) val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 01003000 00300001 val 1 sop 0 eop 0 mty 0 dav 1 D138F5BA BAF538D1 val 1 sop 0 eop 0 mty 0 dav 1 D8D2FF84 84FFD2D8 val 1 sop 0 eop 0 mty 0 dav 1 8A497C15 157C498A val 1 sop 0 eop 0 mty 0 dav 1 2716DDDA DADD1627 val 1 sop 0 eop 0 mty 0 dav 1 F3B7CD8C 8CCDB7F3 val 1 sop 0 eop 0 mty 0 dav 0 3010BFC8 C8BF1030 val 1 sop 0 eop 0 mty 0 dav 0 1F34C835 35C8341F val 1 sop 0 eop 0 mty 0 dav 1 DF6B7265 65726BDF val 1 sop 0 eop 0 mty 0 dav 1 DF6B7265 65726BDF val 1 sop 0 eop 0 mty 0 dav 1 DF6B7265 65726BDF val 1 sop 0 eop 0 mty 0 dav 1 523E4CBC BC4C3E52 val 1 sop 0 eop 0 mty 0 dav 1 A39F4168 68419FA3 val 1 sop 0 eop 0 mty 0 dav 1 CCBCF5E2 E2F5BCCC val 1 sop 0 eop 1 mty 2 dav 1 0000C1C1 C1C10000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 1 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 02003000 00300002 val 1 sop 0 eop 0 mty 0 dav 1 3856DA8B 8BDA5638 val 1 sop 0 eop 0 mty 0 dav 1 4E1D101E 1E101D4E val 1 sop 0 eop 0 mty 0 dav 1 9544372D 2D374495 val 1 sop 0 eop 0 mty 0 dav 1 0A9CE36B 6BE39C0A val 1 sop 0 eop 0 mty 0 dav 1 D21E53C7 C7531ED2 val 1 sop 0 eop 0 mty 0 dav 0 5256C93A 3AC95652 val 1 sop 0 eop 0 mty 0 dav 0 E14279C8 C87942E1 val 1 sop 0 eop 0 mty 0 dav 1 409E1A9D 9D1A9E40 val 1 sop 0 eop 0 mty 0 dav 1 409E1A9D 9D1A9E40 val 1 sop 0 eop 0 mty 0 dav 1 409E1A9D 9D1A9E40 val 1 sop 0 eop 0 mty 0 dav 1 644ED592 92D54E64 val 1 sop 0 eop 0 mty 0 dav 1 864EF32D 2DF34E86 val 1 sop 0 eop 0 mty 0 dav 1 CAE46D7F 7F6DE4CA val 1 sop 0 eop 1 mty 2 dav 1 0000107C 7C100000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 0 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 1 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 00000000 00000000 val 1 sop 0 eop 0 mty 0 dav 1 03003000 00300003 val 1 sop 0 eop 0 mty 0 dav 1 AE9F6EC5 C56E9FAE However, if you look closer, you will see that the relation between ‘dav’ (= tx_ready as generated by the MAC) is set to ‘0’, while the inputs keep coming, and after ‘dav’ turns to ‘1’, it repeats the signal two more clocks. So, the signal is delayed with 2 clocks compared to the ‘dav’ (which makes perfectly sense if you look at the logic of the ‘idle generator’: the ‘dav’ signal from the MAC needs 1 clock to propagate back to the adapter, and the signal from the adapter needs 1 clock to propagate to the MAC. Together 2 clocks (so, the idle generator makes an erroneous signal). So far so good. But how then can the test pass?????? Let us look at the signal coming from the TX side of the MAC into the PHY, did it indeed take out correctly the signals that are invalid? ( that is: 2 of the 3 DF6B7265’s?). Of course not, it took out 3010BFC8 and 1F34C835, as it should, as indicated by the ‘dav’ signal. Yet, the test passed! So, the evaluation gives a false positive! (a sort of corona test that gives a false negative, although the person tested is positive). Can you explain / elucidate? Do I have some misunderstanding somewhere? Or does someone @ intel has to do some homework? Regards, Pieter

Deshi_Intel · ‎04-20-2020

Hi Pieter,

Sorry for getting back late to you.

It seems like you are experimenting with different design logic to interact with 10G MAC and you also build your own Ethernet traffic generator/checker ? and you are seeing weird behaviour on your own Ethernet traffic generator/checker or are you trying to feedback there is some issue in Ethernet example design ?

Did you enable ECC option in 10G MAC so that you can check for potential error bit with ECC error correction signal ?
There are some timing diagram on both Avalon ST Tx and Rx interface data transfer that you can cross check as expected reference vs your design data transfer
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_32b_10g_ethernet_mac.pdf

Thanks for the detail explanation but sorry, I don't think I understand and following your issue here.

We don't restrict how customer build their user logic to interact with MAC.
If adding extra buffer/FIFO helps to smooth out the data traffic transfer pipeline then it's not a bad choice either.
Also always ensure design is able to meet timing closure in Quartus Time Quest Analysis. Sometime weird result behaviour is due to data transfer failure caused by design timing issue.

Thanks.

Regards,

dlim

Deshi_Intel · ‎06-01-2020

Hi Pieter,

I have not hear back from you for sometime.

Hopefully everything goes well at your side.

I am now setting this case to closure.

Thanks.

Regards,

dlim

PVanL · ‎06-02-2020

Hi Deshill, Thanks for asking. I am still working on this, and find every now and them some irrularities, that I normally can solve. Can you help me with the next step: I want to couple the low latency INTEL MAC in the 10GBASE-R Example to the INTEL PCIExpress. The only PCI Express example that I found, is one that communicates to a memory. Do you also have an example that has the PCI Express coupled to the Low Latency MAC? Thanks in advance, Regards, Pieter van Leeuwen

Deshi_Intel · ‎06-03-2020

Hi Pieter van Leeuwen,

Intel agent support structure goes by IP area and on a case by case support basic.

For instance, I am supporting Ethernet IP while some other Intel agent is supporting PCIe enquiry.

Unfortunately I am not familiar with PCIe and won't be able to help you out.

My suggestion to you is

We can close this existing case
while you can file new forum thread mention about the enquiry support on PCIe then this new case will be routed to Intel PCIe support agent that can provide better assist to you.

Thanks.

Regards,

dlim

Deshi_Intel · ‎06-10-2020

Hi Pieter van Leeuwen,

I hope you had file new forum thread for your PCIe enquiry as per my earlier suggestion.

For now, I am setting this case back to closure.

Thanks.

Regards,

dlim

PVanL · ‎06-16-2020

Hi Deshil,

Sorry for not answering earlier. I solved now all my hardware issues with my version of PHY connected to the INTEL MAC. Thanks for that.

I found several UG's and AN documents describing the PCIexpress IP and Examples. At a certain moment in time, i managed to generate a .par file, from which QUARTUS lite 18.0 generated PCIE- IP. However, i cannot find a way to generate the <pcie_a10_hip_0_example_design> as described in ux-dex-a10-pcie-avst.pdf, found at https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug-dex-a10-pcie-avst.pdf.

Can your PCI Express collegue support me in generating this design example. Second question to him: there is a choice between the avalon-st protocol implementation, and a avalon-mm implementation. As the Low Latency MAC interfaces with an Avalon-ST interface, I thought this would be the more easy choice. Is that right, or are there reasons to use the mm version? Regards, Pieter

Deshi_Intel · ‎06-17-2020

Hi Pieter,

Np and good to know Ethernet issue is resolved at your end.

Yes, Intel PCIe support agent can support you.

My only request to you if pls file new forum thread and post your PCIe questions there so that your new forum case can be assigned to Intel PCIe support agent to help you up. :)

This is how Intel support structure works. Thanks for your understanding.

Regards,

dlim

Deshi_Intel · ‎06-18-2020

Hi Pieter,

I can see that you already filed new forum thread on PCIe enquiry

Hence, I will be setting this case back to closure.

Thanks.

Regards,

dlim