Stratix 10 MX development kit top and bottom HBM2 AXI-4 switch example design

cancel
Showing results for 
Search instead for 
Did you mean: 
363 Discussions

Stratix 10 MX development kit top and bottom HBM2 AXI-4 switch example design

Stratix 10 MX development kit HBM2 top and bottom example design with AXI-4 switch interface and system console accessible efficiency counters

Description

This article is dedicated to users that are looking for a Stratix 10 MX HBM2 Example design with AXI-4 switch user side interfaces on both top and bottom Stratix 10 MX development kit HBM2 memory.  The design in this article leverages an top HBM2 AXI-4 switch example design that was created using Quartus Pro 21.1 using the High Bandwidth Memory (HBM2) Interface Intel® FPGA IP.  The design was adjusted to add the bottom HBM2 with AXI4-switch interface.  In addition, the design has system console accessible efficiency counters to monitor efficiency in the physical hardware.  Using system console makes it easier to script reading the counter values and pasting them into a spreadsheet for analysis.

This design can also be used to preserve the HBM2 within the Stratix 10 MX family devices.  Similar to transceiver qsf preserve settings that are used to preserve unused transceivers for future use, the HBM2 also needs to be preserved if it is currently unused with plans for future use.  There are no qsf settings to preserve the HBM2.  Since the Stratix 10 MX should be immediately configured to preserve transceiver links, the same image should also contain HBM2 IP for both top and bottom HBM2 stacks to preserve them.  Not preserving the HBM2 could result in premature degradation of the HBM2 memory.

Documentation links

Refer to the following user guides to better understand the HBM2 IP:

Stratix 10 MX FPGAs  

Stratix 10 MX Support and Documentation 

High Bandwidth Memory (HBM2) Interface Intel® FPGA IP User Guide 

Stratix 10 MX development kit 

HBM2_S10MX_devkit.PNG

The Stratix 10 MX development kit used for this example contains a 4GB HMB2 on top and 4G HBM2 on the bottom.  The example in this article will use both top and bottom HBM2.

Example

The design created in this article is here:

qts_hbm_top_bottom_21_1_0_169.qar - See attached at the bottom of this article.

This example will run on the Stratix 10 MX development kit top and bottom HBM2.  

 

Here is what the IP settings look like for the top HBM2.

mik_Intel_0-1637620736651.png
mik_Intel_3-1637622316456.png
mik_Intel_1-1637620772699.png

Controller configurations 1 through 7 match that of controller 0.

- Each ed_synth_tg*_*.v file was edited to change the TEST_DURATION parameter from SHORT to INFINITE.
- Infinite patterns are used to saturate the total count on the efficiency counters and allow a user to test on on going pattern.
- The location from the project directory for the top HBM2 is ./ip/ed_synth/ed_synth_tg0_0/synth/
- There are 16 traffic generators, tg_0_0 through tg_7_1. The traffic generators are used for both top and bottom HBM2.
mik_Intel_2-1637621174712.png

 

- The ed_synth.v file was edited to add in the efficiency counters and counter status registers
- The location of the file is ./ed_synth/synth/
- The ed_synth_bottom.v file was also edited to add in the efficiency counters and counter status registers
- The location of the file is ./ed_synth_bottom/synth/

Keep in mind that when using the HBM2 IP wizard to create an example design with the AXI-4 switch, the addressing will be such that very poor efficiency will be realized because the higher two bits will be toggling between all 4 pseudo channels.

In the HBM2 IP User Guide:

High Bandwidth Memory (HBM2) Interface Intel® FPGA IP User Guide 

mik_Intel_0-1637686968804.png

To prove per channel efficiency in this article example design, the MSB two bits of the address was pegged to each of the 4 traffic generators within the 4 HBM2 pseudo channels per AXI-4 switch.  ed_synth.v and ed_synth_bottom.v were updated to only send address bits 27:0 to the HBM2 AXI-4 Switch interface. 

mik_Intel_5-1637687372151.png

 

**** If the example design is re-generated or if Quartus Pro is changed to "Always regenerate design files for IP cores" in Quartus Settings -> IP Settings, the ed_synth.v, ed_synth_bottom.v, and ed_synth_tg*_*.v will be overwritten and will need to be edited again with the changes outlined above.

Within ed_synth.v and ed_synth_bottom.v there is a Platform designer module called counters that is used to connect the output of the efficiency counters to registers that can be read from System Console.  You can open counters.qsys within Quartus to see what is in the Platform Designer System. It will contain clock inputs, reset inputs, PIO (Parallel I/O) input registers, and a JTAG to Avalon Master that allows a user to access the PIO registers via System Console.  There are a number of PIO registers in the counters.qsys design that hold the efficiency counter numbers for each pseudo channel (16 for top HBM2 and 16 for bottom HBM2).  So the counter.qsys design is used in the ed_synth.v to allow a user to read the efficiency counters for the top HBM2 and it is also used on the ed_synth_bottom.v to allow a user to read the efficiency counters for the bottom HBM2.

Running the example

To run the example on the Stratix 10 MX development kit, open Quartus Pro 21.1 and then open Tools -> Programmer.   Turn on the development kit and make sure the micro USB on the development kit is connected to the computer being used to download the sof file.   No clock changes are required from the default clock settings on the board after power up.  Do an Auto Detect and then Add File to add the qts_hbm_top.sof file.  The chain should look like this:

mik_Intel_1-1637623341425.png

Hit Start and program the Stratix 10 MX development kit.

Now, the HBM2 top and bottom instances need to be reset.

In Quartus, go to Tools -> In-System Sources and Probes Editor.  You may see a pop-up that says "No instances found in the current project or on the device".  Click OK, then in the Hardware pull down, select the Stratix 10 MX development kit.

mik_Intel_0-1637623203037.png

It may take up to two minutes for all the signals and probes to show up.  Once all the instances show up, scroll down to the last index which is index 70, instance ID HBM.

mik_Intel_0-1637623800564.png

Click on the Continuously Read Probe Data" button at the top.

mik_Intel_1-1637623894989.png

Expand the 6 sources so each bit is shown for the 70 HBM instance at the bottom.  It should look like this:

mik_Intel_2-1637623992987.png

In the top level file qts_hbm_top.v, the mapping of the 6 sources is as follows:

mik_Intel_3-1637624127874.png

Source bit 3 is hbm_only_reset and source bit 5 is bhbm_only_reset which are used to reset the top and bottom HBM2 respectively.  Toggle bit 3 and 5 to 1 and then back to 0.

mik_Intel_4-1637624294955.png

mik_Intel_2-1637623992987.png

Top and bottom traffic generator/checkers and HBM2 should now be running.

In Quartus, go to Tools -> Signal Tap Logic Analyzer to open Signal Tap.  You will see 16 instances that are used to probe each traffic generator for the top HBM2 to help verify traffic is running.  

The HBM2 IP can also create efficiency counters for simulation, but this method does not separate read and write transactions and is only available for AXI interface mode, thus efficiency counters have been added manually to this example.

Click on each instance and click on the continuous run button to see the AXI transactions occurring real time and to see the AXI efficiency counters for the chosen test generator. The below picture shows a snapshot from tg_0_0.

mik_Intel_5-1637624472274.png

The efficiency counters are 32 bits wide and count the number of times the readdatavalid and writedatavalid signals are asserted.  Rready, wready, arvalid, awvalid, arrready, and awready are also counted to understand when the read and writes are being held off by the HBM2 controller.  The total count is used to know the total number of clock periods and stops once the 32 bit counter is saturated.  All AXI efficiency counters in Signal Tap are setup to be in unsigned decimal mode.  

In a previous example for HBM2 listed here:

https://community.intel.com/t5/FPGA-Wiki/Stratix-10-MX-HBM2-Example-design-with-Avalon-MM-user-side/ta-p/1213143

a user was required to manually look at each counter in Signal Tap and then write the numbers into a spreadsheet.

The example design in this article has enhanced functionality that allows a user to read out the efficiency counters via a System Console script.

In Quartus, go to Tools -> System Debugging Tools -> System Console to open System Console.  Then, load the design:

mik_Intel_0-1637685406236.png

Once the design has loaded, go to the Tcl Console and cd to the System_Console directory and source the HBM2_Counter_Reads.tcl script which can be found at the bottom of this article as HBM2_Counter_Reads.txt.  Make sure you create the System_Console directory under the project directory for the script location and for the script to output its results.

mik_Intel_1-1637685607349.png

The script should output a file called HBM2_Top_Bottom_counters.txt csv delimited file that looks as follows:

mik_Intel_2-1637685637969.png

....

through tg7_1

mik_Intel_3-1637685698395.png

....

through tg7_1

The output file above can be pasted into Excel and column separated by the commas.  Efficiency numbers can then be calculated with formulas as shown below in column D.

mik_Intel_4-1637686095748.png

 

Efficiency numbers will be highly dependent on the addressing scheme used to write and read data to/from the HBM2.

Stratix 10 Datasheet 

HBM2_datasheet_HBM2_4G_Performance.PNG

 

Attachments
Version history
Last update:
‎11-24-2021 07:26 AM
Updated by:
Contributors