HBM2 interfacing with the PCI express hard IP STRATIX 10 MX

[rsing108]

Hi,

I am trying to interface HBM2 with pci express hard ip+ on stratix 10 MX. I am referencing to this example design https://fpgacloud.intel.com/devstore/platform/18.1.0/Pro/pci-express-gen3-x16-avmm-dma-with-hbm2-memory-reference-design/?wapkw=hbm2%20interface%20pci%20express , but in this I am not able to understand that how the BAM master address width is 30 bits whereas when I am generating the same IP inside my qsys system for the same configuration I am getting 16 bits as the address width. The BAR size in both of the cases is 16 bits.

Is there any way I can manually edit this? 

[rsing108]

One difference I see is that the reference design's device is 1sm21bhu2f53e2vgs1 whereas my device is 1sm21chu2f53e2vg. But I do not think this should create any problem.

[BoonT_Intel]

Hi Sir,

Yes, the BHU and CHU is the difference, for BHU device, the HBM2 is 8G and CHU device, CHU is 16G. So you see the address bits is 2x diff.

1sm21chu2f53e2vg

[rsing108]

Thanks @BoonT_Intel for the reply. The difference in size is for the BAM_master address of PCIE IP and not the HBM ip. Also the CHU one has 30 bit address in the reference design and chu has 16 bit address for the same PCIE hard ip+ configuration.

[rsing108]

please note the typo in my last comment: "Also the BHU one has 30 bit address in the reference design and CHU has 16 bit address for the same PCIE hard ip+ configuration."

[BoonT_Intel · Accepted solution]

Hi, Sorry for the mistake.

I tried the same and I only can see the 16bits on the address. These address bits are affected by the setting of "Address width of bursting master" of the PCIe IP. I believe the developer hack the IP and change the address width to 30 instead.

If you need, you can change this by modifying the g3x16_hbm2_avmm_bridge_512_0.ip file.

Search for Address width of Bursting Master and you can change the value to 30.

[rsing108]

Thanks @BoonT_Intel , it indeed appears that this a some kind of a hack by the developer. Thanks! One a side note, can you refer me to maybe an example design/reference design where I can understand the working/usage of multiple BARS?

[BoonT_Intel]

Hi Sir,

I thought the AN881 is already a multiple BAR design?

It can access DDR4 and HBM2.

BAR 2 for HBM2 and BAR4 for DDR4.

By the way, you can check the design store and see if any design meet your needs - https://fpgacloud.intel.com/devstore/

Thanks.

[rsing108]

Hi @BoonT_Intel ,

A881 in the documentation uses multiple bars but the link to reference design is broken. The reference design I mentioned earlier uses only 1 bar i.e. for HBM.

Thanks

[BoonT_Intel]

Hi, Are you getting 403 access denied? I tried the same and get this error. I think the link should work, let me check with the internal team. From your side, maybe can see if there any link that you can click to "contact intel" and report to IT from external. Thanks.

[rsing108]

Hi @BoonT_Intel , I am also getting the same 403 error. If you could get me the link that would be great. Meanwhile I will also try to file an intel IT request.

Thanks.

[BoonT_Intel]

Great and thanks. If anyone of us gets the response/link fixed, we can come back here to give an update.

[rsing108]

Hi @BoonT_Intel , did you get any response? I did not receive any.

[BoonT_Intel]

Hi Sir,

They told me the design file is corrupted in the server. Now they're trying to approach the design owner to get back the same file.

I will get back to you immediately after I get the file. Very sorry for the inconvenience caused.

[BoonT_Intel]

Hi Sir,

Are you able to make any progress there?

Perhaps, I can get the design by end of this month. Thanks.

[rsing108]

Hi @BoonT_Intel ,

I am currently stuck. It would be great if you could get me the design. Also, if possible can you get this design for 1sm21chu rather than 1sm21bhu? if not then 1sm21bhu is also fine.

Thanks for your help.

[BoonT_Intel]

Hi Sir,

I get the design zip file. However, the design is not yet validated. They will only publish the design after re-validation.

However, I believe it will be helpful for you for reference purpose. Thus, please note that the design is provided AS-IS basis only.

The file is too big, thus I can’t upload it here. I share the file with you using the other channel.

Here is the step to download the file:-

Download the filezilla and install - https://filezilla-project.org/download.php?type=client

After install, open filezilla and login using -

Host: secureftp.intel.com

Username: tmp01798

Password: Hello12!

Port:21

You will see and file name AN881.zip at the bottom right corner.

right click the file and download.

[rsing108]

Hi @BoonT_Intel ,

Thanks for the reference design. I changed the device to 1sm21chu2f53e2vg which is the device I have along with the HDL changes required and PIN assignments. I have not changed anything else. But I am getting this warning now.

Warning(332049): Ignored create_clock at pcie_to_hbm2_ddr4_top.sdc(2): Incorrect assignment for clock. Source node: REFCLK_GXBL1C_CHTP already has a clock(s) assigned to it. Use the -add option to assign multiple clocks to this node. Clock was not created or updated.
Info(332050): create_clock -name {REFCLK_GXBL1C_CHTP} -period "100Mhz" [get_ports {REFCLK_GXBL1C_CHTP}]

REFCLK_GXBL1C_CHTP is the iopll reference clock for HBMC. Can you tell me what can be the reason for this as according to me it is just defined in the project sdc file once.

[rsing108]

Hi @BoonT_Intel, please I have been working on the reference design that you provided. There are a few bugs that I found in one of the files but the design is still not working. I am attaching the updated file here. I suspect there are other problems with it too. 

Can you please tell me when will the final design be posted after validation? Hoping that all the bugs will be removed by then.

Also, can I do something to accelerate this process?

Thanks.

[BoonT_Intel]

Hi Sir,

Due to the pandemic, we having the challenge to access the hardware to test the design.

I will get back to you once it tested.

By the way, are you also seeing the problem if you just compile the design without making any changes?

[rsing108]

Hi @BoonT_Intel , 

Yes, I am seeing the problem when I compile without making any changes. I also ran simulations where one can see that many important/necessary axi4 signals to the HBM2, as I mentioned in my previous comment are not even being driven because of the problems with the source code.