AN881 reference design need to be upgraded for device 1SM21CH.

[CHung]

Hi,

The following reference design is based on 1SM21BH device.

https://fpgacloud.intel.com/devstore/platform/19.1.0/Pro/an881-pcie-avmm-dma-gen3x16-ddr4-and-hbm2/

We use Stratix 10 MX 1SM21CH device, so we cannot synthesize this reference design.

I guess the difference is HBM controller interface because of the size.

Could you please provide a version for 1SM21CH device?

This reference is important to us, or we don't know how to access HBM2/DDR4 by PCIe driver.

​Thanks.

[CHung]

Yes, we did.

We 'll try your sof and let you know the result. Thanks!

[BoonT_Intel]

Hi I attached the SOF, please give it a try.

[CHung]

Hi,

Programmer said the device is incompatible.

Could you please generate it again using 1SM21CHU1F53E1VG? This is full name of device.

Thank you.

[BoonT_Intel]

Sorry, previously I use 1SM21CHU1F53E1VGS1 and able to re-generate the IP and recompile.

However, when i change the device to 1SM21CHU1F53E1VG, I am unable to generate the HBM IP with errors as below, I will work with internal team on this errors.

Can you briefly explain how you upgrade the design and able to generate the IP without error?

At the same time, maybe you can use other non-hbm design and upgrade to 1SM21CHU1F53E1VG and retry. The goal now is want to get a non working design that able to let linux detect the device.

Error(20327): 2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_1_araddr declared with width 28 which is declared with width 29 in file hbmc_0.ip

2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_0_awaddr declared with width 28 which is declared with width 29 in file hbmc_0.ip

2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_1_awaddr declared with width 28 which is declared with width 29 in file hbmc_0.ip

2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_0_araddr declared with width 28 which is declared with width 29 in file hbmc_0.ip

2019.11.13.15:18:28 Error: The system cannot be generated when there are errors. 

[CHung]

Hi,

I also met these errors when I used Quartus v19.3.

You could try v19.1, it could upgrade HBM2 smoothly.

[BoonT_Intel]

Other non-HBM design that I mentioned can be as below:

https://fpgacloud.intel.com/devstore/platform/18.1.0/Pro/pci-express-gen3-x16-avmm-dma-with-on-chip-memory-reference-design/

[CHung]

Thanks.

I'll try it.

[BoonT_Intel]

Hi I need some time to setup the server at my side.

For your side, can you give a quick try by go to bios, change the slot PCIe gen to Gen2 instead.

Also, give a try on other non-hbm design like I suggest previously.

[CHung]

Hi,

We noticed a strange phenomenon when changing BIOS setting.

1. When AN881 SOF is programmed to FPGA
   1. If we set PCIe slot as AUTO detect, the computer boots successfully.
   2. If we set PCIe slot as gen3x16, gen2x16, or gen1x16, the computer cannot boot, and the beep code is 8-short which means "display memory read/write test failure".
2. When nothing is programmed to FPGA
   1. computer boot successfully no matter which BIOS setting is applied.

​So we guess, BIOS did detect something when BIOS setting is not AUTO and FPGA has SOF.

However, we cannot enter Linux normally under this case.

[BoonT_Intel]

Maybe there are issue from the DDR4 or HBM that caused the rd/wr fail, as mentioned before, can you give a try by using the pcie to on-chip memory design and give it a try?

[CHung]

Hi, yes, we tried the PCIe exampled without HBM2.

The result was the same.

[CHung]

Hi,

We changed Quartus version from v19.1 to v19.3, and now Linux could recognize the board!

All ​PCIe reference designs includes AN881 could be detected by "lspci" command.

However, there is still something wrong.

We use AN881 driver, it could read configuration space, but couldn't access HBM2/DDR4.

​Besides, the AN881 document doesn't describe how to set DMA clearly.

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/an/an881.pdf​

What's the flow to use above gui?​

Thanks.

[CHung]

Hi

Only "4. read configuration space" works, we could read vendor ID.

"0. Link Test​" we got 100 R/W/mismatch errors.

"5. Change BAR" we got "Could not change BAR"​

Do you have any suggestion?

[BoonT_Intel]

Hi Sir,

Can we get the signal tap waveform for now?

From the screenshot, I can see you are unable to change to DDR4. Can you type 2 and try to change it to HBM2?

[CHung]

Hi,

Yes. The signalTap shows lane_act = 0x10, and ltssmstate  = 0x11.

If you need other signals, please let me know.

We have tried -1, 2, and 4, all of the options showed "Could not change BAR!".

[CHung]

Hi,

As below figure, it seems BAR setting has something wrong.

The path is our FPGA PCIe device​:

/sys/bus/pci/devices/0000:01:00.0

And then we see resources are zero:

[BoonT_Intel]

Hi,

When you download the original reference design, the device is 1SM21CHU1F53E1VG or 1SM21CHU1F53E2VG?

The dev kit is using 1SM21CHU1F53E1VG.

I just got the MX BHU board today. When I attempt to program the board I just aware that the design is using 1SM21CHU1F53E2VG (even the AN881 documentation also mentioned E2 instead of E1.

Are you download the design and change it from E2 to E1 then regenerate the design?

[CHung]

Hi,

The original reference design we downloaded from below website is 1SM21BHU2F53E2VG.

https://fpgacloud.intel.com/devstore/platform/19.1.0/Pro/an881-pcie-avmm-dma-gen3x16-ddr4-and-hbm2/

In all previous experiments, we changed the device to 1SM21CHU1F53E1VG.

[CHung]

Add more information:

1.​

We find that in the original reference design, BAR2 and BAR4 configuration of PCIe are disable, it's different from document.

But driver still cannot change BAR after we enable them.

​2.

Although driver can read PCIe configuration space,

but the value seems not the same with AN881 document.

Only Vendor ID is expected 0x1172.

​

[BoonT_Intel]

Hi, I am able to setup the design and I also facing the same problem as yours. I will try to contact the design author and clarify.

One thing to check with you is when i run the driver, I am unable to see HBM2=2 and DDR4=4 n preset 5. I only can see change BAR -1=None. For this I am using the unzip software folder under the reference design. Are you using other software driver?

at the same time, can give a check with command as below to see if the BAR2 and BAR4 si detect in the system.

Lspci -nn | grep 1172 to check the BDF.

Eg, if theBDF is 5:0.0 then run command as blow:

Lspci -s 05:00.0 -vvv

[CHung]

Hi,

I also use the driver from reference design, but mine is called software_hbm2_ddr4 as the attached file.

Are you using the first one of following reference designs?

https://fpgacloud.intel.com/devstore/platform/?acds_version=any&family=stratix-10&board=104

Mine is AN881: PCIe AVMM DMA Gen3x16 DDR4 and HBM2 .

Our message is: