Arria 10 GX: 64 bit atomic access to Avalon Bus through PCIe?

[MehmetFide]

Hi,

We have an external processor Arm Cortex A53 with Linux 5.4 aarch64 connected to an Altera Arria 10 GX FPGA through a PCIe bus.

On the FPGA side, we have an Avalon bus, all addresses are set as 64bit. All implementation is done using Quartus 22.4 and QSys.

And I have no problem to access to the avalon bus though PCIe (DMA works, MSI interrupt works) on the Linux side as long as I use 8/16 or 32 bit accesses. Whenever I use 64 bit access for example using ioread64(), it returns 0xffffffffffffffff and I observe that PCIe Config page on the FPGA side gets corrupted. After that I have to restart FPGA to make it functional again.

My original question is here:

https://stackoverflow.com/questions/75097169/any-known-issue-with-ioread64-iowrite64-on-a-pcie-bus

Any idea that if Avalon bus and PCIe hard core in the Arria 10 GX together supports 64 bit io read and write operations?

Thank you.

[Wincent_Altera]

Hi,

Can I know which design example that you are using ?
AVST ? AVMM ? MCDMA ?

Regards,

Wincent_Intel

[MehmetFide]

Hello,

Our design is based on Arria 10 - AN690: PCIe Gen3x8 AVMM DMA ( here )

Regards,

Fide.

[Wincent_Altera]

Hi,

I am not familiar with the platform and code you are using.

Is that feature inside Quartus ? or it is from the driver ?

Why you need to change the IOread ?

Regards,

Wincent_Intel

[Wincent_Altera]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

[MehmetFide]

Hi @Wincent_Altera,

All BAR registers definition and implementations are inside the Quartus.

When we try to access the memory area in FPGA from Linux, there are couple of options like Altera doing it here 

Those are

ioread8 -> Works fine

ioread16 -> Works fine

ioread32 -> Works fine

ioread64 -> Doesn't work and causes FPGA PCIe to be crashed

Avalon DMA -> Works fine

I'm trying to understand what is special with 64 bit direct access to make it non functional.

Thank you.

[Wincent_Altera]

Hi,

If you just want to use PCIe IP that interact with memory directly, then pls consider checking out below example design that use (PCIe) only, no DMA

• https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug-dex-a10-pcie-avmm.pdf
• This is the example design that you can generate from PCIe IP directly
• This example design currently connect to on chip RAM but you can replaced with DDR4 if you want

Regards,

Wei Chuan

[Wincent_Altera]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

[MehmetFide]

Hi @Wincent_Altera,

As you suggested, we have tested it with both internal RAM and external DDR ram without DMA.

The problem exists at both conditions. To be more specific, accessing bars as 8/16/32 bit data work fine. But 64 bit accesses (especially read attempt) corrupts the PCIe config page in the FPGA and it fully stops functioning.  It only recovers when we reconfigure the FPGA.

My question is that anyone on Intel ever tested the PCIe hard core in the Arria 10 GX to do 64 bit data access?

Thank you.

[Wincent_Altera]

Hi,

I check previous record, we had tested and it able to run.
I lay down some suggestion so that we can narrow down this further.

The issue might be caused by data packing limitation in interconnect on AXI/Avalon boundary. If your AXI is 32bit wide and PCIe Avalon interface is 64bit wide. In such situation multiple two 32bit transactions will be used to implement 64bit transfer.

If you want to further analyze, Singal TAP traces showing transactions on PCIe Avalon interface and HPS AXI should show what is going on.

Please try

1.  can you please try disabling the "Enable Reordering" option (under the Controller tab)?
2. At the same time, please try running the Calibrate Termination command in the EMIF Toolkit (Tools menu -> System Debugging Tools -> External Memory Interface Toolkit)  to verify the optimal ODT settings.

Regards,

Wincent_Intel

[Wincent_Altera]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

Regards,

Wincent_Intel

[Wincent_Altera]

Hi

We have not hear from you and this Case is idling. It is not recommended to idle for too long.

Therefore following our support policy, I have to put this case in close status. My apologies if any inconvenience cause

Hence, This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If you feel your support experience was less than a 9 or 10,

please allow me to correct it before closing or let me know the cause so that I may improve your future support experience.

Regards,

Wincent_Intel