Avalon-MM to PCIe address translation table settings

[BobSD]

Trying to DMA from DDR3 memory through PCIe Txs port back to external CPU memory using a mSGDMA engine in a Cyclone V GT device. The mSGDMA is working and I can see the Txs Avalon-MM interface transacting correctly in SignalTap. The data and addresses are correct and the DMA completes normally. But the DMA data never shows up in external CPU memory (at least not where it's supposed to).

We walked through how to set up the address translation registers very carefully and thought we nailed it, but something is wrong and everything points to a PCIe addressing problem. Below is how we set the registers. Hoping that someone can check our work and indicate what, if anything, we screwed up.

The translation settings in the PCIe hard IP core are 2 pages with 16MB per page. There are 25 address bits on the Txs Avalon-MM port.

The 32-bit base address of the Txs port in Platform Designer is 0x8000_0000.

The destination start address in external CPU memory is 0x3DD39B10.

Per the "Cyclone V Avalon-MM Interface for PCIe Solutions User Guide", bits [23:0] of the Txs address pass through to the PCIe address and bit [24] selects between the two pages of the translation table.

We're setting the translation table registers as follows:

0x1000 = 0x3C00_0000

    - Bits [31:25] = top 7 bits of target address 0x3DD39B10 = binary 0011110.

    - Bit [24] = bit [24] of Txs Avalon-MM address which is 0 in this first page.

    - Bits [23:2] = 0 (these address bits pass through to PCIe and are not translated)

    - Bits [1:0] = 2'b00 for 32-bit PCIe address

0x1004 = 0 (ignored for 32-bit addressing)

0x1008 = 0x3D00_0000

    - Bits [31:25] = top 7 bits of target address 0x3DD39B10 = binary 0011110.

    - Bit [24] = bit [24] of Txs Avalon-MM address which is 1 in this second page.

    - Bits [23:2] = 0 (these address bits pass through to PCIe and are not translated)

    - Bits [1:0] = 2'b00 for 32-bit PCIe address

0x100C = 0 (ignored for 32-bit addressing)

It's possible that there's a software problem in the external CPU, but I want to make 100% sure that the address translation table is correct before I try to blame the software guy.

Thanks in advance for any help provided.

[Wincent_Altera]

Hi,

Thanks for contacting Intel. I'm assigned to support request.

I'll investigate and get back to you soon. Thanks for your patience.

Meanwhile, we have a design for PCIe-MSGDma cyclone V which have a simular implementation to you

https://www.rocketboards.org/foswiki/Projects/A10AVCVPCIeRootPortWithMSILTS

Perhaps you can refer to the qsys connection as a reference and try to implement this in your design.

Best regards,

Wincent_Intel

[BobSD]

@Wincent_Altera -

Thank you for responding. Just to clarify, our PCIe endpoint in Platform Designer works fine. The processor card (Windows based) in the system can see and talk to the Cyclone V card via PCIe and can write and read back registers, etc. It is able to trigger the mSGDMA to DMA from DDR3 to the PCIe Txs port (which I can see in SignalTap), but the data never makes it into the processor card memory (at least not where software expects it to be). There are three possible problems here as I see it:

1. The address translation registers in the Cyclone V PCIe endpoint are not being set up correctly, so the data is not going where it's supposed to.
2. The wrong address is being written as the destination (write) address in the mSGDMA descriptor. There is some confusion/doubt by the software team about virtual vs physical addresses. They're currently working on resolving that.
3. There is something wrong with the PCIe bus or backplane in our system.

My goal with the post here is to address only #1 above. If the translation table setup is wrong then we'll fix it. If it's correct then we'll move on to #2 and #3. I looked at the design you linked above to see if I could find how the address translation table in the Cyclone V endpoint is initialized in that system, but I was not able to find that information.

I look forward to hearing from you.

Bob in SD

[Wincent_Altera]

Hi BobSD,

What is the number of Msgdma that you connected with the PCIe HIP ?
Single or multiple ? is it more than 4 ?

For the address translation table of tx slave you can refer to 
https://cdrdv2-public.intel.com/655090/ug-01110-1_5.pdf under page 95/288

there are some clear instructions on how you can configure it.

Let me know if further clarification is needed.

Regards,

Wincent_Intel

[BobSD]

Just one mSGDMA in the design. As I said in my original post, "We walked through how to set up the address translation registers very carefully ...". All I'm asking you to do is check our work as described in the original post above and tell me if we did it correctly or not.

Thanks,

Bob

[Wincent_Altera · Accepted solution]

Hi BobSD,

From what you provided compared with the user guide. It is seen to be correct.
But I cannot 100 % confirm that as it is your custom design.

As you are using a single MSGDMA.
What alternative you can do to double confirm is to refer to the design example provided in the previous post.
Set the parameter accordingly and monitor if you are able to get the desired output/result.

Let me know if there is any other things I can help you.

Regards,

Wincent_Intel

[BobSD]

That’s all I needed, @Wincent_Altera. Thank you for the help!

Bob

[Wincent_Altera]

Hi

Glad that that help you.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If your support experience falls below a 9 out of 10, I kindly request the opportunity to rectify it before concluding our interaction. If the issue cannot be resolved, please inform me of the cause so that I can learn from it and strive to enhance the quality of future service experiences. 

Regards,

Wincent_Intel

p/s: If any answer from the community or Intel Support is helpful, please feel free to give the best answer or rate 9/10 survey.

[BobSD]

Final update:

We were never able to get address translation to work so I changed the PCIe Avalon-MM address setting from 32 bits to 64 bits.

From this:

To this:

This change eliminated the Avalon to PCIe address translation and the DMAs into external CPU memory across the PCIe bus began to work immediately.