How do you configure the pcie root port on the Stratix 10 to allow the HPS to map an I/O BAR?

bralston7 · ‎02-14-2023

I need help configuring the pcie root port on the Stratix 10 to allow the HPS to map an I/O BAR?. I started out with the Rocketboards Pcie example (https://www.rocketboards.org/foswiki/Projects/Stratix10PCIeRootPortWithMSI), but its device tree doesn't map an IO space for the pcie bridge. It only maps MEM spaces. I went back to Qsys Designer and added the txs (non-hptxs) space to the design, hoping that I could carve out IO space from that, but its not working. There are no root port examples out there that show how to do this. The manual is also completely lacking information in this realm, and the BFM actually states it doesn't support IO space. Modern PCI cards (video cards) all require an IO bar, and my root port implemention doesn't seem to support it.

Wincent_Altera · ‎02-16-2023

Hi,

Configuring PCIe root port on Stratix 10 to allow the HPS to map an I/O BAR can be a complex process.

To be honest , I never try such deep before, but I can lay down some of the tips/suggestion for you to move on further.

Modify the device tree: The device tree is used by the kernel to determine the system's hardware configuration. You can modify the device tree to map an IO space for the PCIe bridge. You can use the "ranges" property in the device tree to specify the base address and size of the I/O space. The "reg" property can be used to specify the base address and size of the memory space. Here's an example of what the device tree entry might look like:

pcie0: pcie@ffe201000 {

compatible = "pci-host-ecam-generic";

reg = <0x0 0xffe201000 0x0 0x1000>;

#address-cells = <3>;

#size-cells = <2>;

ranges = <0x81000000 0x0 0x00000000 0x00010000 0x00000000 0x00010000>,

<0x82000000 0x0 0x00000000 0x00010000 0x00000000 0x00010000>;

interrupts = <0 184 4>;

num-lanes = <8>;

max-functions = <8>;

dma-coherent;

status = "okay";

};

Enable the I/O space in the PCIe root port configuration: In the Qsys Designer, you can enable the I/O space in the PCIe root port configuration. Under the "Root Port" tab, set the "I/O Space Enable" option to "Enabled".
Verify that the PCIe bridge supports I/O space: Not all PCIe bridges support I/O space. You can verify that the PCIe bridge in your system supports I/O space by checking the device datasheet or manual. If the PCIe bridge does not support I/O space, you may need to use a different bridge that does support it.
Check the PCIe endpoint configuration: The PCIe endpoint device (e.g. video card) must be configured to use the I/O space instead of memory space. This is usually done using the device's configuration registers. The configuration registers can be accessed using the "setpci" command in Linux.

Hope this is able to help.

Regards,

Wincent_Intel

bralston7 · ‎02-16-2023

Thank you for the response. I've had the device tree side all worked out for a bit now, but its what you describe in 1) is what I am missing. There is no root port tab in Qsys Designer (yes, my type is set to Root Port, not Endpoint), and no checkbox anywhere for "I/O Space Enable". Ive been all over that GUI and I can't find it. There's no way a full implementation of a PCI root port IP doesn't include a way to make an IO space. How could they leave something like that out? There is an I/O space described in the documentation (ug_s10_pcie_avmm.pdf) called the CRA, but it appears to have a specific purpose for the driver, and not for creating an I/O space.

Could you please reach out to somebody who is intimately familiar with the PCIe IP?

Wincent_Altera · ‎02-19-2023

Hi,

Which varient of Stratix 10 board you are testing on ? GX or SOC version ?

Regards,

Wincent_Intel

Wincent_Altera · ‎02-19-2023

Hi,

Just to confirm that the step is correct to enable the IO space.
Can you please check and let me know if you still cannot get it ?

Open the Quartus Prime software and open your project.
Open the "Platform Designer" tool by clicking on "Tools" > "Platform Designer".
In Platform Designer, click on "System" in the left-hand pane, and then click on the "System Info" tab.
Under the "Address Spaces" section, find the "I/O" row and check if the "Enable" column is set to "Yes". If it is already set to "Yes", then I/O Space is already enabled.
If the "Enable" column is set to "No", click on the pencil icon to edit the I/O address space.
In the "Edit I/O Space" dialog box, check the "Enable I/O Space" box, and then click "OK".
Save the Platform Designer design and exit.
In Quartus Prime, recompile the project and program the FPGA with the updated design.

Looking forward to hear back from you. Please let me know if you are unable to get the feature.
Regards,

Wincent_Intel

bralston7 · ‎02-20-2023

Right, I don't seem to be able to follow your steps. Perhaps we have our Platform designer windows layed out totally differently. I made best guesses and have posted a screenshot.

In your 3) above, I do not have anything labeled "System" in the left hand pane. I do have a "System View" in the middle". I also do have a "System Info" on the right pane, and I have left that selected in the screenshot. I'm assuming you intended for me to have my pcie IP selected first.

In your 4) above, I do not have an "Address Spaces" section. I do have an "Address Map" in the middle section. I have most of my pcie components listed there, but there is no "I/O" row to be found.

Wincent_Altera · ‎02-20-2023

Hi,

the IO Space enable is at bit[0] of command register.

Can you please try it out and see if you are able to do that ?

Regards,

Wincent_Intel

bralston7 · ‎02-20-2023

Sorry, I cannot see where I could possibly do that in Platform designer. I can't find anything resembling 'command register' on the PCIe IP settings

Did you look at my screenshot? Could you show me a screenshot of what you are looking at?

Wincent_Altera · ‎02-20-2023

Wincent_Altera · ‎02-20-2023

Hi,

Please ignore my previous message as different version of quartus and different design might have different approach.
I share the IO Space enable is at bit[0] of command register to you.

Is it answering your question ?

Regards,

Wincent_Intel

bralston7 · ‎02-21-2023

While I appreciate the follow-up, that is just a diagram from the PCI spec. It doesn't tell me anything about making this work in the Intel PCIe IP component for Stratix 10. The Platform Designer GUI does not have the granularity or visibility to the command status registers.

Wincent_Altera · ‎02-21-2023

Hi,

If refer to the document Stratix 10 Avalon-MM Interface for PCI Express Solutions User Guide, under Figure 73 there is I/O space address map. Refer to 10.4.3, there are steps to configure the Root Port and Endpoint Configuration Space registers before you issue transactions to the Endpoint. Step 3a mentions that the I/O is assigned in the Endpoint BAR register. Hence, you do not need to assign the I/O bar address on the rootport. The Root Port BFM does not support accesses to Endpoint I/O space BARs.

Regards,

Wincent_Intel

Wincent_Altera · ‎02-23-2023

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

Wincent_Altera · ‎02-23-2023

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

bralston7 · ‎02-23-2023

Wincent, my problem has not been solved. I don't think the question has been understood. Let me try another direction. Look back at those address' back in the linux-side device tree that you posted your very first response:

ranges = <0x81000000 0x0 0x00000000 0x00010000 0x00000000 0x00010000>,

<0x82000000 0x0 0x00000000 0x00010000 0x00000000 0x00010000>;

You added the line that starts with an 0x81000000, which is correct, because that declares an I/O BAR. Now look at the red highlighted 64-bit addresses. These point to an address of the pci bus controller. They can't be the same thing. They also must have a backing on the FPGA side created by the PCIe IP. This backing exists for the second line that declares a MEM bar, which is called the txs or hptxs on the FPGA side. But the backing does not exist for the first line. The whole point of this question is how to make a compatible IO space on the FPGA side so that I can point to it with that 0x81 line from the device tree.

Wincent_Altera · ‎02-26-2023

Hi,

I take a deep look at it, the Rootport for AVMM configuration is support IO read and write via CRA port.

https://www.intel.com/content/www/us/en/support/programmable/articles/000077505.html

if refer to the rocketboard website , you can try to configure it by using the address offset.

https://www.rocketboards.org/foswiki/Projects/Stratix10PCIeRootPortWithMSI

Also, friendly reminder, the root port testbench is not available for AVMM..

Stratix 10 Avalon-MM Interface for PCI Express Solutions User Guide

Hope this answering your question. Let me know if you need any further clarification.

Regards,

WIncent_Intel

Wincent_Altera · ‎02-27-2023

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

Wincent_Altera · ‎02-28-2023

Hi,

I wish to follow up with you about this IPS case.

Hoping to hear back from you so that we can proceed for next step.

Regards,

Wincent_Intel

bralston7 · ‎02-28-2023

I'm not really sure how to proceed, as I don't think we're going to come to a conclusion here. I have been working from the rocketboards PCI link from day 1, which doesn't use an I/O bar, and I have also read the knowledge base article you linked on several occasions while googling. The problem with what the knowledge base solution (using the CRA port), is that it nowhere does it tell you where it is legal to map to in that region. The CRA port is used by the bridge driver and has custom translations for driver-ish configuration stuff. But its external interface is not defined. Only its base address. You can't just map to a random part of it or everything breaks. I've actually tried it.

So, if you don't have any further suggestions, then go ahead and close this and I'll explore other options.

bralston7 · ‎03-03-2023

Wincent, you may close this issue. I believe I have been convinced by someone else at Intel that an IO bar is NOT required for my device, and that the driver has legacy firmware that asks for one, but doesn't necessarily need it. I've still got a PCIe-related driver problem, but its likely not because I need an IO bar. Thanks for your help.

-Ben

Wincent_Altera · ‎03-05-2023

Hi ,

Thanks for your confirmation to close this case.

Apologies for cannot help much on that, as the information of configure the IO space is not stated detail in the datasheet.

I will feedback to the document team to double check, if the IO space is configurable, will update in future release of document, else will mention that it is not supported.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If you feel your support experience was less than a 9 or 10,

please allow me to correct it before closing or let me know the cause so that I may improve your future support experience.

Regards,

Wincent_Intel