- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all,
I have been working on a system with an Arria10 SoCFPGA with a M.2 SSD connection on the hard PCIe controller. This design is based on https://rocketboards.org/foswiki/Projects/A10AVCVPCIeRootPortWithMSILTS 1
As far as I can see this is all implemented as the example.
We are running based on a vanilla kernel V4.15.
The PCIe controller is detected correctly and the output of lspci is:
00:00.0 PCI bridge: Altera Corporation Device e000 (rev 01)
01:00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03)
Only during the startup of the NVMe drivers a timeout occurs during detection.
Has anyone had any experience with NVMe or other PCIe devices on an Arria10?
Kind regards,
Rienk de Jong
The relevant kernel logs:
[ 0.100998] OF: PCI: host bridge /sopc@0/pcie@0xd0000000 ranges:
[ 0.101034] OF: PCI: MEM 0xc0000000…0xdfffffff -> 0x00000000
[ 0.101227] altera-pcie d0000000.pcie: PCI host bridge to bus 0000:00
[ 0.101243] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.101257] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xdfffffff] (bus address [0x00000000-0x1fffffff])
[ 0.103448] PCI: bus0: Fast back to back transfers disabled
[ 0.106569] PCI: bus1: Fast back to back transfers disabled
[ 0.106844] pci 0000:00:00.0: BAR 8: assigned [mem 0xc0000000-0xc00fffff]
[ 0.106861] pci 0000:01:00.0: BAR 0: assigned [mem 0xc0000000-0xc0003fff 64bit]
[ 0.107027] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 0.107084] pci 0000:00:00.0: bridge window [mem 0xc0000000-0xc00fffff]
[ 61.290164] nvme nvme0: I/O 3 QID 0 timeout, disable controller
[ 61.450409] nvme nvme0: Identify Controller failed (-4)
[ 61.455625] nvme nvme0: Removing after probe failure status: -5
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
We have fixed our issue.
The problem was not related to the kernel drivers or settings of the PCIe IP itself.
The example on the rocketboards page is correct ( as I expected ).
Our problem was in the memory assigned to the Kernel and mapped to the PCIe core.
The board we have has a total of 4GB of ram for the HPS, which is effectively 3GB for the HPS because the HPS to FPGA bridges and other IO claims the upper 1GB of the available memory space.
The Avalon master port of the PCIe core must be connected to the memory through a address expanded and the window this provides to the RAM can be set in power of two increments so 1GB, 2 GB or 4GB.
In our case we had our boot-loader configured to give the Linux kernel 3GB of memory, and the PCIe core had a 2GB window to the RAM.
My guess is that the kernel had memory allocated for PCIe above the 2GB window.
The solution for our system was to limit the memory for the kernel to 2GB by configuring this in the boot-loader.
So for us the issue is solved, I hope these posts can help someone else that is trying to get PCIe working on there own custom Arria 10 HPS designs.
Regards,
Rienk
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Nathan,
Thanks for the response.
I get what your saying, but I feel your answer is a bit short sighted.
First that the NVME driver is at fault because it reports an error. This is the NVME driver in the mainline Linux kernel, it is used by many different devices. So I don't expect a bug in that part of the kernel.
Second, the NVME driver is actually from Intel, it is in the 3rd line of the core.c driver source. I know that Intel is big and it is form a completely different division. But a comment like this feels dismissive "Not our bug".
Sorry for these comments, but that bugged me a bit today.
Back to a constructive discussion:
I am pretty sure the problem is in either our HDL / Qsys design or the configuration of the kernel through the device tree.
Below is a the PCIe part of the device tree.
We have the PCIe Txs port connected to the high performance master of the HPS at offset 0x10000000
The PCIe Cra is connected to the low performance port at offset 0x0
The MSI interrupt controller is connected to the low performance port at offset 0x4080 and 0x4000
Attached is the archive of our project that we use now and it only contains the HPS and the PCIe core.
I hope you can help me figure out where we made our mistake.
Kind regards,
Rienk
pcie_0_pcie_a10_hip_avmm: pcie@0xd0000000 {
status = "okay";
compatible = "altr,pcie-root-port-16.1", "altr,pcie-root-port-1.0";
reg = <0xd0000000 0x10000000>,
<0xff210000 0x00004000>;
reg-names = "Txs", "Cra";
interrupt-parent = <&hps_arm_gic_0>;
interrupts = <0 25 4>; // irq6
interrupt-controller;
#interrupt-cells = <1>;
device_type = "pci"; /* embeddedsw.dts.params.device_type type STRING */
bus-range = <0x00000000 0x000000ff>;
ranges = <0x82000000 0x00000000 0x00000000 0xc0000000 0x00000000 0x10000000
0x82000000 0x00000000 0x10000000 0xd0000000 0x00000000 0x10000000>;
msi-parent = <&pcie_0_msi_to_gic_gen_0>;
#address-cells = <3>;
#size-cells = <2>;
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie_0_pcie_a10_hip_avmm 1>,
<0 0 0 2 &pcie_0_pcie_a10_hip_avmm 2>,
<0 0 0 3 &pcie_0_pcie_a10_hip_avmm 3>,
<0 0 0 4 &pcie_0_pcie_a10_hip_avmm 4>;
}; //end pcie@0x010000000 (pcie_0_pcie_a10_hip_avmm)
pcie_0_msi_to_gic_gen_0: msi@0xff214080 {
status = "okay";
compatible = "altr,msi-1.0", "altr,msi-1.0";
reg = <0xff214080 0x00000010>,
<0xff214000 0x00000080>;
reg-names = "csr", "vector_slave";
num-vectors = <32>; /* embeddedsw.dts.params.num-vectors type NUMBER */
interrupt-parent = <&hps_arm_gic_0>;
interrupts = <0 24 4>; // irq5
clocks = <&h2f_lw_clk>;
msi-controller = <1>; /* embeddedsw.dts.params.msi-controller type NUMBER */
}; //end msi@0x100014080 (pcie_0_msi_to_gic_gen_0)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
We have fixed our issue.
The problem was not related to the kernel drivers or settings of the PCIe IP itself.
The example on the rocketboards page is correct ( as I expected ).
Our problem was in the memory assigned to the Kernel and mapped to the PCIe core.
The board we have has a total of 4GB of ram for the HPS, which is effectively 3GB for the HPS because the HPS to FPGA bridges and other IO claims the upper 1GB of the available memory space.
The Avalon master port of the PCIe core must be connected to the memory through a address expanded and the window this provides to the RAM can be set in power of two increments so 1GB, 2 GB or 4GB.
In our case we had our boot-loader configured to give the Linux kernel 3GB of memory, and the PCIe core had a 2GB window to the RAM.
My guess is that the kernel had memory allocated for PCIe above the 2GB window.
The solution for our system was to limit the memory for the kernel to 2GB by configuring this in the boot-loader.
So for us the issue is solved, I hope these posts can help someone else that is trying to get PCIe working on there own custom Arria 10 HPS designs.
Regards,
Rienk

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page