Issue with updating ARIA 10 PCIe core to 20_1_1

[mwf]

We are migrating to a new hardware design server and are converting existing FPGA designs from Quartus 17.1 to Quartus 22.3. We have a number of PCIe Gen 3 x 8 designs that build without issue and run fine on the hardware. This is with both Quartus 17.1 and 22.3. Using 22.3 requires an update of the PCIe core to 20_1_1. This core works fine when configured as Gen 3 x 8.

We also have a number of PCIe Gen 2 x 8 designs. They all work fine when built with Quartus 17.1. But the first design I tried builds with 22.3 but doesn't work on the hardware. I get various errors including PC hangs, no access to PIO slave registers, and DMA failures from our FPGA to host (PC) memory.  Changing seeds changes behavior slightly. 

I also tried Quartus 19.2 which has an optional PCIe core update to 19_1. This core appears to work fine when configured as Gen 2 x 8.

The PCIe core and a lot of our IP run at 250Mhz. There are no timing errors reported by either Quartus version.

Thanks for any ideas,

Mike

[wchiah]

Hi,

We do not recommend direct migrating the design from the previous version to the latest.

As some IP feature might be obsolete/upgraded.

Hence, my best suggestion to you will be re-generate a new design by refering to IP catalog or FPGA design store.

https://www.intel.com/content/www/us/en/support/programmable/support-resources/design-examples/design-store.html

Hope this clarified,

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

[mwf]

Regenerating the PCIe core from scratch didn't help. I still get random results. We have done many builds, due to design enhancements and bugs, in Quartus 17.1. We have never had PCIe issues on any of our builds.

To summarize where we are right now:

All three of the Quartus 17.1 Gen 3 x 8 designs moved to Quartus 22.3 build fine and work fine on our hardware.

Two Quartus 17.1 Gen 2 x 8 designs moved to Quartus 22.3 build fine but fail to run on our hardware. 

I thought I had some success with Quartus 19.2 with PCIe core version 19_1 but a slight change in the design caused failures with PCIe DMA.

Thanks,

Mike

[wchiah]

Hi,

What is the fail about ?

Any error code shows ?

Regards,

Wincent_Intel

[mwf]

Hi Wincent.

The failure varies build to build. I am looking at the most obvious error which is the FPGA enumerates okay in the PCIe but all host to FPGA accesses return 0xffffffff. What I just noticed in the lab is the LEDs seem to indicate a good portion of the FPGA is not working. It programmed and loaded okay but more than just PCIe appears to be broken.

Other failures include lockups, DMA (device to host failures), and host to FPGA access returning wrong data (other than 0xffffffff).

I performed a DSE run and out of 11 builds:

3 - work okay

4 - experience the 0xfffffff problem

4 - have other issues as listed above.

I did a DSE run on our other design which is Gen 3 x 8 and targets the exact same board. All 11 builds work fine. 

I will continue to look at one of failing builds.

Thanks,

Mike

[wchiah]

Hi,

Did you means all work okay and only one is fail ?

Can you check the dip switch and please ensure all return to default as per mention in the user guide.

Get back to me your finding.

Regards,

Wei Chuan

[mwf]

Hi Wei.

On the Gen 2 x 8 design 7 out of 11 builds fail to run correctly on the hardware.

On the Gen 3 x 8 design all 11 builds run correctly on the hardware.

Just to reiterate the same board was used to test all builds. 

This is our own hardware design so it doesn't have a dip switch. I am not sure what you want me to check.

Thanks,

Mike

[wchiah]

Hi,

If not all fail, that normally means not the example design or IP issue.

Based on my experience on of the possibility is the dip switch had been adjusted and not in the default.

Please confirm on it.

Let me know if still cannot solve.

Regards,

Wincent_Intel

[mwf]

Hi. Can you explain further about the dip switch? I am sorry but I have no idea what you are referring to.

Thanks,

Mike

[wchiah]

Hi,

Please refer to the user guide under section 3.1 Development Board Setup

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_a10-fpga-prod-devkit.pdf

Please ensure that all switch is on the default setting as per mention in the user guide.

Regards,

Wincent_Intel

[mwf]

I don't have a development board. This design targets our own custom hardware.

[wchiah]

Hi,

Can you see any burn marks on your custom hardware ? 
or there is any connection fail on the power cable ?

Regards,
Wincent_Intel

[mwf]

Hello.

There is nothing wrong with the board. Let me summarize again:

1. We have two FPGA designs that target the exact same board design. 

2. FPGA design A supports 4 lanes of video which each run at 5Gbps. This design uses the PCIe Gen 2 x 8 core.

3. FPGA design B supports 4 lanes of video which each run at 10Gbps. This design uses the PCIe Gen 3 x 8 core.

4. The same physical board was used for all testing.

5. The PCIe Gen 2 x 8 build fails in various ways detailed earlier in this thread. I have done multiple DSE runs and all have a success rate of 30 - 40%, meaning some results work as expected. 13 out of 33 builds work.

6. The PCIe Gen 3 x 8 build always works 100%. I have done two seed sweeps of 10 and including base that means 22 builds work fine.

7. When I perform a seed sweep on the Gen 2 x 8 design using Quartus 17.1 all results work fine. 11 out of 11 work.

We need to understand what breaks the Gen 2 x 8 design in Quartus 22.3.

Thanks,

Mike

[wchiah]

Hi,

Sorry for late reply due to holiday session

I check internal information, for Arria 10 we do not suggest direct migration from gen2x8 to gen3.

I will raise an internal ticket for this, hope the related team will work on this in next release version of update (but I cannot promise that)

I lay down some of the suggestion.

• Continue to use quartus v17.1 on the specific design
• Recompile/rebuild the whole design in newest quartus. If the same error still happen, please let me know.

Regards,

Wincent_Intel

[mwf]

1. We are not migrating a design from Gen 2 x8 to Gen 3 x 8. By the way that works fine but is not part of this discussion. I can move freely from Gen 2 x 8 with a 125 mhz 256  bit internal bus to a Gen 3 x 8 with a 250 mhz 256 bit bus. That is not the issue here. This issue is when we migrated a Gen 2 x 8 design from Quartus 17.1 to Quartus 22.3 it stopped compiling correctly. A Gen 3 x 8 design migrated from 17.1 to 22.2 works fine.

2. That could be an option but what is the point in using 6 year old software on a design that has a production part on it.

3. Sigh, that is what this entire thread is about. We recomplied a design in the newest version of Quartus and it stopped compiling reliably.

[wchiah]

Hi,

My fault did not check correctly before sent out the mail.
What I mean was direct migration from Quartus v17.1 to any newer version instead of direct migration from gen2x8 to gen3.

Can you share with me the .qar file ? I can try to compile on my place and see if I able to run full compilation or not.

Regards,

Wincent_Intel

[mwf]

Hi Wincent. We already have sent a qar file via our distributor Arrow and Intel was able to compile it successfully.

The issue is not the compile but the compilation result not working.

Here is some more information that may be useful.

This is a working PCIe status:

Failing, you can see the latency is 0xff and the Max Payload is different.

Above is the most serious error. As I mentioned some builds work, some have the above issue, and others hang the PC when I try to do DMA to the host.

This shows there is something fundamentally wrong with the build. 

Thanks,

Mike

[wchiah]

Hi,

Can I confirm with you this error happens intermittently in Quartus v22.4?

or it fall all the time ? from your previous reply, I understand that it is intermittent.

As I mention, direct migration are not suggested as there is some features been update/out of service.

It is hard for us to trace one by one which one is the root cause.

Is it possible for you to rebuild/regenerate the whole design in Quartus v22.4 ?

Get back to us if you still see the same error.

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this case. Any update after rebuild ?

Regards,

Wincent_Intel

[mwf]

Hi Wincent. I did another scrub of the differences between the designs, specifically the .qsf and source files. Nothing stood out. I then compared the .sdc files and found a flase path constraint in the Gen 2 x 8 design that was not present in the Gen 3 x 8 design. Once I removed the constraint, which was a bit heavy handed, the design runs fine on the hardware.

Thanks for your assistance,

Mike