How to reliabily program multiple FPGAs using quartus_pgm?

[swm]

I have a server with 8 x Stratix 10 SX boards (Terasic DE10Pro), which I would like to be able to program over usb-to-jtag.  When I have one or two baords installed, I can reliably program a board, e.g. using:

quartus_pgm -c 1 -m jtag -o "p;output_files/DE10_Pro.sof@2"

However, once all eight FPGA baords are connected, the programming process times out:

Info (18942): Configuring device index 2
Error (18939): Unexpected error in JTAG server: Timeout
Error (18939): Unexpected error in JTAG server: Timeout
Error (18952): Error status: Synchronization failed
Error (209012): Operation failed

Is quartus_pgm known to work with eight or more boards?

- Simon

[Farabi]

Hello Simon,

How did you connect the JTAG chain? Suspect JTAG signal integrity issue when there is too long connection in the JTAG chain. Can tap this signal and observe in oscilloscope.

Its not quartus programmer issue.

regards,

Farabi

[swm]

Hello Farabi, thanks for your reply.  I believe that I don't have a signal integrity problem on the USB chain due to the following experiements that did not involve changing the USB wiring:

1. Disable some USB ports in software, effectively removing some FPGAs visible to jtagd.  The remaining FPGAs could then be repeatedly programmed successfully without and reported errors.
2. Run eight Docker containers each with one jtagd that can only see one FPGA.  The FPGAs can then be repeatedly programmed reliably.

Do you know if USB-to-JTAG programming of at least eight FPGAs connected to one PC has been tested?

[FvM]

There's a general problem when working with multiple programmers connected to one PC (= one JTAG server). Establishing the connection to a programmer can fail if another programmer is active at that time. Connecting all programmers first and than starting activities should work. 

This might be a problem with the intended command line operation but it works in GUI operation of multiple programmer or debug tool instances.

[swm]

Thanks for confirming that you have also seen difficulties with multiple programmers connected to one PC.

I did try the GUI for programming, but this didn't improve programming reliability; I see the same behaviour.  I am only programming one FPGA at a time to avoid any issues with jtagd struggling with parallel clients.

[Fakhrul]

Hi swm,

While I'm trying to look into your question, have you tried to look at the following documents?

It may perhaps help to troubleshoot your issue.

JTAG Multi-Device Configuration

Debugging Suggestions

Regards

Fakhrul

[swm]

Hello Fakhrul,

Thanks for investigating.

The JTAG Multi-Device Configuration documentation appears to only cover multiple FPGAs on the same JTAG chain.  For my setup I have a USB-to-JTAG adapter per FPGA (on the Terasic DE10Pro board) rather than chaining many FPGAs on the same JTAG chain.  Simularly for Debugging Suggestions the bug isn't with multiple FPGAs on the same JTAG chain.

I'm pretty certain the bug is in software, most likely in the jtagd demon but possibly in quartus_pgm.

I have the eight FPGAs connected via two 4-port USB hubs that have per-port controllable power on/off.  If I use uhubctl (under Linux) to turn off all ports exept for one FPGA and restart jtagd so that rescans USB ports, then I'm able to program that one FPGA reliably.  If I enable any two FPGAs, then I'm able to program them.  But if I enable all eight FPGAs the programming usually fails.

- Simon

[Fakhrul]

Hi swm,

Understood, I am checking this internally for more clarification.

I'd appreciate it if you could provide me with more information such as:

1. Could you please provide the screenshots of the project; Programmer; JTAG configuration etc?

2. Screenshots of the error message

3. Version of Quartus

Just to let you know that it might take some time, will get to you once there are any findings.

Regards,

Fakhrul

[swm]

Hello Fakhrul,

I'm using Quartus 21.3Pro at present.  I have 8 x Terasic DE10Pro cards that contain a Stratix 10 SX part and an on-board USB Blaster II that I'm using to program them in FPGA mode only (i.e. I'm not using the HPS at present).  The details of the FPGA image don't appear to matter.  In fact the specific evaluation card doesn't appear to matter either: we've seen similar problems programming several Stratix V boards, e.g. the Terasic DE5.

The DE10Pro boards are in a GPU-style server (a dual-socket AMD Epyc running Ubuntu 20.04.4 LTS) plugged into PCIe slots.  The USB tree is as follows: 2 x native USB3 boards on the motherboard each connected to a four-port USB hub.  Four boards are plugged into one hub and four into the other.

I'm programming the boards from the command-line, e.g.:

quartus_pgm -m jtag -c "DE10-Pro [5-2.3.1]" -o "p;../output_files/DE10_Pro.sof@1"

I've since done a bit of Python scripting (attached, with .py replaced with .txt so that it woudl upload) to monitor the output of quartus_pgm and retry programming on any errors, which at least gives me a good way forward.  When using this script to reprogram all 8 boards 100 times I'm seeing around 10 to 20 retries, which isn't too bad.  An example error report from quartus_pgm is also attached (quartus_pgm_error.txt).  While this is probably now good enough that we can use the systems for automated regression test, etc., it is still curious that there are some failures.  

I also notice that quartus_pgm checked for licenses but doesn't require a license to run.  Making LM_LICENSE_FILE environment variable to an empty string causes quartus_pgm to not check for a license, which speeds things up.

- Simon

[Fakhrul]

Hi Simon,

Thank you for the information. Just to let you know, I am still checking this internally for more clarification

I will get to you once there are any findings.

Thank you.

Regards,

Fakhrul

[Fakhrul]

Hi Simon,

I'm still checking this with the internal teams. Unfortunately, I have not heard back from them yet.

I will plan on letting you know as soon as I learn more information.

Thanks for your patience!

Warm Regards,

Fakhrul

[swm]

Many thanks Fakhrul.  I look forward to hearing from you.

[Fakhrul]

Hi Simon,

I'm still checking this with the internal teams. Unfortunately, I have not heard back from them yet.

I will plan on letting you know as soon as I learn more information.

At the mean time, could you please share what module, brand of USB hub you're using?

Thanks for your patience!

Warm Regards,

Fakhrul

[swm]

Hello Fakhrul,

Thanks for continuing to look into this.  I'm still very interested to know about the general case rather that the specifics of my setup.  It would be interesting to know if programming eight or more FPGA evaluation cards connected to one PC with a USB-to-JTAG adapter for each board has been tested regardless of the particular FPGA baord.  That said, programming large FPGAs like the Stratix 10 with a full design is far more likely to provoke the issues reported.

Our primary FPGA board at present is the DE10Pro board from Teraisc:
https://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=164&No=1144#contents

The USB-to-JTAG interface is embedded on this board.

I don't believe the USB hub we are using is a factor since we've seen the same problems using a range of different USB host controllers and USB hubs.

Thanks,

Simon

[Fakhrul]

Hi Simon,

Understood. I am still checking this internally for more clarification

I will get to you once there are any findings

Thanks for your patience!

[Fakhrul]

Hi Simon,

Based on the feedback I got. Unfortunately, we doesn't have a specific limitation test for this particular setup.

But, though you might be able to connect it like the way you do it right now, we highly recommend that the best way to connect multiple FPGA boards while avoiding any JTAG signal integrity issue is through the daisy chain configuration.

Regards,

Fakhrul

[Fakhrul]

As we do not receive any response from you to the previous answer that I provided.

This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

[swm]

Thanks for investigating this issue, but your answer is technically incorrect.  I only have one FPGA on each JTAG chain not several.  So there are no signal integrity issues with the JTAG.  The problem is that I have multiple USB connections, one for each FPGA, and the jtagd driver does not cope well with multiple FPGAs each on their own USB-to-JTAG adapter.  I'm certain that this is a software issue and not a hardware issue as previsouly explained:

1. Disable some USB ports in software, effectively removing some FPGAs visible to jtagd.  The remaining FPGAs could then be repeatedly programmed successfully without and reported errors.
2. Run eight Docker containers each with one jtagd that can only see one FPGA.  The FPGAs can then be repeatedly programmed reliably.

I would still like somebody at Intel to test jtagd when at least four evaluation cards are connected to the same PC using a USB-to-JTAG link for each board.  The bug manifests itself most clearly if the FPGAs and the bit image for the FPGAs is large.

Given that jtagd is designed to support multiple FPGA evaluation cards, I am looking for confirmation that there is a regression test that it does indeed work correctly when several cards are connected to the same machine, e.g. that jtagd can in fact reliabily handle multiple FPGA evaluation cards connected.  This seems like a reasonable request.

[Fakhrul]

Hi Simon,

When you've mentioned that you're facing the issue with quartus_pgm, how about using Programmer GUI?

Regards,

Fakhrul

[swm]

I have used both the Programer GUI and the command-line quartus_pgm tool with identical results.

[Fakhrul]

Hi Simon,

I see, I've talked with some experts, and he said that it might be due to the hub issue. Maybe the power to the blaster is not fully sufficient. Is there any issue if only connected to 3 blaster on each hub?