Intel® Quartus® Prime Software
Intel® Quartus® Prime Design Software, Design Entry, Synthesis, Simulation, Verification, Timing Analysis, System Design (Platform Designer, formerly Qsys)
Announcements
Intel Support hours are Monday-Fridays, 8am-5pm PST, except Holidays. Thanks to our community members who provide support during our down time or before we get to your questions. We appreciate you!

Need Forum Guidance? Click here
Search our FPGA Knowledge Articles here.
15543 Discussions

How to reliabily program multiple FPGAs using quartus_pgm?

swm
Beginner
512 Views

I have a server with 8 x Stratix 10 SX boards (Terasic DE10Pro), which I would like to be able to program over usb-to-jtag.  When I have one or two baords installed, I can reliably program a board, e.g. using:

quartus_pgm -c 1 -m jtag -o "p;output_files/DE10_Pro.sof@2"

However, once all eight FPGA baords are connected, the programming process times out:

Info (18942): Configuring device index 2
Error (18939): Unexpected error in JTAG server: Timeout
Error (18939): Unexpected error in JTAG server: Timeout
Error (18952): Error status: Synchronization failed
Error (209012): Operation failed

Is quartus_pgm known to work with eight or more boards?

 

- Simon

0 Kudos
15 Replies
Farabi
Employee
501 Views

Hello Simon,


How did you connect the JTAG chain? Suspect JTAG signal integrity issue when there is too long connection in the JTAG chain. Can tap this signal and observe in oscilloscope.


Its not quartus programmer issue.


regards,

Farabi


swm
Beginner
486 Views

Hello Farabi, thanks for your reply.  I believe that I don't have a signal integrity problem on the USB chain due to the following experiements that did not involve changing the USB wiring:

  1. Disable some USB ports in software, effectively removing some FPGAs visible to jtagd.  The remaining FPGAs could then be repeatedly programmed successfully without and reported errors.
  2. Run eight Docker containers each with one jtagd that can only see one FPGA.  The FPGAs can then be repeatedly programmed reliably.

Do you know if USB-to-JTAG programming of at least eight FPGAs connected to one PC has been tested?

FvM
New Contributor I
447 Views

There's a general problem when working with multiple programmers connected to one PC (= one JTAG server). Establishing the connection to a programmer can fail if another programmer is active at that time. Connecting all programmers first and than starting activities should work. 

This might be a problem with the intended command line operation but it works in GUI operation of multiple programmer or debug tool instances.


swm
Beginner
421 Views

Thanks for confirming that you have also seen difficulties with multiple programmers connected to one PC.

I did try the GUI for programming, but this didn't improve programming reliability; I see the same behaviour.  I am only programming one FPGA at a time to avoid any issues with jtagd struggling with parallel clients.

Fakhrul
Employee
438 Views

Hi swm,


While I'm trying to look into your question, have you tried to look at the following documents?

It may perhaps help to troubleshoot your issue.


JTAG Multi-Device Configuration

Debugging Suggestions


Regards

Fakhrul



swm
Beginner
420 Views

Hello Fakhrul,

Thanks for investigating.

The JTAG Multi-Device Configuration documentation appears to only cover multiple FPGAs on the same JTAG chain.  For my setup I have a USB-to-JTAG adapter per FPGA (on the Terasic DE10Pro board) rather than chaining many FPGAs on the same JTAG chain.  Simularly for Debugging Suggestions the bug isn't with multiple FPGAs on the same JTAG chain.

I'm pretty certain the bug is in software, most likely in the jtagd demon but possibly in quartus_pgm.

I have the eight FPGAs connected via two 4-port USB hubs that have per-port controllable power on/off.  If I use uhubctl (under Linux) to turn off all ports exept for one FPGA and restart jtagd so that rescans USB ports, then I'm able to program that one FPGA reliably.  If I enable any two FPGAs, then I'm able to program them.  But if I enable all eight FPGAs the programming usually fails.

- Simon

 

Fakhrul
Employee
391 Views

Hi swm,


Understood, I am checking this internally for more clarification.


I'd appreciate it if you could provide me with more information such as:

1. Could you please provide the screenshots of the project; Programmer; JTAG configuration etc?

2. Screenshots of the error message

3. Version of Quartus


Just to let you know that it might take some time, will get to you once there are any findings.


Regards,

Fakhrul



swm
Beginner
373 Views

Hello Fakhrul,

I'm using Quartus 21.3Pro at present.  I have 8 x Terasic DE10Pro cards that contain a Stratix 10 SX part and an on-board USB Blaster II that I'm using to program them in FPGA mode only (i.e. I'm not using the HPS at present).  The details of the FPGA image don't appear to matter.  In fact the specific evaluation card doesn't appear to matter either: we've seen similar problems programming several Stratix V boards, e.g. the Terasic DE5.

The DE10Pro boards are in a GPU-style server (a dual-socket AMD Epyc running Ubuntu 20.04.4 LTS) plugged into PCIe slots.  The USB tree is as follows: 2 x native USB3 boards on the motherboard each connected to a four-port USB hub.  Four boards are plugged into one hub and four into the other.

I'm programming the boards from the command-line, e.g.:

quartus_pgm -m jtag -c "DE10-Pro [5-2.3.1]" -o "p;../output_files/DE10_Pro.sof@1"

I've since done a bit of Python scripting (attached, with .py replaced with .txt so that it woudl upload) to monitor the output of quartus_pgm and retry programming on any errors, which at least gives me a good way forward.  When using this script to reprogram all 8 boards 100 times I'm seeing around 10 to 20 retries, which isn't too bad.  An example error report from quartus_pgm is also attached (quartus_pgm_error.txt).  While this is probably now good enough that we can use the systems for automated regression test, etc., it is still curious that there are some failures.  

I also notice that quartus_pgm checked for licenses but doesn't require a license to run.  Making LM_LICENSE_FILE environment variable to an empty string causes quartus_pgm to not check for a license, which speeds things up.

- Simon

Fakhrul
Employee
326 Views

Hi Simon,


Thank you for the information. Just to let you know, I am still checking this internally for more clarification

I will get to you once there are any findings.


Thank you.


Regards,

Fakhrul


Fakhrul
Employee
282 Views

Hi Simon,


I'm still checking this with the internal teams. Unfortunately, I have not heard back from them yet.

I will plan on letting you know as soon as I learn more information.


Thanks for your patience!


Warm Regards,

Fakhrul



swm
Beginner
272 Views

Many thanks Fakhrul.  I look forward to hearing from you.

Fakhrul
Employee
219 Views

Hi Simon,


I'm still checking this with the internal teams. Unfortunately, I have not heard back from them yet.

I will plan on letting you know as soon as I learn more information.


At the mean time, could you please share what module, brand of USB hub you're using?


Thanks for your patience!


Warm Regards,

Fakhrul


swm
Beginner
210 Views

Hello Fakhrul,

 

Thanks for continuing to look into this.  I'm still very interested to know about the general case rather that the specifics of my setup.  It would be interesting to know if programming eight or more FPGA evaluation cards connected to one PC with a USB-to-JTAG adapter for each board has been tested regardless of the particular FPGA baord.  That said, programming large FPGAs like the Stratix 10 with a full design is far more likely to provoke the issues reported.

 

Our primary FPGA board at present is the DE10Pro board from Teraisc:
https://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=164&No=1144#contents

 

The USB-to-JTAG interface is embedded on this board.

 

I don't believe the USB hub we are using is a factor since we've seen the same problems using a range of different USB host controllers and USB hubs.

 

Thanks,

Simon

Fakhrul
Employee
175 Views

Hi Simon,


Understood. I am still checking this internally for more clarification

I will get to you once there are any findings


Thanks for your patience!


Fakhrul
Employee
78 Views

Hi Simon,


Based on the feedback I got. Unfortunately, we doesn't have a specific limitation test for this particular setup.

But, though you might be able to connect it like the way you do it right now, we highly recommend that the best way to connect multiple FPGA boards while avoiding any JTAG signal integrity issue is through the daisy chain configuration.


Regards,

Fakhrul


Reply