Crash when programming .sof file

OHarb1 · ‎05-04-2020

I saw this crash while programming a .sof file on Ubuntu:

Has anyone else run into this?

*** Fatal Error: Segment Violation at 0x22f700000076

Module: quartus_pgm

Stack Trace:

0xf8ea: rml::internal::internalPoolFree(rml::internal::MemoryPool*, void*) + 0x4a (tbbmalloc.so.2)

0x1729: operator delete(void*) + 0x9 (tbbmalloc_proxy.so.2)

0x2083e: S4_SII_BIT::~S4_SII_BIT() + 0x3e (pgm_build_npcf)

0x20901: S4_SII::~S4_SII() + 0x5b (pgm_build_npcf)

0x209de: NPCF_DEVICE_INFO::set_s4_s2_list(std::vector<S4_SII*, std::allocator<S4_SII*> >*, bool, bool) + 0x4c (pgm_build_npcf)

0x24655: NPCF_DEVICE_INFO::~NPCF_DEVICE_INFO() + 0x169 (pgm_build_npcf)

0x246e7: NPCF_INFO::~NPCF_INFO() + 0x41 (pgm_build_npcf)

0xe183c: FBGEN_GEN::~FBGEN_GEN() + 0x5c (pgm_fbgen)

0xe1e19: FBGEN_GEN::~FBGEN_GEN() + 0x9 (pgm_fbgen)

0x420be3: PGMIO_FBGEN_PROXY::~PGMIO_FBGEN_PROXY() + 0x13 (pgm_pgmio)

0x2c2d5c: PGMIO_F2P::create_bitstream(PGM_CHAIN_ELEMENT*, std::vector<std::string, std::allocator<std::string> >*, PGMIO_CCF*) + 0x1dc (pgm_pgmio)

0x284c14: PGM_CHAIN_ELEMENT::generate_bv_list(bool) + 0x104 (pgm_pgmio)

0x28868d: PGM_CHAIN_ELEMENT::create_chain_element(PGM_CHAIN_ELEMENT*, bool, FIO_PATH*, bool, PGMIO_CONFIG_SCHEME, bool, bool) + 0xd2d (pgm_pgmio)

0x232d9: PGME_PROGRAMMER::lookup_device(PGM_CHAIN_ELEMENT*, PGMIO_CONFIG_SCHEME, bool, bool, bool) + 0x29 (pgm_pgme)

0x2175d: QPGM_FRAMEWORK::create_element(std::string, std::string, unsigned int, unsigned int) + 0x601 (quartus_pgm)

0x23b91: QPGM_FRAMEWORK::process_operation(std::string*) + 0x1e93 (quartus_pgm)

0x24cde: QPGM_FRAMEWORK::post_check_arguments() + 0x2d6 (quartus_pgm)

0x1c08f: qexe_standard_main(QEXE_FRAMEWORK*, QEXE_OPTION_DEFINITION const**, int, char const**) + 0x1bc (comp_qexe)

0x1fd97: qpgm_main(int, char const**) + 0x5e (quartus_pgm)

0x40720: msg_main_thread(void*) + 0x10 (ccl_msg)

0x602c: thr_final_wrapper + 0xc (ccl_thr)

0x407df: msg_thread_wrapper(void* (*)(void*), void*) + 0x62 (ccl_msg)

0xa559: mem_thread_wrapper(void* (*)(void*), void*) + 0x99 (ccl_mem)

0x8f92: err_thread_wrapper(void* (*)(void*), void*) + 0x27 (ccl_err)

0x63f2: thr_thread_wrapper + 0x15 (ccl_thr)

0x427e2: msg_exe_main(int, char const**, int (*)(int, char const**)) + 0xa3 (ccl_msg)

0x1fe21: main + 0x26 (quartus_pgm)

0x270b3: __libc_start_main + 0xf3 (c.so.6)

JohnT_Intel · ‎05-05-2020

Hi,

May I know which device you are programming? Have you tried on other system?

OHarb1 · ‎05-05-2020

This is an Arria 10 device.

We've set up an automated test-server (to test incoming pull request) and we're seeing this failure occasionally on that server.

That this is not stable is problematic as it reduces the throughput of that test-server.

JohnT_Intel · ‎05-05-2020

Hi,

May I know if you performed programming at the same time or you are running other thing? From the error message, the issue might be facing some memory issue which cause the Quartus Programmer to crash.

OHarb1 · ‎05-05-2020

Nothing else was running at the time. This is an automated script, so I know nothing else is going on on that machine. It is a Jenkins slave that's dedicated to regression teseting our FPGA bit file. The Arria 10 PC is a PCIe card in that machine and we're programming it using a loopback USB cable(we can't use CvP as we need to change the clock frequency of the design).

JohnT_Intel · ‎05-05-2020

Hi,

What do you mean by loopback USB cable? Could you provide me more detail information on your script and how to duplicate the issue? May I know what is the probability of the issue occurring?

OHarb1 · ‎05-05-2020

I need more data on the frequency, but I guesstimate the chances of failure to be 1:10.

The loopback cable means that the the Arria 10 PCIe eval card is in the PC and that I have connected the USB Blaster of that eval card to the same PC.

The command I used to program the FPGA is the below. The second time I ran it, using the same .sof file, just 30 minutes afterwards without any other change that I'm aware of, it worked.

Hmm.... come to think of it... We've upgraded to Quartus 20.1 elsewhere, but the niso2-configure-sof command is still from Quartus 17.3 (as I recall).

# program FPGA

nios2-configure-sof top.sof

The output is:

Searching for SOF file:

in .

top.sof