Solved: Quartus crash with "Internal Error: Sub-system: THR"

HKim27 · ‎10-23-2019

Hi, all.

I use Intel® Acceleration Stack Version 1.2 with Intel® Programmable Acceleration Card with Intel Arria® 10 GX FPGA on CentOS Linux release 7.2.1511.

When I compile my design, Quartus keeps crashing with the following error:

Internal Error: Sub-system: THR, File: /quartus/ccl/thr/thr_interprocess_mutex.cpp, Line: 122
err == EACCES || err == EAGAIN
Stack Trace:
     0x5db7: thr_internal_error + 0x1e (ccl_thr)
     0x9d09: thr_ip_mutex_lock_helper(THR_IP_MUTEX_STRUCT*, bool) + 0x81 (ccl_thr)
     0x9d5c: thr_ip_mutex_lock + 0xe (ccl_thr)
    0x89eb1: RDB_CONCURRENT_ACCESS::lock(std::string const&, RDB_REPORT_TYPE) + 0x2bf (db_rdb)
    0x438a1: FIT2_REPORTING::initialize_rdb(CMP_FACADE&, QHD_ENUM_IMPL<QHD_STATE::TYPE_BASE>, bool) + 0xb3 (comp_fit2)
    0x3fa25: FIT2_DATABASE_EXPERT_QDB::load(FIT2_ENV::STAGE, std::string, unsigned int const*, bool, bool, bool) + 0x679 (comp_fit2)
    0x253f4: fit2_create_fitter_netlist + 0x1154 (comp_fit2)
    0x4c977: TclNRRunCallbacks + 0x47 (tcl8.6)
    0x4e1ca: TclEvalEx + 0x96a (tcl8.6)
    0x4e4d6: Tcl_EvalEx + 0x16 (tcl8.6)
    0x4e4fd: Tcl_Eval + 0x1d (tcl8.6)
    0x1c9e1: atcl_tcl_eval(Tcl_Interp*, std::string const&) + 0x12d (ccl_atcl)
    0x44bba: fit2_run_create_fitter_netlist(Tcl_Interp*, std::string const&, ATCL_ARGS&, int&) + 0x5cd (comp_fit2)
    0x26663: fit2_fit_plan_init + 0xd3 (comp_fit2)
    0x4c977: TclNRRunCallbacks + 0x47 (tcl8.6)
    0x14973: fit2_fit_plan + 0x2ff (comp_fit2)
    0x4c977: TclNRRunCallbacks + 0x47 (tcl8.6)
    0x4e1ca: TclEvalEx + 0x96a (tcl8.6)
    0xf68c6: Tcl_FSEvalFileEx + 0x266 (tcl8.6)
    0xf69ce: Tcl_EvalFile + 0x2e (tcl8.6)
    0x11d8b: qexe_evaluate_tcl_script(std::string const&) + 0x390 (comp_qexe)
    0x1611d: qexe_do_tcl(QEXE_FRAMEWORK*, std::string const&, std::string const&, std::list<std::string, std::allocator<std::string> > const&, bool, bool) + 0x6c0 (comp_qexe)
    0x1720e: qexe_run_tcl_option(QEXE_FRAMEWORK*, char const*, std::list<std::string, std::allocator<std::string> >*, bool) + 0x5df (comp_qexe)
    0x39069: qcu_run_tcl_option(QCU_FRAMEWORK*, char const*, std::list<std::string, std::allocator<std::string> >*, bool) + 0xf20 (comp_qcu)
    0x1ab65: qexe_standard_main(QEXE_FRAMEWORK*, QEXE_OPTION_DEFINITION const**, int, char const**) + 0x486 (comp_qexe)
   0x403630: qfit2_main(int, char const**) + 0xd0 (quartus_fit)
    0x3f160: msg_main_thread(void*) + 0x10 (ccl_msg)
     0x5b4c: thr_final_wrapper + 0xc (ccl_thr)
    0x3f21f: msg_thread_wrapper(void* (*)(void*), void*) + 0x62 (ccl_msg)
     0xac5c: mem_thread_wrapper(void* (*)(void*), void*) + 0x5c (ccl_mem)
     0x8b49: err_thread_wrapper(void* (*)(void*), void*) + 0x27 (ccl_err)
     0x5b8f: thr_thread_wrapper + 0x15 (ccl_thr)
    0x41b11: msg_exe_main(int, char const**, int (*)(int, char const**)) + 0xb2 (ccl_msg)
    0x21b15: __libc_start_main + 0xf5 (c)
End-trace

I also attach the full log.

Reinstalling the stack did not solve the problem.

Does anyone have experience with this problem?

HKim27 · ‎11-01-2019

Hi, I actually solved the problem yesterday.

My compile system uses Lustre filesystem. The filesystem setup was a bit broken, and it was returning ENOSYS for fcntl(..., F_SETLK, ...) call.

After fixing the filesystem, compilation works perfectly.

Here is the script I used to debug:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
 
int main() {
  int fd = open("a.txt", O_RDWR);
  if (fd == -1) {
    printf("open error (errno=%d)\n", errno);
    exit(1);
  }
 
  struct flock lock;
  lock.l_type = F_WRLCK;
  lock.l_whence = SEEK_SET;
  lock.l_start = 0;
  lock.l_len = 1;
  int ret = fcntl(fd, F_SETLK, &lock);
  if (ret == -1) {
    printf("fcntl error (errno=%d)\n", errno);
    exit(1);
  }
 
  printf("no error\n");
 
  return 0;
}

If someone get "fcntl error (errno=38)", that's the problem.

View solution in original post

MEIYAN_L_Intel · ‎10-23-2019

Hi,

May I know that do you trying to open project which is created in a different version of Quartus(example if the project is created in sdt version and trying to open in pro version)?

May you try to open the project with an appropriate auto upgrade?(if all the IP are supported in the Quartus version, Project will be opened without and any error)

Thanks

HKim27 · ‎10-23-2019

Hi, @MeiYanL_Intel .

I use OPAE included in Intel Acceleration Stack Version 1.2, so the project is created and compiled automatically with the following script.

afu_synth_setup --source hw/rtl/sources.txt --platform discrete_pcie3 build_synth
cd build_synth
$(OPAE_PLATFORM_ROOT)/bin/run.sh

So I guess it is not version mismatch problem. (will be created and compiled in Quartus Prime Pro 17.1.1)

I opened the project with Quartus, and all IPs seem fine.

MEIYAN_L_Intel · ‎10-23-2019

Hi,

I found that the issues occurred on OPAE with Intel FPGA Basic Building Block. I am sorry that we have no support on this.

i have found that you can report the bugs on http://github.com/OPAE/intel-fpga-bbb/issues.

Thanks

HKim27 · ‎10-23-2019

Hi, @MeiYanL_Intel

How do you know the issues come from Intel FPGA BBB?

Because I should have some evidences that the issues come from Intel FPGA BBB, to report issues on their side :)

Thanks in advance.

MEIYAN_L_Intel · ‎10-24-2019

Hi,

Can you help to provide all the neccessary file (i.e Kernal *.cl files and etc) that were able to duplicate the error above? We will try in our side to see if this can be duplicated. From there, we will try to get a workaround for you if they are.

Thanks

HKim27 · ‎10-24-2019

Hi, I sent you a message.

You can compile with "make" command inside __all___workspace directory.

Note that Intel® Acceleration Stack Version 1.2 should be installed and environmental variable FPGA_BBB_CCI_SRC should point to intel-fpga-bbb. (e.g., /home/heehoon/intel-fpga-bbb)

MEIYAN_L_Intel · ‎10-24-2019

Hi,

I had received and downloaded the file. I would like to have sometime to setup the environment.

Thanks

MEIYAN_L_Intel · ‎11-01-2019

Hi,

I am still on progressing for some setup.

Thanks

HKim27 · ‎11-01-2019

Hi, I actually solved the problem yesterday.

My compile system uses Lustre filesystem. The filesystem setup was a bit broken, and it was returning ENOSYS for fcntl(..., F_SETLK, ...) call.

After fixing the filesystem, compilation works perfectly.

Here is the script I used to debug:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
 
int main() {
  int fd = open("a.txt", O_RDWR);
  if (fd == -1) {
    printf("open error (errno=%d)\n", errno);
    exit(1);
  }
 
  struct flock lock;
  lock.l_type = F_WRLCK;
  lock.l_whence = SEEK_SET;
  lock.l_start = 0;
  lock.l_len = 1;
  int ret = fcntl(fd, F_SETLK, &lock);
  if (ret == -1) {
    printf("fcntl error (errno=%d)\n", errno);
    exit(1);
  }
 
  printf("no error\n");
 
  return 0;
}

If someone get "fcntl error (errno=38)", that's the problem.

MEIYAN_L_Intel · ‎11-12-2019

Hi,

Thanks for the update.