DevCloud: Cannot compile Stratix 10 kernel (Tcl error)

[Mdcc_UVa]

Hello.

I used to be able to compile my OpenCL kernel targetting Stratix 10 PACs on the DevCloud without any problem. That was before you had to source init_env.sh or any other bash scripts at the beginning of a compilation job.

Now, however, whenever I try to compile the very same kernel for Stratix 10 PACs, the compilation process reports a Tcl error and aborts. It is specially annoying since this usually happens about 1 hour into compilation.

Here is the compilation script, built based on the documentation found here: https://devcloud.intel.com/oneapi/documentation/job-submission/

#/bin/bash

source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/init_env.sh
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/init_opencl.sh
export FPGA_BBB_CCI_src=/usr/local/intel-fpga-bbb
export PATH=/glob/intel-python/python2/bin:${PATH}

aoc -v -report -march=emulator mykernel.cl -o mykernel_emu.aocx
aoc -v -report -board-package=/glob/development-tools/versions/oneapi/2022.3.1/oneapi/intel_s10sx_pac mykernel.cl -o mykernel.aocx

 To compile the kernel, I create the following job:

qsub -l nodes=1:fpga_compile:ppn=2 -d . compile.sh

 Here is the stdout of the job:

# For the emulation compilation
aoc: Environment checks are completed successfully.
aoc: Cached files in /var/tmp/aocl/ may be used to reduce compilation time
You are now compiling the full flow!!
aoc: Selected default target board pac_s10_dc

# For the FPGA binary compilation
aoc: Environment checks are completed successfully.
aoc: Cached files in /var/tmp/aocl/ may be used to reduce compilation time
You are now compiling the full flow!!
aoc: Selected default target board pac_s10
aoc: Running OpenCL parser....
aoc: OpenCL parser completed successfully.
aoc: Linking Object files....
aoc: Optimizing and doing static analysis of code...
aoc: Linking with IP library ...
aoc: Checking if memory usage is larger than 100%...
aoc: Memory usage is not above 100.
# Trimmed for presentation purposes
aoc: First stage compilation completed successfully.
Compiling for FPGA. This process may take a long time, please be patient.
Error (23035): Tcl error:
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings

And here is the stderr:

# For the emulation compilation
Error: OpenCL kernel compilation FAILED

# For the FPGA binary compilation
Error: Compiler Error, not able to generate hardware

As you can see, I am doing everything as it is said in the linked documentation. I am using the latest BSP I could find, and the default values for Quartus compilation. I don't even know what Tcl is, let alone how to edit the Tcl scripts.

Can somebody help me understand what is going on?

Thanks.

[hareesh]

Hi,

i am working on this issue. just give me some time.

Thank you,

[hareesh]

Hi,

ca pls share your TCL script file?

[Mdcc_UVa]

Hi. Thank you for your efforts.

As stated in the last paragraph of my post, I basically don't know anything about TCL scripts besides what is said in the error message. I don't know how they work (thus neither how to edit them), and I don't even know where they are located. I am assuming I am using some kind of "default script" for the Quartus/OpenCL installation in DevCloud.

If you can point me to the location in the DevCloud system of that "default script", I will be able to share it with you. Nevertheless, it should not differ from everybody else's script.

[Mdcc_UVa]

Hello again.

Upon further inspection, this error also applies to the compilation of OpenCL kernels for Arria 10 devices, not only Stratix 10.

This is the compilation script for the Stratix 10 kernel:

#/bin/bash

source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/init_env.sh
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/init_opencl.sh
export FPGA_BBB_CCI_src=/usr/local/intel-fpga-bbb
export PATH=/glob/intel-python/python2/bin:${PATH}

aoc -v -report -board-package=/opt/intel/oneapi/intel_s10sx_pac mykernel.cl -o mykernel.aocx

This is the compilation script for the Arria 10 kernel:

#/bin/bash

source /glob/development-tools/versions/fpgasupportstack/a10/1.2.1/inteldevstack/init_env.sh
source /glob/development-tools/versions/fpgasupportstack/a10/1.2.1/intelFPGA_pro/hld/init_opencl.sh
export FPGA_BBB_CCI_src=/usr/local/intel-fpga-bbb
export PATH=/glob/intel-python/python2/bin:${PATH}

aoc -v -report -board-package=/opt/intel/oneapi/intel_a10gx_pac mykernel.cl -o mykernel.aocx

Both scripts were made based on the corresponding documentation on this page:  https://devcloud.intel.com/oneapi/documentation/job-submission/

Here's the error for a Stratix 10 compilation:

aoc: First stage compilation completed successfully.
Compiling for FPGA. This process may take a long time, please be patient.
Error (16556): The synthesis RTL for ip/ddr_board/ddr_board_acl_memory_bank_divider_1.ip has not been generated. Generate the synthesis RTL from within Platform Designer.
Error: Flow failed:
Error: Quartus Prime Synthesis was unsuccessful. 2 errors, 357 warnings
Error (23035): Tcl error:
Error (23031): Evaluation of Tcl script compile_script.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings
Error (23035): Tcl error:
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings

 Here's the error for an Arria 10 compilation:

aoc: First stage compilation completed successfully.
aoc: Compiling for FPGA. This process may take several hours to complete.  Prior to performing this compile, be sure to check the reports to ensure the design will meet your performance targets.  If the reports indicate performance targets are not being met, code edits may be required.  Please refer to the Intel FPGA SDK for OpenCL Best Practices Guide for information on performance tuning applications for FPGAs.
Error (19273): Family Stratix 10 is not installed
Error: Flow failed: ERROR: Current design not found
Error: Quartus Prime Synthesis was unsuccessful. 2 errors, 1 warning
Error (23035): Tcl error:
Error (23031): Evaluation of Tcl script compile_script.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings
Error (23035): Tcl error:
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings

(Notice how the Arria 10 error  references the "Family Stratix 10", although the specified BSP for compilation was /opt/intel/oneapi/intel_a10gx_pac).

Both errors reference the TCL scripts compile_script.tcl and build/entry.tcl, used automatically by the Quartus compiler. However, I don't know which is the absolute path to those scripts, so I cannot share them with you. If someone provides me with the absolute path to them, I will be able to share them here.

[hareesh]

Hi,

Still are you facing problem?

[Mdcc_UVa]

Hello.

I cannot confirm yet that the problem is solved. I am currently busy and I have been unable to test it further. If you could kindly wait a little longer, I will get back to you as soon as possible with any update I have.

Thanks.

[hareesh]

hi,

any update on your issue?

[hareesh]

As we do not receive any response from you on the previous question/reply/answer that we have provided. Please login to ‘https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

[hareesh]

hi,

what is the status of your issue?

[hareesh]

please update status of your issue. based on that i will get an idea.

[Mdcc_UVa]

Hello. Sorry for the delay. Some of the compilation times are quite high, and the time I can dedicate to this daily is limited.

First off, as suggested, I followed this guide: https://devcloud.intel.com/oneapi/documentation/job-submission/

I created the following job submission script, compile.sh:

#/bin/bash

source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/init_env.sh
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/init_opencl.sh
export FPGA_BBB_CCI_src=/usr/local/intel-fpga-bbb
export PATH=/glob/intel-python/python2/bin:${PATH}

time aoc -v -report -march=emulator mykernel.cl -o mykernel_emu.aocx
time aoc -v -report -board-package=/opt/intel/oneapi/intel_s10sx_pac mykernel.cl -o mykernel.aocx

I submit it to a Stratix 10 compilation node using the following command:

qsub -l nodes=1:fpga_compile:ppn=2 -d . compile.sh

The following is the output (stdout) for the job, compile.sh.o2114020. The output isn't trimmed in any way, so you have access to the full information. Note the error messages at the end.

########################################################################
#      Date:           Wed 04 Jan 2023 01:23:43 AM PST
#    Job ID:           2114020.v-qsvr-1.aidevcloud
#      User:           u177524
# Resources:           cput=75:00:00,neednodes=1:fpga_compile:ppn=2,nodes=1:fpga_compile:ppn=2,walltime=06:00:00
########################################################################

export QUARTUS_HOME=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/quartus
export OPAE_PLATFORM_ROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237
export AOCL_BOARD_PACKAGE_ROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp
Adding $OPAE_PLATFORM_ROOT/bin to PATH
export INTELFPGAOCLSDKROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld
export ALTERAOCLSDKROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld
Adding $QUARTUS_HOME/bin to PATH
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/init_opencl.sh
INTELFPGAOCLSDKROOT is set to /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld. Using that.

Will use $QUARTUS_ROOTDIR_OVERRIDE= /opt/intel/intelFPGA_pro/19.2/quartus  to find Quartus

AOCL_BOARD_PACKAGE_ROOT is set to /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp. Using that.
Adding /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/bin to PATH
Adding /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/linux64/lib to LD_LIBRARY_PATH
Adding /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/host/linux64/lib to LD_LIBRARY_PATH
Adding /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp/linux64/lib to LD_LIBRARY_PATH
aoc: Environment checks are completed successfully.
aoc: Cached files in /var/tmp/aocl/ may be used to reduce compilation time
You are now compiling the full flow!!
aoc: Selected default target board pac_s10_dc
aoc: Environment checks are completed successfully.
aoc: Cached files in /var/tmp/aocl/ may be used to reduce compilation time
You are now compiling the full flow!!
aoc: Selected default target board pac_s10
aoc: Running OpenCL parser....
aoc: OpenCL parser completed successfully.
aoc: Linking Object files....
aoc: Optimizing and doing static analysis of code...
aoc: Linking with IP library ...
aoc: Checking if memory usage is larger than 100%...
aoc: Memory usage is not above 100.

!===========================================================================
! The report below may be inaccurate. A more comprehensive           
! resource usage report can be found at mykernel/reports/report.html    
!===========================================================================

+--------------------------------------------------------------------+
; Estimated Resource Usage Summary                                   ;
+----------------------------------------+---------------------------+
; Resource                               + Usage                     ;
+----------------------------------------+---------------------------+
; Logic utilization                      ;   59%                     ;
; ALUTs                                  ;   32%                     ;
; Dedicated logic registers              ;   29%                     ;
; Memory blocks                          ;   35%                     ;
; DSP blocks                             ;   22%                     ;
+----------------------------------------+---------------------------;
aoc: First stage compilation completed successfully.
Compiling for FPGA. This process may take a long time, please be patient.
Error (16556): The synthesis RTL for ip/ddr_board/ddr_board_acl_memory_bank_divider_1.ip has not been generated. Generate the synthesis RTL from within Platform Designer.
Error: Flow failed: 
Error: Quartus Prime Synthesis was unsuccessful. 2 errors, 357 warnings
Error (23035): Tcl error: 
Error (23031): Evaluation of Tcl script compile_script.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings
Error (23035): Tcl error: 
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings

########################################################################
# End of output for job 2114020.v-qsvr-1.aidevcloud
# Date: Wed 04 Jan 2023 02:34:58 AM PST
########################################################################

DISCLAIMER: I do not know anything about Tcl scripts. I followed the process described in the official documents/guides for job submission and OpenCL FPGA kernel compilation, which never refer to Tcl scripts. I never opened or modified in any way a Tcl script, and I do not even know where they are located in the system.

The following is the error output (stderr) for the job, compile.sh.e2114020. The output isn't trimmed in any way, so you have access to the full information.  

Error: OpenCL kernel compilation FAILED

real	0m10.103s
user	0m1.544s
sys	0m1.244s
Error: Compiler Error, not able to generate hardware


real	70m58.990s
user	75m20.593s
sys	0m56.989s

The first error message corresponds to the compilation using -march=emuator (i.e., compiling for emulation fails). The second error, which happens past the hour of compilation, corresponds to the compilation targeting a Stratix 10 PAC (i.e., compiling for a real FPGA fails).

After the job finished, the following files are generated: mykernel.aoco, mykernel.aocr, but not mykernel.aocx.

That's what happens when I follow the Job Submission guide. I do not know what the problems are exactly nor how to fix them.

[Mdcc_UVa]

Just out of curiosity, I also tried not following the official guide, and trying to compile the kernel as I used to be able to compile it. I read in some other forum post that setting a specific value for the QUARTUS_ROOTDIR_OVERRIDE environmental variable and using a specific aoc binary for compilation helped fixing some issues with the compilation process.

Thus, I created a new job submission script for the compilation of the OpenCL kernel, x.sh:

#!/bin/bash

QUARTUS_ROOTDIR_OVERRIDE=/opt/intel/intelFPGA_pro/19.2/quartus /glob/development-tools/versions/oneapi/2022.2/oneapi/compiler/2022.1.0/linux/lib/oclfpga/bin/aoc -board-package=/opt/intel/oneapi/intel_s10sx_pac mykernel.cl -o mykernel.aocx

The job is submitted using the same command:

qsub -l nodes=1:fpga_compile:ppn=2 -d . x.sh

Executing this job produces no error output (stderr), and produces a normal, error-less output (stdout):

########################################################################
#      Date:           Wed 04 Jan 2023 01:24:22 AM PST
#    Job ID:           2114025.v-qsvr-1.aidevcloud
#      User:           u177524
# Resources:           cput=75:00:00,neednodes=1:fpga_compile:ppn=2,nodes=1:fpga_compile:ppn=2,walltime=06:00:00
########################################################################

aoc: Running OpenCL parser....
aoc: OpenCL parser completed 
aoc: Linking Object files....
aoc: Optimizing and doing static analysis of code...
aoc: First stage compilation completed successfully.
aoc: Compiling for FPGA. This process may take several hours to complete.  Prior to performing this compile, be sure to check the reports to ensure the design will meet your performance targets.  If the reports indicate performance targets are not being met, code edits may be required.  Please refer to the Intel FPGA SDK for OpenCL Best Practices Guide for information on performance tuning applications for FPGAs.

########################################################################
# End of output for job 2114025.v-qsvr-1.aidevcloud
# Date: Wed 04 Jan 2023 04:36:10 AM PST
########################################################################

This job does produce a mykernel.aocx binary.

I join a FPGA execution node using the following command:

qsub -I -l nodes=1:fpga_runtime:stratix10:ppn=2 -d .

Which assigns me the node s001-n142. Then I execute the OpenCL program that uses this kernel.

This execution works completely fine. Other times I tested it, it seemed to not work. If I find another of those cases again, I will report it as soon as possible.

However, I am worried that the only way for my program to work/be properly compiled is to not to follow the Job Submission guide for OpenCL FPGA kernels. I do not know if that is intended, or something is wrong regarding OpenCL FPGA kernel compilation.

[hareesh]

Hi,

i think this post will helps you. pls once check this.

https://community.intel.com/t5/Intel-DevCloud/Error-on-devcloud-fpga-compile-nodes/m-p/1430277

[Mdcc_UVa]

Hi again.

The issue reported in that post is unrelated to my issue, since my environmental variable QUARTUS_ROOTDIR_OVERRIDEQUARTUS_ROOTDIR_OVERRIDE is not empty.

As I reported in my previous answer, I was able to find a workaround to my problem by using aoc version 2022.1.0 instead of the latest version (or any version later than that). However, as of recently, I can no longer use that version of the compiler since accessing the containing directory on DevCloud now requires special permissions. Thus, I can not longer use the workaround and I am unable to compile OpenCL kernels to FPGA again.

Having compiled a kernel using an earlier, now inaccessible version of the compiler makes me think that it is a compiler issue/bug in recent versions of the compiler. However I can no longer test this hypothesis as I can no longer use the seemingly working version of the compiler.

[hareesh]

hi,

any update about your issue?

in previous comment i shared one case. that is same has your issue please follow that

[hareesh]

please update after tried that process

[hareesh]

As we do not receive any response from you on the previous question/reply/answer that we have provided. Please login to ‘https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

[Mdcc_UVa · Accepted solution]

This issue was not solved.

However, there is another post discussing the same issue, which has been acknowledged and reproduced by Intel staff. A solutions is supposedly being worked on, and updates will be posted in that post.

Thus, discussion on this issue has been moved to that post:

https://community.intel.com/t5/Intel-DevCloud/FPGA-Builds-Broken-on-DevCloud/m-p/1439447