FPGA Builds Broken on DevCloud

[Christoph9]

Hello,

since today I encouter error when compiling for the Arria10 PAC. The build runs fine at the beginning, but crashes near the end of it.

The .o file of the job gives no real hints what's going wrong:

aoc: Compiling for FPGA. This process may take several hours to complete.  Prior to performing this compile, be sure to check the reports to ensure the design will meet your performance targets.  If the reports indicate performance targets are not being met, code edits may be required.  Please refer to the oneAPI FPGA Optimization Guide for information on performance tuning applications for FPGAs.
Error (23035): Tcl error: 
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings

The .e file neither:

For more details, full Quartus compile output can be found in files quartuserr.tmp and quartus_sh_compile.log.
Error: Compiler Error, not able to generate hardware

llvm-foreach: 
icpx: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
make[3]: *** [cuda/level2/particlefilter/CMakeFiles/particlefilterLib.fpga.dir/build.make:100: cuda/level2/particlefilter/particlefilterLib.fpga] Error 1
make[2]: *** [CMakeFiles/Makefile2:3295: cuda/level2/particlefilter/CMakeFiles/particlefilterLib.fpga.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:2951: cuda/level2/particlefilter/CMakeFiles/particlefilterLib_fpga.dir/rule] Error 2
make: *** [Makefile:1236: particlefilterLib_fpga] Error 2

Here a short snippet of the quartus log (complete file as attachement):

Info: Quartus Prime Shell was successful. 0 errors, 237 warnings
    Info: Peak virtual memory: 3474 megabytes
    Info: Processing ended: Sat Dec 17 05:41:47 2022
    Info: Elapsed time: 00:01:38
run.sh: PLL_METADATA/PLL_METADATA_FILE is: 
clock-frequency-low:195 clock-frequency-high:390
ERROR: [Errno 2] No such file or directory: './output_files/afu_import.green_region.rbf'
ERROR: packager tool failed to create .gbs file.
Info: *******************************************************************
Info: Running Quartus Prime Shell
    Info: Version 19.2.0 Build 57 06/24/2019 SJ Pro Edition
    Info: Copyright (C) 2019  Intel Corporation. All rights reserved.
    Info: Your use of Intel Corporation's design tools, logic functions 
    Info: and other software and tools, and any partner logic 
    Info: functions, and any output files from any of the foregoing 
    Info: (including device programming or simulation files), and any 
    Info: associated documentation or information are expressly subject 
    Info: to the terms and conditions of the Intel Program License 
    Info: Subscription Agreement, the Intel Quartus Prime License Agreement,
    Info: the Intel FPGA IP License Agreement, or other applicable license
    Info: agreement, including, without limitation, that your use is for
    Info: the sole purpose of programming logic devices manufactured by
    Info: Intel and sold by Intel or its authorized distributors.  Please
    Info: refer to the applicable agreement for further details, at
    Info: https://fpgasoftware.intel.com/eula.
    Info: Processing started: Sat Dec 17 04:36:27 2022
Info: Command: quartus_sh -t build/entry.tcl import
Info: Quartus(args): import
Info: Using INI file /home/u153009/tmp/main-4a28c3-d5e0be/quartus.ini
Info: Compiling revision import
Error (23035): Tcl error: 
    while executing
"qexec "bash build/run.sh $revision_name""
    ("default" arm line 9)
    invoked from within
"switch $tcl_platform(platform) {
  windows {
    post_message -type error "Full compiles to generate hardware for the FPGA are available on supported ..."
    (file "build/entry.tcl" line 36)
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings
    Error: Peak virtual memory: 979 megabytes
    Error: Processing ended: Sat Dec 17 05:41:49 2022
    Error: Elapsed time: 01:05:22

Is something wrong with the Quartus-Setup on IntelDevcloud?

Best regards and thanks in advance for your help,
Christoph

[filipeborralho]

Hi @AlekhyaV_Intel ,

For me it still isn't working. I tried compiling some code and I get a similar error to before except now I had to take the QUARTUS_ROOTDIR_OVERRIDE part out of my script since the folder where quartus was seems to not exist anymore. Before I would have something like this:

u162531@s001-n066:~/thesis/fpga_measurements$ QUARTUS_ROOTDIR_OVERRIDE=/opt/intel/intelFPGA_pro/19.2/quartus make fpga_hw

Now I have this:

u162531@s001-n066:~/thesis/fpga_measurements$ make fpga_hw
icpx -fsycl -fintelfpga -Xsboard=/opt/intel/oneapi/intel_a10gx_pac:pac_a10 -Xshardware -Xsprofile -std=c++20 fma.cpp
aoc: Compiling for FPGA. This process may take several hours to complete.  Prior to performing this compile, be sure to check the reports to ensure the design will meet your performance targets.  If the reports indicate performance targets are not being met, code edits may be required.  Please refer to the oneAPI FPGA Optimization Guide for information on performance tuning applications for FPGAs.
Error (23035): Tcl error: 
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings
For more details, full Quartus compile output can be found in files quartuserr.tmp and quartus_sh_compile.log.
Error: Compiler Error, not able to generate hardware

llvm-foreach: 
icpx: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:12: fpga_hw] Error 1

I'll attach the compile log as well as the error file. I also tried changing from -Xsboard to -Xstarget as it seems to be how it's done in the samples but to no change, still the same issue.

Best regards,

Filipe

[robertszafa]

Hi @AlekhyaV_Intel,

Thank you for your support in getting this solved.

I'm afraid I'm still facing the same issue. I get the following error on all fpga_compile nodes: "Full compiles to generate hardware for the FPGA are available on supported ..."

Maybe there is some configuration issue on these nodes since the last SYCL compiler update?

Cheers,

Rob

[AlekhyaV_Intel]

Hi @filipeborralho & @robertszafa ,

Could you please provide us the sample reproducer(the project you're working on, steps to build that sample project and which nodes you tried on). If you tried any of our oneAPI samples, please let us know the names of those fpga samples so that we can try cross-checking those once again.

Regards,

Alekhya

[AlekhyaV_Intel]

Hi @Christoph9 ,

Has the issue resolved? Could you please give us an update regarding this issue?

Regards,

Alekhya

[AlekhyaV_Intel]

Hi christoph,

We assume that your issue is resolved. If you need any further assistance, please post a new question as this thread will no longer be monitored by Intel.

Regards,

Alekhya

[VR2016]

Hi, 

I'm compiling my oneAPI design for S10 PAC, I'm running into same issue. Looks like this is not fixed yet. 

Build command: qsub -l nodes=1:fpga_compile:ppn=2 -d . -l walltime=24:00:00 build_fpga_hw.sh

dpcpp -O2 -fintelfpga -fsycl-link=image -Isrc -Isrc/engine -isystem lib/yaml-cpp/include src/engine/engine_kernel.cpp -o pac_s10.a -Xshardware -Xsboard=/opt/intel/oneapi/intel_s10sx_pac:pac_s10 -Xsbsp-flow=flat -DUSE_REG -DUSE_DEVICE_PTRS

Let me know how to get around this issue.

Thanks,

-VR

This is the PAC OpenCL BSP run.sh script.
Compiling flat revision flow...
ERROR: packager check failed with output ''
Info: *******************************************************************
Info: Running Quartus Prime Shell
Info: Version 19.2.0 Build 57 06/24/2019 Patches 0.05dcp SJ Pro Edition
Info: Copyright (C) 2019 Intel Corporation. All rights reserved.
Info: Your use of Intel Corporation's design tools, logic functions
Info: and other software and tools, and any partner logic
Info: functions, and any output files from any of the foregoing
Info: (including device programming or simulation files), and any
Info: associated documentation or information are expressly subject
Info: to the terms and conditions of the Intel Program License
Info: Subscription Agreement, the Intel Quartus Prime License Agreement,
Info: the Intel FPGA IP License Agreement, or other applicable license
Info: agreement, including, without limitation, that your use is for
Info: the sole purpose of programming logic devices manufactured by
Info: Intel and sold by Intel or its authorized distributors. Please
Info: refer to the applicable agreement for further details, at
Info: https://fpgasoftware.intel.com/eula.
Info: Processing started: Wed Apr 19 20:13:30 2023
Info: Command: quartus_sh -t build/entry.tcl flat
Info: Quartus(args): flat
Info: Using INI file /home/u191491/tmp/engine_kernel-a4dccf-b62ff1/quartus.ini
Info: Compiling revision flat
Error (23035): Tcl error:
while executing
"qexec "bash build/run.sh $revision_name""
("default" arm line 9)
invoked from within
"switch $tcl_platform(platform) {
windows {
post_message -type error "Full compiles to generate hardware for the FPGA are available on supported ..."
(file "build/entry.tcl" line 19)
Error (23031): Evaluation of Tcl script build/entry.tcl unsuccessful
Error: Quartus Prime Shell was unsuccessful. 2 errors, 0 warnings
Error: Peak virtual memory: 979 megabytes
Error: Processing ended: Wed Apr 19 20:13:31 2023
Error: Elapsed time: 00:00:01

===========

# (c) 1992-2020 Intel Corporation.
# Intel, the Intel logo, Intel, MegaCore, NIOS II, Quartus and TalkBack words
# and logos are trademarks of Intel Corporation or its subsidiaries in the U.S.
# and/or other countries. Other marks and brands may be claimed as the property
# of others. See Trademarks on intel.com for full list of Intel trademarks or
# the Trademarks & Brands Names Database (if Intel) or See www.Intel.com/legal (if Altera)
# Your use of Intel Corporation's design tools, logic functions and other
# software and tools, and its AMPP partner logic functions, and any output
# files any of the foregoing (including device programming or simulation
# files), and any associated documentation or information are expressly subject
# to the terms and conditions of the Altera Program License Subscription
# Agreement, Intel MegaCore Function License Agreement, or other applicable
# license agreement, including, without limitation, that your use is for the
# sole purpose of programming logic devices manufactured by Intel and sold by
# Intel or its authorized distributors. Please refer to the applicable
# agreement for further details.

#Full compiles are not supported on Windows
switch $tcl_platform(platform) {
windows {
post_message -type error "Full compiles to generate hardware for the FPGA are available on supported Linux platforms only. Please add -rtl to your invocation of aoc to compile without building hardware. Otherwise please run your compile on a supported Linux distribution."
exit 2
}
default {
#Get revision name from quartus args
if { [llength $quartus(args)] > 0 } {
set revision_name [lindex $quartus(args) 0]
} else {
set revision_name import
}
post_message "Compiling revision $revision_name"
qexec "bash build/run.sh $revision_name"
}
}

[Chithambarash]

Hi,

I'm trying to compile and execute the given vector_add code in the examples. 

The compilation seems to be working fine and no issues have been faced. However, when I try to execute on Arria10 PAC FPGA node, I'm facing some issues on quartus version.