Build Script for Batch does not execute make_fpga command

[bxbxf]

I have the folllowing script (build.sh):

#!/bin/bash
cd path/to/project;
rm -r build
mkdir build
cd build
cmake .. -DFPGA_DEVICE=intel_a10gx_pac:pac_a10;
make fpga

I submit this via 

devcloud_login -b A10OAPI build.sh

It only prints out the following : 

########################################################################
#      Date:           Tue 06 Jun 2023 06:04:47 AM PDT
#    Job ID:           2313242.v-qsvr-1.aidevcloud
# Resources:           cput=75:00:00,neednodes=1:batch:ppn=2,nodes=1:batch:ppn=2,walltime=08:00:00
########################################################################

-- The CXX compiler identification is Clang 16.0.0
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/bin/icpx
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/bin/icpx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring the design to run on FPGA device intel_a10gx_pac:pac_a10
-- Configuring the design to run on FPGA device intel_a10gx_pac:pac_a10
-- Configuring the design to run on FPGA device intel_a10gx_pac:pac_a10
-- Configuring the design to run on FPGA device intel_a10gx_pac:pac_a10
-- Configuring the design to run on FPGA device intel_a10gx_pac:pac_a10
-- Configuring done
-- Generating done
-- Build files have been written to: path/to/build

########################################################################
# End of output for job 2313242.v-qsvr-1.aidevcloud
# Date: Tue 06 Jun 2023 06:05:01 AM PDT
########################################################################

[bxbxf]

Addionally, when I interactively execute the `make fpga` command it executes but it does not with batch. 

Moreover, same happends when I try to run the generated executables.

I also tried the following command 

qsub -l nodes=1:fpga_runtime:arria10:ppn=2 -d . run.sh

This also ends after 10s and gives a blank output and blank error output file :

########################################################################
# Date: Wed Jun 7 05:01:23 PDT 2023
# Job ID: 2314056.v-qsvr-1.aidevcloud
# Resources: cput=75:00:00,neednodes=1:fpga_runtime:arria10:ppn=2,nodes=1:fpga_runtime:arria10:ppn=2,walltime=06:00:00
########################################################################

########################################################################
# End of output for job 2314056.v-qsvr-1.aidevcloud
# Date: Wed Jun 7 05:01:31 PDT 2023
########################################################################

[JyothisV_Intel]

Hi,

Good day to you.

Thanks for posting in Intel Communities.

We tried replicating the issue reported from our side on the official Intel oneAPI FPGA sample documented at the link: https://github.com/oneapi-src/oneAPI-samples/tree/master/DirectProgramming/C%2B%2BSYCL_FPGA/Tutorials/GettingStarted/fast_recompile

The steps we followed:

1. Run the following command in login node:

source /data/intel_fpga/devcloudLoginToolSetup.sh .

2. Created a script build.sh with the following contents in it as created by you (we added a line to run the created samples at the end):

#!/bin/bash
cd ~/FPGA_ISSUE/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/GettingStarted/fast_recompile/
rm -r build
mkdir build
cd build
cmake .. -DFPGA_DEVICE=intel_a10gx_pac:pac_a10;
make fpga
./fast_recompile.fpga

3. Next, we used the following command to submit the job to the Intel Arria10 oneAPI node:

devcloud_login -b A10OAPI build.sh

The generated outputs and errors of the same is attached as a screenshot, we could not find any issues with the building and execution process when submitted using the batch mode.

Next, we tried the same script in interactive mode to confirm if it has issues with interactive method of building and execution.
The screenshot for the same is attached along. We were unable to replicate it in interactive mode also.

To assist you further, can you let us get back to us with the following information:

1. A sample reproducer code/repository that we can use to replicate the issue from our side
2. The node in which you tried and observed this issue in both batch and interactive mode of job submission (this can be printed by adding a hostname command to the above script)

Thanks and Regards,

Jyothis V James

[bxbxf]

Hi,
After you verified that it was working for you, I have reverted some changes that I have done with the makefile and now it builds

EDIT : No, it didn't work when I try to run 

I am working on ac_int example from oneAPI samples

#!/bin/bash

cd ~/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/

./build/ac_int.fpga

it again printed blank output

[bxbxf]

It prints output when I run interactive session but not on batch

```

devcloud_login -b SNN s001-n084 execute.sh

``` 

It works on interactive session which is interesting

qsub: waiting for job 2314948.v-qsvr-1.aidevcloud to start
qsub: job 2314948.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Thu Jun  8 04:43:40 PDT 2023
#    Job ID:           2314948.v-qsvr-1.aidevcloud
# Resources:           cput=75:00:00,neednodes=s001-n084:ppn=2,nodes=s001-n084:ppn=2,walltime=06:00:00
########################################################################

$ cd ~/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/
$ cd build
$ ./ac_int.fpga
Running on device: pac_a10 : Intel PAC Platform (pac_ee00000)
PASSED: all kernel results are correct.

[bxbxf]

If it helps, I cannot run emulation on the login nodes. It gives me the following error message : 

[100%] Linking CXX executable ../bfs.fpga_emu
OpenCL platform ID is empty
OpenCL platform name is empty
Failed to find any of these OpenCL platforms:
  Intel(R) FPGA Emulation Platform for OpenCL(TM)
  Intel(R) FPGA

[JyothisV_Intel]

Hi,

Good day to you.

Thanks for getting back to us with more information.

We tried batch mode of submission using the below procedure for the ac_int sample and we are unable to replicate the issue from our side again:

1. Run the following command in login node:

source /data/intel_fpga/devcloudLoginToolSetup.sh .

2. Created a script build.sh with the following contents in it:

#!/bin/bash
cd ~/FPGA_ISSUE/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/
rm -r build
mkdir build
cd build
cmake .. -DFPGA_DEVICE=intel_a10gx_pac:pac_a10;
make fpga

3. Created another script execute.sh with the following contents in it:

cd ~/FPGA_ISSUE/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/
./build/ac_int.fpga

4. Submit the build script to Intel oneAPI Arria10 nodes using the below command:

devcloud_login -b A10OAPI build.sh

(Screenshot of the output file build.sh.o<job_id> is attached)

5. Once the compilation is complete, submit the execute script to Intel oneAPI Arria10 nodes using the below command:

devcloud_login -b A10OAPI execute.sh

(Screenshot of the output file execute.sh.o<job_id> is attached)

Can you confirm if you are also following the same procedure as mentioned above for building and execution? Do get back to us if you are still facing issues.

Regarding your query related to FPGA emulation, login node should not be used to perform any form of building or execution of any programs. It is a shared system with limited resources intended to be used to submit and manage jobs.

If you wish to perform FPGA emulation, we recommend using Intel oneAPI FPGA nodes which has dedicated FPGA emulation device for this purpose. This can be done via both batch mode and interactive mode.

You can confirm if a node has FPGA emulation device by running the command sycl-ls in the node (interactive mode) or inside script (batch mode). The output of the above command should contain the following highlighted device for FPGA emulation to be successful:

$ sycl-ls

[opencl:cpu:0] Intel(R) OpenCL, Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz 3.0 [2023.15.3.0.20_160000]

[opencl:acc:1] Intel(R) FPGA SDK for OpenCL(TM), pac_a10 : Intel PAC Platform (pac_ee00000) 1.0 [2023.1]

[opencl:acc:2] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2023.15.3.0.20_160000]

For example, FPGA emulation for the ac_int sample can be performed using batch mode similar to the procedure mentioned above making the below changes. Kindly note that FPGA emulation does not require DFPGA_DEVICE parameter for cmake, therefore the build.sh script should be updated as shown below:

#!/bin/bash
cd ~/FPGA_ISSUE/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/
rm -r build
mkdir build
cd build
cmake ..
make fpga_emu

You can use the devcloud_login -b A10OAPI <script_name>.sh to submit the emulation build and execution scripts in batch mode.
(See attached screenshot for reference)

If this resolves your query, kindly accept this as a solution as it will help others with a similar query. If not, do get back to us so that we can assist you better.

Thanks and Regards,

Jyothis V James

[bxbxf]

Hi,
It is the same steps I am also following and the batch mode still does not work. Shall I create a new account and try on that one ? 

Also, thank you for detailed explanation! Appreciated

[bxbxf]

Hi again, 
I have created a new account and git cloned the oneAPI-samples. I still have the same issue. Batch submission does not execute the ./ac_int.fpga

Build.sh:

#!/bin/bash
cd ~/playground/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int
rm -r build
mkdir build
cd build
cmake .. -DFPGA_DEVICE=intel_a10gx_pac:pac_a10;
make fpga
./ac_int.fpga

build.sh.oXXXXX

########################################################################
#      Date:           Fri Jun  9 03:32:44 PDT 2023
#    Job ID:           2316037.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           cput=75:00:00,neednodes=s001-n082:ppn=2,nodes=s001-n082:ppn=2,walltime=06:00:00
########################################################################

-- The CXX compiler identification is Clang 16.0.0
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/bin/icpx
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/bin/icpx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring the design with the following target: intel_a10gx_pac:pac_a10
-- Configuring done
-- Generating done
-- Build files have been written to: /home/uXXXXXX/playground/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/build
Scanning dependencies of target ac_int.fpga
[ 50%] Building CXX object src/CMakeFiles/ac_int.fpga.dir/ac_int.cpp.o
[100%] Linking CXX executable ../ac_int.fpga
warning: -reuse-exe file '/home/uXXXXXX/playground/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/build/ac_int.fpga' not found; ignored
aoc: Compiling for FPGA. This process may take several hours to complete.  Prior to performing this compile, be sure to check the reports to ensure the design will meet your performance targets.  If the reports indicate performance targets are not being met, code edits may be required.  Please refer to the oneAPI FPGA Optimization Guide for information on performance tuning applications for FPGAs.
[100%] Built target ac_int.fpga
Scanning dependencies of target fpga
[100%] Built target fpga

########################################################################
# End of output for job 2316037.v-qsvr-1.aidevcloud
# Date: Fri Jun  9 04:31:12 PDT 2023
########################################################################

[Christoph9 · Accepted solution]

Just to be sure: Insert a new, empty line in your job script. I had some problems where the last line was not parsed when missing a terminating character.

I worked previously with the Arria10 FPGA half a year ago and I did not need any of the additional commands you shown in the post below. You can execute "sycl-ls" in the terminal on all DevCloud nodes. On the FPGA nodes, you should see the FPGA emulator and the corresponding accelerator listed there. If there is no FPGA listed the node is prob misconfigured or the FPGA is dead, I encountered previously some nodes that contain dead FPGAs, especially when targeting the Stratix10 nodes. In this case just try to use a different node, you can list all nodes with Arria10 FPGAs using pbsnodes | sort | grep arria10.

[bxbxf]

Hi Christoph9, 
I remember, you also opened topics when we had issues with Arria10 nodes all down 

I can't thank you enough!!!!!!! I really appreciate your help. You were right, the end of line thing was the issue when I modified my execute.sh script to the following : 

#!/bin/bash
cd ~/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int
cd build
./ac_int.fpga
touch tempfile.txt

It did end up running the command but not creating the temp file so you were right there needs to be an extra line 

########################################################################

Running on device: pac_a10 : Intel PAC Platform (pac_ee00000)
PASSED: all kernel results are correct.

########################################################################

Thank you very much!!!
Have a nice day/night
Best wishes

[bxbxf]

According to tutorial here : https://devcloud.intel.com/oneapi/documentation/job-submission/

I added the following lines : 

#!/bin/bash
source /glob/development-tools/versions/fpgasupportstack/a10/1.2.1/intelFPGA_pro/hld/init_opencl.sh
source /glob/development-tools/versions/fpgasupportstack/a10/1.2.1/inteldevstack/init_env.sh
export FPGA_BBB_CCI_src=/usr/local/intel-fpga-bbb
export PATH=/glob/intel-python/python2/bin:${PATH}
cd ~/playground/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/Tutorials/Features/ac_int/build
./ac_int.fpga

 It gives the following output : 

########################################################################
#      Date:           Fri Jun  9 05:02:02 PDT 2023
#    Job ID:           2316104.v-qsvr-1.aidevcloud
#      User:           uXXXXXXX
# Resources:           cput=75:00:00,neednodes=s001-n083:ppn=2,nodes=s001-n083:ppn=2,walltime=06:00:00
########################################################################

INTELFPGAOCLSDKROOT is set to /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/lib/oclfpga. Using that.

Will use $QUARTUS_ROOTDIR_OVERRIDE= /glob/development-tools/versions/oneapi/2022.3.1/intelFPGA_pro/19.2/quartus  to find Quartus

Adding /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/lib/oclfpga/bin to PATH
Adding /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/lib/oclfpga/linux64/lib to LD_LIBRARY_PATH
Adding /glob/development-tools/versions/oneapi/2023.1.2/oneapi/compiler/2023.1.0/linux/lib/oclfpga/host/linux64/lib to LD_LIBRARY_PATH
export QUARTUS_HOME=/glob/development-tools/versions/fpgasupportstack/a10/1.2.1/intelFPGA_pro/quartus
export OPAE_PLATFORM_ROOT=/glob/development-tools/versions/fpgasupportstack/a10/1.2.1/inteldevstack/a10_gx_pac_ias_1_2_1_pv
export AOCL_BOARD_PACKAGE_ROOT=/glob/development-tools/versions/fpgasupportstack/a10/1.2.1/inteldevstack/a10_gx_pac_ias_1_2_1_pv/opencl/opencl_bsp
Adding $OPAE_PLATFORM_ROOT/bin to PATH
export INTELFPGAOCLSDKROOT=/glob/development-tools/versions/fpgasupportstack/a10/1.2.1/intelFPGA_pro/hld
export ALTERAOCLSDKROOT=/glob/development-tools/versions/fpgasupportstack/a10/1.2.1/intelFPGA_pro/hld
Adding $QUARTUS_HOME/bin to PATH
source /glob/development-tools/versions/fpgasupportstack/a10/1.2.1/intelFPGA_pro/hld/init_opencl.sh
Error: Symbol aocl_mmd_shared_mem_alloc not found in board library(message: /opt/intel_2023.1/oneapi/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so: undefined symbol: aocl_mmd_shared_mem_alloc)
Error: Could not load board library /opt/intel_2023.1/oneapi/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so due to failure to load symbols
Failed to dynamically load board MMD /opt/intel_2023.1/oneapi/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so
Error: Could not load FPGA board libraries successfully.

########################################################################
# End of output for job 2316104.v-qsvr-1.aidevcloud
# Date: Fri Jun  9 05:02:16 PDT 2023
########################################################################

This is the error file :

Caught a SYCL host exception:
No device of requested type available. Please check https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-dpcpp-system-requirements.html -1 (PI_ERROR_DEVICE_NOT_FOUND)
terminate called after throwing an instance of 'sycl::_V1::runtime_error'
  what():  No device of requested type available. Please check https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-dpcpp-system-requirements.html -1 (PI_ERROR_DEVICE_NOT_FOUND)
/var/spool/torque/mom_priv/jobs/2316104.v-qsvr-1.aidevcloud.SC: line 9: 28552 Aborted                 ./ac_int.fpga

[JyothisV_Intel]

Hi,

Good day to you.

Thanks @Christoph9 for providing the solution and sharing it with the community.

Since your initial query is resolved and accepted as solution, do let us know if we can stop monitoring this thread.

Regards,

Jyothis V James

[JyothisV_Intel]

Hi,

Good day to you.

We have not received any response from you.

Intel will stop monitoring this thread as the solution to your initial query has been accepted by you as a solution. If you need any additional assistance with Intel products or services, kindly post in a new thread.

Thanks and Regards,

Jyothis V James