Issue on fpga_compile nodes

[Christoph9]

Hello,

starting from yesterday, I have troubles compiling any FPGA-related stuff on the fpga_compile nodes. Even the GettingStarted example from the OneApi repository (DirectProgramming/DPC++FPGA/Tutorials/GettingStarted) does no longer work.

Trying to compile the emulator-target leads to the following output:

u153009@s001-n058:~/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/GettingStarted/fpga_compile$ make fpga_emu
[ 50%] Linking CXX executable ../fpga_compile.fpga_emu
OpenCL platform ID is empty
OpenCL platform name is empty
Failed to find any of these OpenCL platforms:
  Intel(R) FPGA Emulation Platform for OpenCL(TM)
  Intel(R) FPGA Emulation Platform for OpenCL(TM) (preview)
llvm-foreach: 
dpcpp: error: fpga compiler command failed with exit code 14 (use -v to see invocation)
make[3]: *** [src/CMakeFiles/fpga_compile.fpga_emu.dir/build.make:84: fpga_compile.fpga_emu] Error 14
make[2]: *** [CMakeFiles/Makefile2:235: src/CMakeFiles/fpga_compile.fpga_emu.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:215: src/CMakeFiles/fpga_emu.dir/rule] Error 2
make: *** [Makefile:170: fpga_emu] Error 2

Compiling the hardware-target gives a different error:

u1XXXXX@s001-n058:~/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/GettingStarted/fpga_compile$ make fpga
Scanning dependencies of target fpga_compile.fpga
[ 50%] Building CXX object src/CMakeFiles/fpga_compile.fpga.dir/fpga_compile.cpp.o
[100%] Linking CXX executable ../fpga_compile.fpga
aoc: Compiling for FPGA. This process may take several hours to complete.  Prior to performing this compile, be sure to check the reports to ensure the design will meet your performance targets.  If the reports indicate performance targets are not being met, code edits may be required.  Please refer to the oneAPI FPGA Optimization Guide for information on performance tuning applications for FPGAs.
Error: The board package being used only supports report generation/emulation/runtime, please use the board package shipped in the FPGA addon instead.
Error: Exiting.

llvm-foreach: 
dpcpp: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
make[3]: *** [src/CMakeFiles/fpga_compile.fpga.dir/build.make:84: fpga_compile.fpga] Error 1
make[2]: *** [CMakeFiles/Makefile2:100: src/CMakeFiles/fpga_compile.fpga.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:161: src/CMakeFiles/fpga.dir/rule] Error 2
make: *** [Makefile:144: fpga] Error 2

These compile on the Arria10-nodes without issue, so I wonder why it no longer works on the compile-nodes. It looks like a new Sycl version was installed, are there now additional commands necessary? I tried loading an older version via module load 2021.2, however this did'nt help.

Also running fpga-programs now leads to the following warnings on arria-10 nodes:

Error: Could not load board library /opt/intel/inteloneapi/intelfpgadpcpp/latest/board/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so (error_msg: /opt/intel/inteloneapi/intelfpgadpcpp/latest/board/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so: cannot open shared object file: No such file or directory)
Failed to dynamically load board MMD /opt/intel/inteloneapi/intelfpgadpcpp/latest/board/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so
Error: Could not load board library /opt/intel/inteloneapi/intelfpgadpcpp/latest/board/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so (error_msg: /opt/intel/inteloneapi/intelfpgadpcpp/latest/board/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so: cannot open shared object file: No such file or directory)
Failed to dynamically load board MMD /opt/intel/inteloneapi/intelfpgadpcpp/latest/board/intel_a10gx_pac/linux64/lib/libintel_opae_mmd.so

 These errors/warnings also occur when calling sycl-ls in the terminal. After these however, programs seem to continue normally.

Thanks in advance,
Christoph

[JaideepK_Intel]

Hi,

Thank you for posting in Intel Communities.

Could you please let us know that you are facing this issue only on this (s001-n058) particular node or on all the FPGA nodes ?

Regards,

Jaideep

[Christoph9]

Hey Jaydeep,

I just also tried compiling on the nodes 69 and 70 (both fpga_compile nodes). The exact same error also occurs on those.

Regards,
Christoph

[JaideepK_Intel]

Hi,

Good day to you.

We are able to reproduce the same issue and we are working on this internally to get back to you with an update.

Thanks,

Jaideep

[Dan_P_Intel]

Hi, could you please try again?

Following some tweaks I was able to compile on an fpga_compile node and execute on an arria10 node.

Please keep in mind the following two points:

1. you need to checkout the 2022.3.0 tag.
   That will address the following error that you might encounter otherwise:
   
   "fatal error: 'sycl/sycl.hpp' file not found"
   
   This is just a temporary solution until the latest code changes are merged into the master branch.
   
   $ git clone https://github.com/oneapi-src/oneAPI-samples.git
   
   $ git fetch --all --tags
   Fetching origin
   From https://github.com/oneapi-src/oneAPI-samples
   t [tag update] 2022.3.0 -> 2022.3.0
   
   $ git checkout tags/2022.3.0
   Checking out files: 100% (270/270), done.
   Note: checking out 'tags/2022.3.0'.
   
   $ git branch
   * (HEAD detached at 2022.3.0)
     master
   
   
2. when compiling on an fpga_compile node, you need to specify an additional argument to the cmake command:
   
   # for Arria10
   $ cmake .. -DFPGA_BOARD=/opt/intel/oneapi/intel_a10gx_pac:pac_a10
   
   or
   
   # for Stratix10
   $ cmake .. -DFPGA_BOARD=/opt/intel/oneapi/intel_s10sx_pac:pac_s10
   
   This step is not necessary when compiling on a compute node that hosts a physical FPGA card such as Arria10 or Stratix10.

There are other samples not using cmake but just make instead, such as vector-add.
In this case, in order to compile on an fpga_compile node you need to make changes to the makefile.

Replace the following:

$(CXX) $(CXXFLAGS) -fintelfpga $^ -o $@ -Xshardware

with

# for Arria10
$(CXX) $(CXXFLAGS) -fintelfpga $^ -o $@ -Xshardware -Xsboard=/opt/intel/oneapi/intel_a10gx_pac:pac_a10

or

# for Stratix10
$(CXX) $(CXXFLAGS) -fintelfpga $^ -o $@ -Xshardware -Xsboard=/opt/intel/oneapi/intel_s10sx_pac:pac_s10

Kind regards,

Dan

[Christoph9]

Hello Dan,

thanks for your advises. With them I was able to successfully compile fpga-programs on the fpga_compile-nodes again.

However there are still two major issues with the new SYCL-Version on the DevCloud

1.  The first and pretty severe one is that explicit memory-allocations no longer work. If I place for example the following code snippet in the GettingStartet/fpga_compile project right behind the creation of the queue q:
   
   

   queue q(device_selector, dpc_common::exception_handler);
   if (nullptr == sycl::malloc_host<int>(1, q))
      std::cerr << "Error allocating int!" << std::endl;​

   
   The allocation will always fail and return 0. The implicit memory-allocations using accessors, as in the fpga_compile example, however still work. As the executable prints "PASSED: results are correct". (Build on node 75 (fpga_compile) and executed on arria10-node 81 - happens also on 87 which I also tested) This only happens when using the queue of an fpga accelerator. On CPU, GPU and fpga Emulator this works fine and allocates memory as expected.
   
   Edit: This also happens when using the C-Style mallocs:
   

   int* val = (int *)sycl::malloc_host(42, q);
   if (nullptr == val)
      std::cerr << "Still not allocating :(" << std::endl;​

   
   Could you please also look into that? I need the explicit memory-allocations for my current project.
2. On the fpga_compile nodes I am still unable to compile fpga_emulator targets, the error is the same as in my original posting.

Thanks in advance and Best Regards,
Christoph

[Dan_P_Intel]

I was unable to reproduce the FPGA emulator issue.

On s001-n067 (fpga_compile

$ mkdir build
$ cd build/
$ cmake .. -DFPGA_BOARD=/opt/intel/oneapi/intel_a10gx_pac:pac_a10

-- The CXX compiler identification is Clang 15.0.0
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2022.3/oneapi/compiler/2022.2.0/linux/bin/dpcpp
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2022.3/oneapi/compiler/2022.2.0/linux/bin/dpcpp -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring the design to run on FPGA board /opt/intel/oneapi/intel_a10gx_pac:pac_a10
-- Configuring done
-- Generating done
-- Build files have been written to: /home/u25802/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/GettingStarted/fpga_compile.s001-n067/build

$ clinfo -l

Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM)
 `-- Device #0: Intel(R) FPGA Emulation Device
Platform #1: Intel(R) OpenCL
 `-- Device #0: Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz

$ make fpga_emu

Scanning dependencies of target fpga_compile.fpga_emu
[ 50%] Building CXX object src/CMakeFiles/fpga_compile.fpga_emu.dir/fpga_compile.cpp.o
[100%] Linking CXX executable ../fpga_compile.fpga_emu
[100%] Built target fpga_compile.fpga_emu
Scanning dependencies of target fpga_emu
[100%] Built target fpga_emu

$ ./fpga_compile.fpga_emu

Running on device: Intel(R) FPGA Emulation Device
PASSED: results are correct

On s001-n081 (fpga_runtime/arria10

$ mkdir build
$ cd build/
$ cmake ..

-- The CXX compiler identification is Clang 15.0.0
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2022.3/oneapi/compiler/2022.2.0/linux/bin/dpcpp
-- Check for working CXX compiler: /glob/development-tools/versions/oneapi/2022.3/oneapi/compiler/2022.2.0/linux/bin/dpcpp -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- FPGA_BOARD was not specified.
Configuring the design to run on the default FPGA board intel_a10gx_pac:pac_a10 (Intel(R) PAC with Intel Arria(R) 10 GX FPGA).
Please refer to the README for information on board selection.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/u25802/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/GettingStarted/fpga_compile.s001-n081/build

$ clinfo -l

Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM)
 `-- Device #0: Intel(R) FPGA Emulation Device
Platform #1: Intel(R) FPGA SDK for OpenCL(TM)
 `-- Device #0: pac_a10 : Intel PAC Platform (pac_ee00000)
Platform #2: Intel(R) OpenCL
 `-- Device #0: Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz

$ make fpga_emu

Scanning dependencies of target fpga_compile.fpga_emu
[ 50%] Building CXX object src/CMakeFiles/fpga_compile.fpga_emu.dir/fpga_compile.cpp.o
[100%] Linking CXX executable ../fpga_compile.fpga_emu
[100%] Built target fpga_compile.fpga_emu
Scanning dependencies of target fpga_emu
[100%] Built target fpga_emu

$ ./fpga_compile.fpga_emu

Running on device: Intel(R) FPGA Emulation Device
PASSED: results are correct

Regarding the memory allocations issue, I haven't kept up to date with sycl spec changes so I have to ask around.

[Christoph9]

Hey Dan,

thanks for  your reply. The FPGA Emulation build is now also working. I have to load the 'oclfpga' module first however. Out of the box, even at node 67, clinfo -l just returns the CPU and no fpga emulation device.

I am looking forward to your research on the memory allocation issue. As stated, the allocations work fine when using a queue which does not belong to an FPGA device. Running on the fpga emulator or CPU for example the allocations do work and return a pointer unequal to zero. This happens regardless where I allocate (host or device - both return nullptr).

The SYCL docs state that nullptr returns indicate an error during allocation (as with the normal alloc functions in C). In malloc from C this usually indicates that there is no more free memory available, here however I just allocate a single integer at first, so I think this should not be the case here.

Kind regards,
Christoph

[Christoph9]

Hey @JaideepK_Intel ,

could you reproduce my issue regarding the memory allocations, or are there some other news regarding it?

Thanks in advance,
Christoph

[JaideepK_Intel · Accepted solution]

Hi,

Sorry for the delay, it looks like you are trying to attempt USM allocations on a board variant that doesn't support USM. 

From the simple_host_streaming sample:

Notice: SYCL USM host allocations (and therefore this tutorial) are only supported for the Intel® FPGA PAC D5005 (with Intel Stratix® 10 SX) with USM support (i.e., intel_s10sx_pac:pac_s10_usm) *

If the user wishes to use sycl::malloc_host with the fpga_compile sample they will need to compile to a board variant that supports USM, e.g.

cmake .. -DFPGA_BOARD=/opt/intel/oneapi/intel_s10sx_pac:pac_s10_usm

Thanks,

Jaideep

[JaideepK_Intel]

Hi,

Thanks for accepting our solution. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks,

Jaideep