[Help] compiling my project for FPGA

ManuelCostanzo2 · ‎02-09-2023

Hi!

I'm working on my project: https://github.com/ManuelCostanzo/swsharp_sycl.

The folder there is "SYCL".

I have 3 Makefiles:
SYCL/Makefile

SYCL/swsharp/Makefile => build the library

SYCL/swsharpd/Makefile => buiid the program (that includes the library).

If you clone the repo and compile with "BACK=icpx make", it works perfectly. The problem is when I want to compile for FPGAs, for example: "BACK=fpga make", I'm getting errors.

I don't know if the correct way to compile the library and the program is how I'm doing. For example in the library (folder SYCL/swsharp) I added the following to the Makefile:

ifeq ($(BACK), fpga)

CP = icpx -fsycl -fintelfpga -Xshardware -Xstarget=/opt/intel/oneapi/intel_a10gx_pac:pac_a10 -v ../lib/libswsharp.a

CU = icpx -fsycl -fintelfpga -Xshardware -Xstarget=/opt/intel/oneapi/intel_a10gx_pac:pac_a10 -v ../lib/libswsharp.a

LD = icpx -fsycl -fintelfpga -Xshardware -Xstarget=/opt/intel/oneapi/intel_a10gx_pac:pac_a10 -v ../lib/libswsharp.a

endif

I'm using the fpga_compile node in DevCloud. At the end, the error I'm getting is:

******* Error: Assert failure at ../../../source/acl/llvm-project/llvm/lib/Target/FPGA/Griffin/XNodeUtil.cpp(182) *******
already connected this input (primWireOut -> extract_extract)! from: extract_extract_reg, to: outputBlk
allow_replacement || !toBlock->getPort(toPortName) || toBlock->getPort(toPortName)->getWire() == *wire FAILED
PLEASE submit a bug report to https://software.intel.com/en-us/support/priority-support and include the crash backtrace.
Stack dump:
0. Program arguments: /glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc -march=fpga -board /opt/intel/oneapi/intel_a10gx_pac/hardware/pac_a10/board_spec.xml -pass-remarks-input=pass-remarks.yaml -sycl -dbg-info-enabled swsharpdb.bc -o swsharpdb.v
1. Running pass 'DSDKGenerate Pipeline' on module 'swsharpdb.bc'.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x27884af)[0x555da9bf44af]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x27853bd)[0x555da9bf13bd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f5e4b786420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f5e4789c00b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f5e4787b859]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3f8857f)[0x555dab3f457f]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3f89930)[0x555dab3f5930]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3d99c20)[0x555dab205c20]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3d83611)[0x555dab1ef611]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3e26f9e)[0x555dab292f9e]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3c371b7)[0x555dab0a31b7]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3c250b9)[0x555dab0910b9]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3c29a17)[0x555dab095a17]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3c2aaf2)[0x555dab096af2]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x3c2e8ef)[0x555dab09a8ef]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0x2edcdb1)[0x555daa348db1]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(_ZN3acl25GriffinPipelineLegacyPass11runOnModuleERN4llvm6ModuleE+0xedc)[0x555daa34be8c]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x2fb)[0x555da9381ddb]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(+0xfa41d8)[0x555da84101d8]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(main+0x389)[0x555da82ddea9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f5e4787d083]
/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/llvm/aocl-bin/aocl-llc(_start+0x29)[0x555da8406059]
Aborted
Error: Verilog generator FAILED.
Refer to swsharpdb.prj/swsharpdb.log for details.

llvm-foreach:
icpx: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:90: swsharpdb] Error 1
make: *** [Makefile:56: swsharpdb] Error 2

Can you help me with this please? Feel free to clone and test the repo. Thank you very much, I'm very interested to try the FPGAs and oneAPI.

JaideepK_Intel · ‎02-10-2023

Hi,

Thank you for posting in Intel communities.

We are able to reproduce your issue from our end and we are working on it. We will get back to you with an update.

Thanks,

Jaideep

ManuelCostanzo2 · ‎02-14-2023

Hi, how are you?

Do you have any updates on this? I couldn't have a success compilation yet. Thank you very much!

ManuelCostanzo2 · ‎02-10-2023

@JaideepK_Intel Thank you very much! I'm trying to fix it too

JaideepK_Intel · ‎02-20-2023

Hi,

Sorry for the delay we are still working on this issue internally. We will get back to you with an update.

Thanks,

Jaideep

ManuelCostanzo2 · ‎02-22-2023

Pefect, thank you very much. I'm waiting

JaideepK_Intel · ‎03-02-2023

Hi,

Good day to you.

There are currently some licensing issues for FPGA nodes, which could be the source of this problem. We are working on them internally. We will get back to you with an update.

Thanks,

Jaideep

ManuelCostanzo2 · ‎03-03-2023

Hey, good day.

Ah ok perfect. Do you have my project so you can test the compilation, right? Do you need some information about this?

Thanks,

Manuel.

JaideepK_Intel · ‎03-10-2023

Hi,

Good day to you.

Thanks a lot for your patience, I sincerely apologize for the delay and inconvenience caused. Dev team has been consistently working on this issue, and finally the issue with Stratix10 oneAPI FPGA cards has been resolved. We have tested and verified that the below samples are working fine.

Stratix10 (fpga_compile, fast_recompile), Arria10 (fpga_compile, fast_recompile, simple-add, vector-add) samples are working fine

But there is some linking issue in your code.

BACK=icpx make

BACK=fpga make (after running the above command we are unable to see any errors)

https://github.com/oneapi-src/oneAPI-samples/tree/master/DirectProgramming/C%2B%2BSYCL_FPGA/Tutorials/GettingStarted

https://github.com/oneapi-src/oneAPI-samples/tree/master/DirectProgramming/C%2B%2BSYCL/DenseLinearAlgebra

Thanks,

Jaideep

ManuelCostanzo2 · ‎03-15-2023

Hi @JaideepK_Intel, you moved the content but it seems that I don't have access. Is that correct? Thanks

JaideepK_Intel · ‎03-15-2023

Hi @ManuelCostanzo2,

Sorry for the confusion. You can ignore the above message.

Thanks,

Jaideep

ManuelCostanzo2 · ‎03-15-2023

Oh but my message have been removed. Do I send it again? No problem

JaideepK_Intel · ‎03-15-2023

Hi @ManuelCostanzo2 ,

Could you please send your message again?

Thanks,

Jaideep

ManuelCostanzo2 · ‎03-15-2023

@JaideepK_Intel Hi, sure!

So, before testing with my project, I would like to successfully compile and run a simple USM code. As I believe, only Stratix FPGA has USM enabled, but Arria doesn't have that feature, right?

So, I enter to a compile node:

qsub -l nodes=1:fpga_compile:ppn=2 -I

Then, I setup the environment variables:

export AOCL_BOARD_PACKAGE_ROOT=/opt/intel/oneapi/intel_s10sx_pac
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/init_opencl.sh
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/init_env.sh
export FPGA_BBB_CCI_src=/usr/local/intel-fpga-bbb
export PATH=/glob/intel-python/python2/bin:${PATH}

source /opt/intel/oneapi/setvars.sh

and then I compile my code in two steps:

icpx -fsycl -fintelfpga -Xshardware -c test.cpp -o test.cpp.o
icpx -fsycl -fintelfpga -Xshardware -Xstarget=/opt/intel/oneapi/intel_s10sx_pac:pac_s10_usm test.cpp.o -o test -v

The problem is that in the last command, after 1-2 hours, I get an error saying: "Error: Failed to open quartus_sh_compile.log".

I don't know what am I doing wrong here. Could you help me please? The code is the following.

I really want to thank you for your help all this time. I feel that I am close to being able to compile it successfully.

#include <sycl/sycl.hpp>
#include <array>
#include <iostream>
#include <string>

using namespace sycl;

size_t array_size = 10000;
//************************************
// Vector add in SYCL on device: returns sum in 4th parameter "sum".
//************************************
void VectorAdd(queue &q, const int *a, const int *b, int *sum, size_t size)
{
  // Create the range object for the arrays.
  range<1> num_items{size};

  auto e = q.parallel_for(num_items, [=](auto i)
                          { sum[i] = a[i] + b[i]; });

  e.wait();
}

//************************************
// Initialize the array from 0 to array_size - 1
//************************************
void InitializeArray(int *a, size_t size)
{
  for (size_t i = 0; i < size; i++)
    a[i] = i;
}

int main(int argc, char *argv[])
{
  try
  {
    sycl::queue q((sycl::device::get_devices()[std::stoi(argv[1])]));

    // Print out the device information used for the kernel code.
    std::cout << "Running on device: "
              << q.get_device().get_info<info::device::name>() << "\n";
    std::cout << "Running on device: "
              << q.get_device().get_info<info::device::max_compute_units>() << "\n";
    std::cout << "Running on device: "
              << q.get_device().get_info<info::device::max_work_group_size>() << "\n";
    std::cout << "Running on device: "
              << q.get_device().get_info<info::device::global_mem_size>() << "\n";

    int *a = malloc_shared<int>(array_size, q);
    int *b = malloc_shared<int>(array_size, q);
    int *sum_sequential = malloc_shared<int>(array_size, q);
    int *sum_parallel = malloc_shared<int>(array_size, q);

    if ((a == nullptr) || (b == nullptr) || (sum_sequential == nullptr) ||
        (sum_parallel == nullptr))
    {
      if (a != nullptr)
        free(a, q);
      if (b != nullptr)
        free(b, q);
      if (sum_sequential != nullptr)
        free(sum_sequential, q);
      if (sum_parallel != nullptr)
        free(sum_parallel, q);

      std::cout << "Shared memory allocation failure.\n";
      return -1;
    }

    InitializeArray(a, array_size);
    InitializeArray(b, array_size);

    for (size_t i = 0; i < array_size; i++)
      sum_sequential[i] = a[i] + b[i];

    VectorAdd(q, a, b, sum_parallel, array_size);

    for (size_t i = 0; i < array_size; i++)
    {
      if (sum_parallel[i] != sum_sequential[i])
      {
        std::cout << "Vector add failed on device.\n";
        return -1;
      }
    }

    int indices[]{0, 1, 2, (static_cast<int>(array_size) - 1)};
    constexpr size_t indices_size = sizeof(indices) / sizeof(int);

    for (int i = 0; i < indices_size; i++)
    {
      int j = indices[i];
      if (i == indices_size - 1)
        std::cout << "...\n";
      std::cout << "[" << j << "]: " << j << " + " << j << " = "
                << sum_sequential[j] << "\n";
    }

    free(a, q);
    free(b, q);
    free(sum_sequential, q);
    free(sum_parallel, q);
  }
  catch (exception const &e)
  {
    std::cout << "An exception is caught while adding two vectors.\n";
    std::terminate();
  }

  std::cout << "Vector add successfully completed on device.\n";
  return 0;
}

aikeu · ‎03-15-2023

Hi ManuelCostanzo2,

May I know what is the current status of the issue?

Thanks.

Regards,

Aik Eu

ManuelCostanzo2 · ‎03-16-2023

Hi @aikeu,

I still couldn't successfully compile my project for FPGAs, but first I would like to compile and run the one I wrote in the message above: https://community.intel.com/t5/Intel-High-Level-Design/Help-compiling-my-project-for-FPGA/m-p/1466119#M3053 , where I attached the commands I used and the error I got.

Thanks

ManuelCostanzo2 · ‎03-20-2023

Hi @JaideepK_Intel and @aikeu, just for asking if you are on this topic, trying to solve it Thank you very much!