SEGV Running any emulator examples on RHEL7 Machine

[ndevenish]

Trying to use oneAPI 2023.0.0, building and running anything that tries to use the emulated FPGA:

 % ./fpga_compile.fpga_emu
Running on device: Intel(R) FPGA Emulation Device
zsh: segmentation fault ./fpga_compile.fpga_emu

Including my own code. The stack trace indicates this is happening somewhere in TBB:

* thread #1, name = 'fpga_compile.fp', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
* frame #0: 0x00007ffff475de62 libtask_executor_emu.so.2022.15.12.0`Intel::OpenCL::TaskExecutor::TEDevice::on_scheduler_entry(bool, Intel::OpenCL::TaskExecutor::ArenaHandler&) + 66
frame #1: 0x00007ffff3f59e08 libtbb.so.12`tbb::detail::r1::observer_list::do_notify_entry_observers(this=<unavailable>, last=<unavailable>, worker=<unavailable>) at observer_proxy.cpp:199
frame #2: 0x00007ffff3f73ba8 libtbb.so.12`tbb::detail::r1::nested_arena_context::nested_arena_context(tbb::detail::r1::thread_data&, tbb::detail::r1::arena&, unsigned long) [inlined] tbb::detail::r1::observer_list::notify_entry_observers(this=<unavailable>, last=<unavailable>, worker=<unavailable>) at observer_proxy.h:135
frame #3: 0x00007ffff3f73b96 libtbb.so.12`tbb::detail::r1::nested_arena_context::nested_arena_context(this=<unavailable>, td=<unavailable>, nested_arena=<unavailable>, slot_index=<unavailable>) at arena.cpp:537
frame #4: 0x00007ffff3f5c1a8 libtbb.so.12`tbb::detail::r1::task_arena_impl::execute(ta=<unavailable>, d=<unavailable>) at arena.cpp:690
frame #5: 0x00007ffff4763243 libtask_executor_emu.so.2022.15.12.0`Intel::OpenCL::TaskExecutor::immediate_command_list::LaunchExecutorTask(bool, Intel::OpenCL::Utils::SharedPtr<Intel::OpenCL::TaskExecutor::ITaskBase> const&) + 371
frame #6: 0x00007ffff4ac4dd6 libintelocl_emu.so`Intel::OpenCL::Framework::FrameworkProxy::ExecuteImmediate(Intel::OpenCL::Utils::SharedPtr<Intel::OpenCL::TaskExecutor::ITaskBase> const&) const + 22
frame #7: 0x00007ffff4b100be libintelocl_emu.so`Intel::OpenCL::Framework::BuildTask::Launch() + 158
frame #8: 0x00007ffff4b1bd62 libintelocl_emu.so`Intel::OpenCL::Framework::ProgramService::BuildProgram(Intel::OpenCL::Utils::SharedPtr<Intel::OpenCL::Framework::Program>&, unsigned int, _cl_device_id* const*, char const*, void (*)(_cl_program*, void*), void*) + 6978
frame #9: 0x00007ffff4af296a libintelocl_emu.so`Intel::OpenCL::Framework::Context::BuildProgram(_cl_program*, unsigned int, _cl_device_id* const*, char const*, void (*)(_cl_program*, void*), void*) + 346
frame #10: 0x00007ffff4ad5a49 libintelocl_emu.so`Intel::OpenCL::Framework::ContextModule::BuildProgram(_cl_program*, unsigned int, _cl_device_id* const*, char const*, void (*)(_cl_program*, void*), void*) + 409
frame #11: 0x00007ffff4a8061c libintelocl_emu.so`clBuildProgram + 2396
frame #12: 0x00007ffff75f5707 libsycl.so.6`_pi_result sycl::_V1::detail::plugin::call_nocheck<(sycl::_V1::detail::PiApiKind)42, _pi_program*, int, _pi_device* const*, char const*, std::nullptr_t, std::nullptr_t>(_pi_program*, int, _pi_device* const*, char const*, std::nullptr_t, std::nullptr_t) const + 391
frame #13: 0x00007ffff75f456c libsycl.so.6`sycl::_V1::detail::ProgramManager::build(std::unique_ptr<_pi_program, _pi_result (*)(_pi_program*)>, std::shared_ptr<sycl::_V1::detail::context_impl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, _pi_device* const&, unsigned int) + 4908
frame #14: 0x00007ffff75ef2bb libsycl.so.6`sycl::_V1::detail::ProgramManager::getBuiltPIProgram(long, std::shared_ptr<sycl::_V1::detail::context_impl> const&, std::shared_ptr<sycl::_V1::detail::device_impl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, sycl::_V1::detail::program_impl const*, bool) + 3547
frame #15: 0x00007ffff75f0832 libsycl.so.6`sycl::_V1::detail::ProgramManager::getOrCreateKernel(long, std::shared_ptr<sycl::_V1::detail::context_impl> const&, std::shared_ptr<sycl::_V1::detail::device_impl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, sycl::_V1::detail::program_impl const*) + 1170
frame #16: 0x00007ffff763c3e5 libsycl.so.6`sycl::_V1::detail::enqueueImpKernel(std::shared_ptr<sycl::_V1::detail::queue_impl> const&, sycl::_V1::detail::NDRDescT&, std::vector<sycl::_V1::detail::ArgDesc, std::allocator<sycl::_V1::detail::ArgDesc> >&, std::shared_ptr<sycl::_V1::detail::kernel_bundle_impl> const&, std::shared_ptr<sycl::_V1::detail::kernel_impl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const&, std::vector<_pi_event*, std::allocator<_pi_event*> >&, _pi_event**, std::function<void* (sycl::_V1::detail::AccessorImplHost*)> const&) + 309
frame #17: 0x00007ffff763fbe2 libsycl.so.6`sycl::_V1::detail::ExecCGCommand::enqueueImp() + 10082
frame #18: 0x00007ffff762bcd9 libsycl.so.6`sycl::_V1::detail::Command::enqueue(sycl::_V1::detail::EnqueueResultT&, sycl::_V1::detail::BlockingT, std::vector<sycl::_V1::detail::Command*, std::allocator<sycl::_V1::detail::Command*> >&) + 1017
frame #19: 0x00007ffff764b4a2 libsycl.so.6`sycl::_V1::detail::Scheduler::addCG(std::unique_ptr<sycl::_V1::detail::CG, std::default_delete<sycl::_V1::detail::CG> >, std::shared_ptr<sycl::_V1::detail::queue_impl>) + 1794
frame #20: 0x00007ffff76831b2 libsycl.so.6`sycl::_V1::handler::finalize() + 8626
frame #21: 0x00007ffff76b087a libsycl.so.6`void sycl::_V1::detail::queue_impl::finalizeHandler<sycl::_V1::handler>(sycl::_V1::handler&, sycl::_V1::detail::CG::CGTYPE const&, sycl::_V1::event&) + 202
frame #22: 0x00007ffff76b04f1 libsycl.so.6`sycl::_V1::detail::queue_impl::submit_impl(std::function<void (sycl::_V1::handler&)> const&, std::shared_ptr<sycl::_V1::detail::queue_impl> const&, std::shared_ptr<sycl::_V1::detail::queue_impl> const&, std::shared_ptr<sycl::_V1::detail::queue_impl> const&, sycl::_V1::detail::code_location const&, std::function<void (bool, bool, sycl::_V1::event&)> const*) + 929
frame #23: 0x00007ffff76af945 libsycl.so.6`sycl::_V1::detail::queue_impl::submit(std::function<void (sycl::_V1::handler&)> const&, std::shared_ptr<sycl::_V1::detail::queue_impl> const&, sycl::_V1::detail::code_location const&, std::function<void (bool, bool, sycl::_V1::event&)> const*) + 53
frame #24: 0x00007ffff76af905 libsycl.so.6`sycl::_V1::queue::submit_impl(std::function<void (sycl::_V1::handler&)>, sycl::_V1::detail::code_location const&) + 21
frame #25: 0x0000000000404241 fpga_compile.fpga_emu`main [inlined] sycl::_V1::event sycl::_V1::queue::submit<main::'lambda'(sycl::_V1::handler&)>(this=0x00007fffffffb380, CGF=_ZTSZ4mainEUlRN4sycl3_V17handlerEE_ @ 0x000055f5b83b2d60, CodeLoc=0x00007fffffffb448), sycl::_V1::detail::code_location const&) at queue.hpp:318:18
frame #26: 0x00000000004041e2 fpga_compile.fpga_emu`main at fpga_compile.cpp:62
frame #27: 0x00007ffff702e555 libc.so.6`__libc_start_main + 245
frame #28: 0x0000000000403766 fpga_compile.fpga_emu`_start + 41 

Now... we were advised by Intel that the PAC only worked on Red Hat 7 - this seems to be corroborated by the documentation here. , which clearly specifies that only Red Hat 7 and kernel 3.10.0 were supported.

Previously, there were instructions at https://www.intel.com/content/www/us/en/developer/articles/technical/oneapi-dpcpp-fpga-designs-on-rhel-7.html which described how to solve this using a separately built GCC (and an internal intel machine that we were given access to did indeed have this process done already, albeit for an old 2021 installation of oneAPI). This documentation has disappeared now, and points back to the FPGA front-page, like a lot of the documentation links.

Is this likely to be the cause of the issue? Is this use case not supported any more? How can I use these cards if oneAPI doesn't support the operating system that the cards must be installed on?

Is there anything else I can do to fix this?

[ndevenish]

In fact - the intel_s10sx_pac.tar.gz download for the D5005 explicitly only supports CentOS 7 or Ubuntu 18.04 (with kernel 4.15.*). That seems to conflict with this page which states the requirements for FPGA are strictly:

• RHEL* 9 and 8.x (NOTE: v8.6 is not supported)
• Ubuntu* 22.04 LTS, 20.04 LTS

So - it appears that officially it isn't possible to use these cards?

I note however that Intel DevCloud is using Ubuntu 18.04 for oneAPI 2023.0, so it must be possible to run on older OS versions that actually support the FPGA?

[hareesh]

Hi,

thanks for posting your issue here. just give me a some time i needs to discourse with my team with this issue.

Thanks,

[hareesh]

Hi,

i want know one more thing, that is which card are trying to access and version details. please share those information .

Thanks,

[ndevenish]

Hi,

We have two D5005 (Stratix 10). Installing the BSP works fine; and the cards can be enumerated by a plain SYCL (no kernel) program. But trying to run the fpga_compile example causes the SEGV - so it isn't actually even trying to access the card.

My suspicion is that this is because it is RHEL 7 - but the card BSP requires that, and apparently the old instructions for how to do this don't work any more (and have been removed).

Nick

[ndevenish]

Here is a Dockerfile to reproduce the CentOS 7 install, following the instructions Intel previously published for running on CentOS 7. It ends with attempting to run the fpga_compile example (which then SEGV).

FROM centos:7

RUN yum install -y wget git gcc gcc-c++ make xz bzip2 file gdb

WORKDIR /opt/build

# Get a newer version of CMake than in the base CentOS repos (2.8)
RUN curl -L https://github.com/Kitware/CMake/releases/download/v3.25.2/cmake-3.25.2-linux-x86_64.tar.gz \
| tar -zxf - --strip-components=1 -C /usr/local

# Get and build gcc 9.5.0 - this version chosen because it matches the
# default of Ubuntu 20.04, the lowest "officially" supported OS. The
# devcloud runs on Ubuntu 18.04, with gcc 7.4 (which the old instructions
# for getting this to work on CentOS 7 used)
RUN wget -q --no-check-certificate https://mirrorservice.org/sites/sourceware.org/pub/gcc/releases/gcc-9.5.0/gcc-9.5.0.tar.xz && \
tar -xf gcc-9.5.0.tar.xz && \
( cd gcc-9.5.0 && contrib/download_prerequisites ) && \
mkdir _build_gcc && \
cd _build_gcc && \
/opt/build/gcc-9.5.0/configure --disable-multilib --enable-languages=c,c++ --prefix=/opt/gcc-9.5 --enable-shared && \
make -j $(grep -c ^processor /proc/cpuinfo) && \
make install

# Get and install oneAPI
RUN wget https://registrationcenter-download.intel.com/akdlm/irc_nas/19079/l_BaseKit_p_2023.0.0.25537_offline.sh
RUN sh ./l_BaseKit_p_2023.0.0.25537_offline.sh -a -s -c --eula accept --install-dir=/opt/intel

# Get the oneAPI samples repository
RUN cd /opt && git clone https://github.com/oneapi-src/oneAPI-samples.git -b 2023.0.0 --depth 1
RUN mkdir /opt/build/fpga_compile
WORKDIR /opt/build/fpga_compile

# Set the environment variables to use a more recent libstdc++ base for icpx.
# These were previously at:
# https://www.intel.com/content/www/us/en/developer/articles/technical/oneapi-dpcpp-fpga-designs-on-rhel-7.html
# but the documentation appears to have been removed.
ENV LD_PRELOAD=/opt/gcc-9.5/lib64/libstdc++.so
ENV CXXFLAGS=--gcc-toolchain=/opt/gcc-9.5
ENV CFLAGS=--gcc-toolchain=/opt/gcc-9.5
ENV LD_LIBRARY_PATH=/opt/gcc-9.5/lib64:$LD_LIBRARY_PATH

# Build the fpga_compile example
RUN source /opt/intel/setvars.sh && \
cmake /opt/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/GettingStarted/fpga_compile && \
cmake --build .

# Running this for emulator SEGV
RUN source /opt/intel/setvars.sh && \
./fpga_compile.fpga_emu

I've been trying to track down the issue in GDB - I suspect the corruption causing the crash is "warning: Corrupted shared library list" which happens in khrIcdOsLibraryLoad, attempting to dlopen("libintelocl_emu.so"). I haven't found out why loading this corrupts the memory, yet.

If there is some instructions for getting the toolkit working on CentOS7, or some way to download the older version of oneAPI and fpga base toolkits? I believe I have copies of oneAPI 2022.1, 2022.2, but without the fpga addon download I'm not sure how to use these (not to mention there were several bugs I encountered with the early 2022 editions that would be nice to fix).

Failing that, some way to actually use the PAC on a newer OS would be good - we were told that it would only work on RHEL/CENTOS 7, and although the card is discontinued, you do seem to still be selling them.

[hareesh]

Hi,

sorry for late responding.

i can understand your issue. please give me some time. need to discourse with team about how to get older version.

Thanks,

[hareesh]

Hi,

To get older version in oneAPI, there needs to be certain license required for that.

please go through bellow post for license requesting process.

https://community.intel.com/t5/oneAPI-Registration-Download/Download-an-older-versions-of-oneAPI-toolkits/m-p/1354501

once requested the license then can go 'registration center' for the download one

please check bellow post

https://community.intel.com/t5/oneAPI-Registration-Download/Where-can-I-find-older-versions-2021-3-of-Intel-OneAPI-Base/m-p/1323750

Thank you,

[ndevenish]

Hi Hareesh,

The older downloads are only available for users that have purchased a commercial license

This seems to imply the answer to my question is no, without paying for a licence.

Are you confirming that it's now actually impossible to use the Stratix-generation cards with oneAPI without paying for a licence? You are still advertising this on https://www.intel.com/content/www/us/en/developer/tools/oneapi/fpga.html !

Nick

[hareesh]

Hi,

without license we can use older version only. if you want new version you need buy license. that choice only we have. sorry for saying like this. because we don't have another way to access older versions.

Thanks,

[hareesh]

As we do not receive any response from you on the previous question/reply/answer that we have provided. Please login to ‘https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

[ndevenish]

I don't want to use an old version, I want to know how to get the current version (which says that Stratix is supported?) to work. But that is impossible?