Exception: The program was built for 1 devices

[Weiqun_Z_]

Hi, I am a developer of AMReX (https://github.com/AMReX-Codes/amrex).  We are currently porting AMReX to use DPC++ as a backend for Intel GPUs.  For development, we are running on DevCloud.  If I compile the following code with "-O0" and link to AMReX, an exception, "The program was built or 1 devices", is caught at runtime.  The code below has no calls to any AMReX functions. But it runs fine if I compile it with "-O3".  So why is this happening?  What does the exception message mean?  I have put the test code with a makefile at https://github.com/WeiqunZhang/dpcpp/tree/master/exception.

Thanks!

Weiqun

#include <CL/sycl.hpp>
namespace sycl = cl::sycl;

int main (int argc, char* argv[])
{
    {
        sycl::gpu_selector my_selector;
        sycl::device my_device(my_selector);
        sycl::context my_context(my_device);
        sycl::ordered_queue q(my_context, my_selector);
        try {
            q.submit([&] (sycl::handler& h) {
                h.parallel_for(sycl::nd_range<1>(sycl::range<1>(8),sycl::range<1>(8)),
                [=] (sycl::nd_item<1> item)
                {
                    //
                });
            });
        } catch (sycl::exception const& ex) {
            std::cout << "sycl::exception caught: " << ex.what() << std::endl;
        }
    }
    std::cout << "End of program\n";
}

[AbhishekD_Intel · Accepted solution]

Hi Weiqun,

We build the AMReX from your specified git location and also tried building the Makefile specified in "https://github.com/WeiqunZhang/dpcpp/tree/master/exception" with your main.cpp code, but despite having USE_DPCPP = TRUE in Makefile its selecting g++ compiler and not dpcpp.

Please make changes in MakeFile and resend it to us.

We also manually tried to compile the main.cpp and linked with AMReX using -O0 and -O3 flag and with working fine, for more details we have specified the commands:

Compiling:

dpcpp -Werror=return-type -g -O0  -pthread  -DNDEBUG -DAMREX_GIT_VERSION=\"20.02\" -DBL_GCC_VERSION=7.4.0 -DBL_GCC_MAJOR_VERSION=7 -DBL_GCC_MINOR_VERSION=4 -DAMREX_LAUNCH="" -DAMREX_DEVICE="" -DAMREX_CUDA_FORT_GLOBAL="" -DAMREX_CUDA_FORT_DEVICE="" -DAMREX_CUDA_FORT_HOST="" -DAMREX_CUDA_FORT_HOST_DEVICE="" -DBL_SPACEDIM=3 -DAMREX_SPACEDIM=3 -DBL_FORT_USE_UNDERSCORE -DAMREX_FORT_USE_UNDERSCORE -DBL_Linux -DAMREX_Linux -I. -I../..//Src/Base -c main.cpp -o tmp_build_dir/o/3d.gnu.EXE/main.o

Linking:

dpcpp  -Werror=return-type -g -O0  -pthread  -DNDEBUG -DAMREX_GIT_VERSION=\"20.02\" -DBL_GCC_VERSION=7.4.0 -DBL_GCC_MAJOR_VERSION=7 -DBL_GCC_MINOR_VERSION=4 -DAMREX_LAUNCH="" -DAMREX_DEVICE="" -DAMREX_CUDA_FORT_GLOBAL="" -DAMREX_CUDA_FORT_DEVICE="" -DAMREX_CUDA_FORT_HOST="" -DAMREX_CUDA_FORT_HOST_DEVICE="" -DBL_SPACEDIM=3 -DAMREX_SPACEDIM=3 -DBL_FORT_USE_UNDERSCORE -DAMREX_FORT_USE_UNDERSCORE -DBL_Linux -DAMREX_Linux -I. -I../..//Src/Base -L. -L/usr/lib/gcc/x86_64-linux-gnu/7/ -o main3d.gnu.ex tmp_build_dir/o/3d.gnu.EXE/main.o tmp_build_dir/o/3d.gnu.EXE/AMReX.o tmp_build_dir/o/3d.gnu.EXE/AMReX_error_fi.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BlockMutex.o tmp_build_dir/o/3d.gnu.EXE/AMReX_ParmParse.o tmp_build_dir/o/3d.gnu.EXE/AMReX_parmparse_fi.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Utility.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Random.o tmp_build_dir/o/3d.gnu.EXE/AMReX_DistributionMapping.o tmp_build_dir/o/3d.gnu.EXE/AMReX_ParallelDescriptor.o tmp_build_dir/o/3d.gnu.EXE/AMReX_ForkJoin.o tmp_build_dir/o/3d.gnu.EXE/AMReX_ParallelContext.o tmp_build_dir/o/3d.gnu.EXE/AMReX_VisMF.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Arena.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BArena.o tmp_build_dir/o/3d.gnu.EXE/AMReX_CArena.o tmp_build_dir/o/3d.gnu.EXE/AMReX_DArena.o tmp_build_dir/o/3d.gnu.EXE/AMReX_EArena.o tmp_build_dir/o/3d.gnu.EXE/AMReX_NFiles.o tmp_build_dir/o/3d.gnu.EXE/AMReX_parstream.o tmp_build_dir/o/3d.gnu.EXE/AMReX_GpuLaunch.o tmp_build_dir/o/3d.gnu.EXE/AMReX_GpuControl.o tmp_build_dir/o/3d.gnu.EXE/AMReX_GpuDevice.o tmp_build_dir/o/3d.gnu.EXE/AMReX_GpuUtility.o tmp_build_dir/o/3d.gnu.EXE/AMReX_GpuAsyncArray.o tmp_build_dir/o/3d.gnu.EXE/AMReX_GpuElixir.o tmp_build_dir/o/3d.gnu.EXE/AMReX_FabConv.o tmp_build_dir/o/3d.gnu.EXE/AMReX_FPC.o tmp_build_dir/o/3d.gnu.EXE/AMReX_IntConv.o tmp_build_dir/o/3d.gnu.EXE/AMReX_VectorIO.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Box.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BoxIterator.o tmp_build_dir/o/3d.gnu.EXE/AMReX_IntVect.o tmp_build_dir/o/3d.gnu.EXE/AMReX_IndexType.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Orientation.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Periodicity.o tmp_build_dir/o/3d.gnu.EXE/AMReX_RealBox.o tmp_build_dir/o/3d.gnu.EXE/AMReX_RealVect.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BoxList.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BoxArray.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BoxDomain.o tmp_build_dir/o/3d.gnu.EXE/AMReX_FArrayBox.o tmp_build_dir/o/3d.gnu.EXE/AMReX_IArrayBox.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BaseFab.o tmp_build_dir/o/3d.gnu.EXE/AMReX_MultiFab.o tmp_build_dir/o/3d.gnu.EXE/AMReX_MFCopyDescriptor.o tmp_build_dir/o/3d.gnu.EXE/AMReX_iMultiFab.o tmp_build_dir/o/3d.gnu.EXE/AMReX_FabArrayBase.o tmp_build_dir/o/3d.gnu.EXE/AMReX_MFIter.o tmp_build_dir/o/3d.gnu.EXE/AMReX_CoordSys.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Geometry.o tmp_build_dir/o/3d.gnu.EXE/AMReX_MultiFabUtil.o tmp_build_dir/o/3d.gnu.EXE/AMReX_MultiFabUtil_Perilla.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BCRec.o tmp_build_dir/o/3d.gnu.EXE/AMReX_PhysBCFunct.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BCUtil.o tmp_build_dir/o/3d.gnu.EXE/AMReX_PlotFileUtil.o tmp_build_dir/o/3d.gnu.EXE/AMReX_PlotFileDataImpl.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BLProfiler.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BLBackTrace.o tmp_build_dir/o/3d.gnu.EXE/AMReX_MemPool.o tmp_build_dir/o/3d.gnu.EXE/AMReX_Machine.o tmp_build_dir/o/3d.gnu.EXE/AMReX_constants_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_FILCC_3D.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BLutil_F.o tmp_build_dir/o/3d.gnu.EXE/AMReX_BLProfiler_F.o tmp_build_dir/o/3d.gnu.EXE/AMReX_filcc_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_omp_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_acc_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_fort_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_error_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_parmparse_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_string_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_bc_types_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_io_mod.o tmp_build_dir/o/3d.gnu.EXE/AMReX_ParallelDescriptor_F.o tmp_build_dir/o/3d.gnu.EXE/AMReX_mempool_mod.o  -lgfortran -lquadmath

And for both -O0 and -O3 its working fine.

Will you send me modified Makefile so that we can debug it more.

-Abhishek

[Weiqun_Z_]

Thanks for looking into this!  The reason turns out to be `-DNDEBUG`, not `-O0` vs. `-O3`.  We added `-DNDEBUG` in our make sytem when compiling for non-debug build.  If `-DNDEBUG` is not used, it can also be fixed by linking with `/opt/intel/inteloneapi/compiler/latest/linux/lib/libsycl-glibc.o`.  Similar things happen for cmath functions too, and it can be fixed by linking with `libsycl-cmath.o`.  I am using oneAPI beta4.

Weiqun

[AbhishekD_Intel]

Hello,

Glad to know your issue is resolved, we will look at your resolution.

-Abhishek

[Weiqun_Z_]

Thank you!  The thread can be closed now.

[AbhishekD_Intel]

Thank you for your confirmation, you are always welcome to post a new thread if you have any issue.

-Abhishek