AVX preprocessor macro in bvh.cpp, bvh_statistics.cpp, primitive.cpp

[Wayne_H_]

Hello! I would like to know the purpose of some preprocessor macros that do specific template instantiation and function definition. To be specific:

kernels/xeon/bvh/bvh.cpp b/kernels/xeon/bvh/bvh.cpp

#if defined(__AVX__)
  template class BVHN<8>;
#else
  template class BVHN<4>;
#endif

kernels/xeon/bvh/bvh_statistics.cpp

#if defined(__AVX__)
  template class BVHNStatistics<8>;
#else
  template class BVHNStatistics<4>;
#endif

kernels/xeon/geom/primitive.cpp

  /********************** Bezier1v **************************/

#if !defined(__AVX__)
  Bezier1v::Type::Type () 
    : PrimitiveType("bezier1v",sizeof(Bezier1v),1) {} 
  
  size_t Bezier1v::Type::size(const char* This) const {
    return 1;
  }

  Bezier1v::Type Bezier1v::type;
#endif

and various similar Type () definition in this file

The reason to ask the question is we are using scons building system instead of the embree built-in cmake for our piepline. We noticed that while we are trying to move to embree 2.10.0 there are linkage errors when we link our codebase to an avx build embree:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHN<4>::preBuild(std::string const&)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::QuadMv<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHN<4>::set(embree::BVHN<4>::NodeRef, embree::BBox<embree::Vec3fa> const&, unsigned long)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHNStatistics<4>::bytesUsed() const'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHNStatistics<4>::str()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHN<4>::postBuild(double)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::QuadMiMB<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHN<4>::BVHN(embree::PrimitiveType const&, embree::Scene*)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::Object::type'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::SubdivPatch1Eager::type'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHNStatistics<4>::BVHNStatistics(embree::BVHN<4>*)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::TriangleMi<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::LineMi<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::SubdivPatch1Cached::type'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::Bezier1i::type'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHN<4>::clearBarrier(embree::BVHN<4>::NodeRef&)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::QuadMi<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::BVHN<4>::layoutLargeNodes(unsigned long)'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::TriangleMv<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::TriangleM<4>::Type::Type()'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::Bezier1v::type'
/rel/folio/embree_arras/embree_arras-2.10.0.x.0.0.2-1/lib/avx/libembree.so: undefined reference to `embree::TriangleMvMB<4>::Type::Type()'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For now we change the preprocessor macro in those files to avoid the linkage error, but it would be really helpful to know the reason that why cmake build works fine but scons build fail (we suspect there are some per file compile flags in cmake we didn't put in scons to cause this problem) so that we don't need to hack around the embree codebase to get the build working. Thanks a lot for the info.

Regards,

Wayne

[BenthinC_Intel]

Hi Wayne,

Something like

#if defined(__AVX__)

template class BVHN<8>;
#else
  template class BVHN<4>;
#endif
 

makes sure that we only enable a 8-wide BVH (BVHN<8>) when we compile for the AVX instruction set. In this case __AVX__ is defined. Embree is compiled with various code paths for different ISAs and selects the optimal code path at run-time, so it is important that all different versions of the same code but with different ISAs are compiled properly. Also BVHN<4> is basically the base BVH representation and BVHN<8> is an extension (if the underlying ISA supports it).

I'm afraid that I cannot help with the "scons" build system as I have no experience with it.

[Wayne_H_]

Hello Carsten: thanks for the reply. Does that mean intersector like Bezier1v, Bezier1i is not used in AVX ISA? (it's in the #if !defined(__AVX__) ) (kernels/xeon/geometry/primitive.cpp )

You mentioned that embree can be compiled with __AVX__ on but select __SSE__ codepath in the runtime if executing machine only has SSE instruction set. Doesn't that mean BVHN<4>, BVHN<8> should both be instantiated so that SSE ISA can fallback to BVHN<4>?

Thanks for the clarification.

[BenthinC_Intel]

Hi Wayne,

Been a while since I last looked at this but it the reason should be the following: Bezier1i is also supported in AVX mode, but the general type information for Bezier1i which are in this file are only generated for the base ISA (SSE mode = non-AVX mode). If you look at kernels/xeon/CMakeList.txt you will see that we compile the same file multiple times with different ISA configurations. We actually compile Embree with support for all supported ISAs in particular SSE,AVX,AVX2, and AVX512.

You are correct both BVHN<4> and BVHN<8> should be instantiated to allow a fallback if the machine does not support AVX.

Hope this helps.

[Wayne_H_]

Hello Carsten:

thanks a lot for the info! This does help a lot to know cmake compile actually compile same file multiple time to covers all the definition. (I did some nm and saw those missing definition are found in the cmake version build) We are trying to use scons to mimic the same behavior (compile certain file multiple times with different flag) and hopefully this can resolve our issue.

[HPC-TAMU]

I was trying to use EasyBuild to build some of the later Embrees and ran into this problem.  In the end, it turned out that my addition of -xHost on an AVX system interfered with the -x* options Embree uses at different points (e.g. -xAVX* vs -xSSE*).   In the end, the simplest solution was removing -xHost from CFLAGS before running cmake.

(FWIW, easyconfig for EasyBuild below)

Thanks for the clues in helping me understand the problem.

Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-perdue@tamu.edu    http://hprc.tamu.edu
HPRC Helpdesk: help@hprc.tamu.edu

<pre>

# Authors:: Jack Perdue <j-perdue@tamu.edu> - TAMU HPRC - http://hprc.tamu.edu

easyblock = 'CMakeMake'

name = 'Embree'
version = '2.10.0'

homepage = 'https://embree.github.io/'
description = """
 Embree is a collection of high-performance ray tracing kernels, developed
 at Intel. The target user of Embree are graphics application engineers that
 want to improve the performance of their application by leveraging the
 optimized ray tracing kernels of Embree. The kernels are optimized for
 photo-realistic rendering on the latest Intel® processors with support
 for SSE, AVX, AVX2, and AVX512. Embree supports runtime code selection
 to choose the traversal and build algorithms that best matches the
 instruction set of your CPU.
"""

toolchain = {'name': 'intel', 'version': '2016a'}
toolchainopts = {'optarch': False, 'pic': True, 'openmp': True}

# https://github.com/embree/embree/archive/v2.10.0.tar.gz
source_urls = ['https://github.com/embree/embree/archive']
sources = [SOURCELOWER_TAR_GZ]

builddependencies = [('CMake', '3.5.2')]

dependencies = [
    ('tbb', '4.4.2.152', '', True),
    ('ispc', '1.9.1', '', True),
    ('libjpeg-turbo', '1.4.2'),
    ('libpng', '1.6.21'),
    ('freeglut', '3.0.0', '-Mesa-11.2.1'),
    ('libGLU', '9.0.0', '-Mesa-11.2.1'),
    ('libXmu', '1.1.2'),
    ('libXi', '1.7.6'),
]

separate_build_dir = True

preconfigopts = """
    ml -t
"""

configopts = """\
    -DTBB_INCLUDE_DIR=$EBROOTTBB/tbb/include\
    -DTBB_LIBRARY=$EBROOTTBB/tbb/lib/libtbb.so\
    -DTBB_LIBRARY_MALLOC=$EBROOTTBB/tbb/lib/libtbbmalloc.so\
    -DOPENGL_INCLUDE_DIR=$EBROOTMESA/include\
    -DOPENGL_gl_LIBRARY=$EBROOTMESA/lib/libGL.so\
"""

vmaj = version[0]

sanity_check_paths = {
    'dirs': ['bin/%%(namelower)s%s' % vmaj, 'include/%%(namelower)s%s' % vmaj,
             'lib64', 'share/doc/%%(namelower)s%s' % vmaj],
    'files': ['bin/%%(namelower)s%s/benchmark' % vmaj,
              'include/%%(namelower)s%s/rtcore.h' % vmaj,
              'lib64/lib%(namelower)s.so.%(version)s',
              'share/doc/%%(namelower)s%s/LICENSE.txt' % vmaj],
}

modextrapaths = {
    'PATH': 'bin/%%(namelower)s%s' % vmaj,
}

moduleclass = 'lib'
# EOF

</pre>