Intel® Embree Ray Tracing Kernels
Discussion forum on the open source ray tracing kernels for fast photo-realistic rendering on Intel® CPU(s)
279 Discussions

Performance issues with TBB 2021.1.1

Mike_M_6
Beginner
2,797 Views

have an embree-3.12.2-based application (built on ubuntu 18.04) where if I update tbb from 2020.3 to 2021.1.1, the throughput performance drops significantly (I would estimate up to an order of magnitude slower) when under load. It is very consistent and obvious.

Both tbb and embree are rebuilt locally from scratch. The only application source code changed during the migration is the removal of the usage of tbb::task_scheduler_init during the initialization phase, which I have understood was effectively non-functional anyway.

Any ideas/recommendations?

If it helps here's what embree logs under 2021.1.1:

Embree Ray Tracing Kernels 3.12.2 ()
Compiler : GCC 7.5.0
Build : Release
Platform : Linux (64bit)
CPU : Xeon Sky Lake (GenuineIntel)
Threads : 36
ISA : XMM YMM ZMM SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 POPCNT AVX F16C RDRAND AVX2 FMA3 LZCNT BMI1 BMI2 AVX512F AVX512DQ AVX512CD AVX512BW AVX512VL
Targets : SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AVXI AVX2 AVX512
MXCSR : FTZ=1, DAZ=1
Config
Threads : default
ISA : XMM YMM ZMM SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 POPCNT AVX F16C RDRAND AVX2 FMA3 LZCNT BMI1 BMI2 AVX512F AVX512DQ AVX512CD AVX512BW AVX512VL
Targets : SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AVXI AVX2 AVX512 (supported)
SSE2 SSE4.2 AVX AVX2 (compile time enabled)
Features: intersection_filter
Tasking : TBB2021.1 TBB_header_interface_12010 TBB_lib_interface_12010

0 Kudos
10 Replies
FlorianR_Intel
Employee
2,794 Views

Hi,

thanks for reporting this. A performance regression of an order of magnitude sounds really serious. Is this affecting the BVH build performance or the BVH traversal performance (ray intersection) or both?

Can you please double check your results and make sure that you build in release mode both times? I'll try to reproduce this on our side in the meanwhile.

Cheers 

0 Kudos
Mike_M_6
Beginner
2,784 Views

Thanks for the reply.

The BVH build performance is not obviously a problem, but ray intersection is clearly a bottleneck.

Now that you mention it, the cmake command used for the 2021.1.1 build was:

cmake ../oneTBB-${TBB_VERSION}/ -DCMAKE_CXX_STANDARD=14 

... whereas previously that command didn't exist because the build wasn't cmake based -- just a simple make invocation.

I will try:

cmake ../oneTBB-${TBB_VERSION}/ -DCMAKE_CXX_STANDARD=14 -DTBB_TEST=OFF -DCMAKE_BUILD_TYPE=Release

 ... and see if that makes any difference.

That said, the default build type should work in theory... 

0 Kudos
FlorianR_Intel
Employee
2,774 Views

The default build type is not equivalent to Release. If you don't specify a build type there will be no optimization flags used. See, for example, https://blog.kitware.com/cmake-and-the-default-build-type/

 

I've run some tests specifically comparing tbb 2020.3 and oneTBB 2021.1.1 on two different machines (CentOS 7.4, gcc, dual socket Xeon and Ubuntu 2020.4, clang, 16 core Skylake)  and I see no performance impact on the ray tracing performance.

 

My hypothesis is that you have some issue in your config/build setup. If you still think that this is an Embree issue, please provide some more details about your machine and the compiler you use.

 

BTW: With oneTBB you can now build TBB and Embree conveniently by adding them as subdirectories/submodules to your project and use add_subdirectory in CMake. Just be sure to first add_subdirectory(path_to_tbb) and then add_subdirectory(path_to_embree). TBB then defines CMake targets that will be picked up and used by Embree for compilation and so forth.

 

Cheers

0 Kudos
FlorianR_Intel
Employee
2,771 Views

Sorry, ignore my comment about the default build type. I did not see the link you provided.

0 Kudos
Mike_M_6
Beginner
2,765 Views

Interestingly, my ray-tracing unit tests crash now having switched to the `Release` tbb build configuration.  These tests have been used for a couple of years now -- I've not seen them crash like this before.  The plot thickens...

0 Kudos
Mike_M_6
Beginner
2,729 Views

I have a parallel discussion of the issue going on the TBB github site.  This comment I posted there in particular might be useful information here as well:  https://github.com/oneapi-src/oneTBB/issues/355#issuecomment-790674138

I am fairly convinced now this is not a build issue.

0 Kudos
FlorianR_Intel
Employee
2,727 Views

OK, thank you!

This seems to be a TBB issue fundamentally.

Are you fine with closing this thread and continue the discussion on the TBB github issue tracker?

0 Kudos
Mike_M_6
Beginner
2,719 Views

Ha, that's a complicated question... 

About that unit test crash that I mentioned earlier...

*That* problem appears to be related to the use of ISPC 1.15.0.  When I revert back to ISPC 1.13.0 (and leaving everything else unchanged), the crash goes away.  I verified the crash occurs in our ISPC code but little insight into why it crashes exactly.  These unit tests have not changed significantly in months, and the code has been very stable in production for longer.

I did not mention the ISPC version until now because I didn't think it was pertinent -- we upgraded to ISPC 1.15.0 at the same time as the TBB 2021.1.1 upgrade.  I verified that the TBB behavior was the same with and without the ISPC change and so I assumed it was not an issue by itself.  And then that unit test started crashing and now I don't know what to think...

I'm sure at this point you're probably thinking I must be some kind of idiot but I'm really trying to be as diligent about this as possible.   It's true that we changed two dependencies (TBB and ISPC) at the same time, but reverting only one of the dependencies seems to only fix about half of the issues.  If I revert both, (it appears) all of the issues are resolved.

For the time being, I'm reverting back to ISPC 1.13.0 because I can only deal with one problem at a time (assuming there are two problems!).  But eventually, it would be nice to get to the bottom of the ISPC issue as well...

0 Kudos
Mike_M_6
Beginner
2,717 Views

I see now in the TBB issue thread linked above that the unit test crash might be related to issues with the TBB Release build configuration -- sticking to the default ReleaseWithDebugInfo configuration is the recommended workaround.  When/if I get some time/energy to investigate, I will look into this further.

With that, it is probably okay to close this thread.  

Thanks for your attention.

0 Kudos
FlorianR_Intel
Employee
2,714 Views

Thanks! We will check if we can reproduce issues with ISPC 1.15 on the Embree side.

0 Kudos
Reply