Performance issues with TBB 2021.1.1

[Mike_M_6]

I have an embree-3.12.2-based application (built on ubuntu 18.04) where if I update tbb from 2020.3 to 2021.1.1, the throughput performance drops significantly (I would estimate up to an order of magnitude slower) when under load. It is very consistent and obvious.

Both tbb and embree are rebuilt locally from scratch. The only application source code changed during the migration is the removal of the usage of tbb::task_scheduler_init during the initialization phase, which I have understood was effectively non-functional anyway.

Any ideas/recommendations?

If it helps here's what embree logs under 2021.1.1:

Embree Ray Tracing Kernels 3.12.2 ()
Compiler : GCC 7.5.0
Build : Release
Platform : Linux (64bit)
CPU : Xeon Sky Lake (GenuineIntel)
Threads : 36
ISA : XMM YMM ZMM SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 POPCNT AVX F16C RDRAND AVX2 FMA3 LZCNT BMI1 BMI2 AVX512F AVX512DQ AVX512CD AVX512BW AVX512VL
Targets : SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AVXI AVX2 AVX512
MXCSR : FTZ=1, DAZ=1
Config
Threads : default
ISA : XMM YMM ZMM SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 POPCNT AVX F16C RDRAND AVX2 FMA3 LZCNT BMI1 BMI2 AVX512F AVX512DQ AVX512CD AVX512BW AVX512VL
Targets : SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AVXI AVX2 AVX512 (supported)
SSE2 SSE4.2 AVX AVX2 (compile time enabled)
Features: intersection_filter
Tasking : TBB2021.1 TBB_header_interface_12010 TBB_lib_interface_12010

[FlorianR_Intel]

Hi,

thanks for reporting this. A performance regression of an order of magnitude sounds really serious. Is this affecting the BVH build performance or the BVH traversal performance (ray intersection) or both?

Can you please double check your results and make sure that you build in release mode both times? I'll try to reproduce this on our side in the meanwhile.

Cheers 

[Mike_M_6]

Thanks for the reply.

The BVH build performance is not obviously a problem, but ray intersection is clearly a bottleneck.

Now that you mention it, the cmake command used for the 2021.1.1 build was:

cmake ../oneTBB-${TBB_VERSION}/ -DCMAKE_CXX_STANDARD=14 

... whereas previously that command didn't exist because the build wasn't cmake based -- just a simple make invocation.

I will try:

cmake ../oneTBB-${TBB_VERSION}/ -DCMAKE_CXX_STANDARD=14 -DTBB_TEST=OFF -DCMAKE_BUILD_TYPE=Release

 ... and see if that makes any difference.

That said, the default build type should work in theory... 

[FlorianR_Intel]

The default build type is not equivalent to Release. If you don't specify a build type there will be no optimization flags used. See, for example, https://blog.kitware.com/cmake-and-the-default-build-type/

I've run some tests specifically comparing tbb 2020.3 and oneTBB 2021.1.1 on two different machines (CentOS 7.4, gcc, dual socket Xeon and Ubuntu 2020.4, clang, 16 core Skylake)  and I see no performance impact on the ray tracing performance.

My hypothesis is that you have some issue in your config/build setup. If you still think that this is an Embree issue, please provide some more details about your machine and the compiler you use.

BTW: With oneTBB you can now build TBB and Embree conveniently by adding them as subdirectories/submodules to your project and use add_subdirectory in CMake. Just be sure to first add_subdirectory(path_to_tbb) and then add_subdirectory(path_to_embree). TBB then defines CMake targets that will be picked up and used by Embree for compilation and so forth.

Cheers

[FlorianR_Intel]

Sorry, ignore my comment about the default build type. I did not see the link you provided.

[Mike_M_6]

Interestingly, my ray-tracing unit tests crash now having switched to the `Release` tbb build configuration.  These tests have been used for a couple of years now -- I've not seen them crash like this before.  The plot thickens...

[Mike_M_6]

I have a parallel discussion of the issue going on the TBB github site.  This comment I posted there in particular might be useful information here as well:  https://github.com/oneapi-src/oneTBB/issues/355#issuecomment-790674138

I am fairly convinced now this is not a build issue.

[FlorianR_Intel]

OK, thank you!

This seems to be a TBB issue fundamentally.

Are you fine with closing this thread and continue the discussion on the TBB github issue tracker?

[Mike_M_6]

Ha, that's a complicated question... 

About that unit test crash that I mentioned earlier...

*That* problem appears to be related to the use of ISPC 1.15.0.  When I revert back to ISPC 1.13.0 (and leaving everything else unchanged), the crash goes away.  I verified the crash occurs in our ISPC code but little insight into why it crashes exactly.  These unit tests have not changed significantly in months, and the code has been very stable in production for longer.

I did not mention the ISPC version until now because I didn't think it was pertinent -- we upgraded to ISPC 1.15.0 at the same time as the TBB 2021.1.1 upgrade.  I verified that the TBB behavior was the same with and without the ISPC change and so I assumed it was not an issue by itself.  And then that unit test started crashing and now I don't know what to think...

I'm sure at this point you're probably thinking I must be some kind of idiot but I'm really trying to be as diligent about this as possible.   It's true that we changed two dependencies (TBB and ISPC) at the same time, but reverting only one of the dependencies seems to only fix about half of the issues.  If I revert both, (it appears) all of the issues are resolved.

For the time being, I'm reverting back to ISPC 1.13.0 because I can only deal with one problem at a time (assuming there are two problems!).  But eventually, it would be nice to get to the bottom of the ISPC issue as well...

[Mike_M_6]

I see now in the TBB issue thread linked above that the unit test crash might be related to issues with the TBB Release build configuration -- sticking to the default ReleaseWithDebugInfo configuration is the recommended workaround.  When/if I get some time/energy to investigate, I will look into this further.

With that, it is probably okay to close this thread.  

Thanks for your attention.

[FlorianR_Intel]

Thanks! We will check if we can reproduce issues with ISPC 1.15 on the Embree side.