Odd results on Mac, fixed via MallocNanoZone=1 environment variable

[Gary_B_]

I wouldn't normally post something as vague as this, but I'm getting close to moving on without understanding why this fix works.

My support dept reported anomalies in a couple of models using a build on Mac OS 10.10.3 with Embree enabled. Debugging the problem, I found it went away whenever I launched the program via the Xcode debugger, but when launched from the Mac finder the problem consistently reappeared.  I finally tracked down the only (meaningful) runtime difference, which is that Xcode adds an environment variable "MallocNanoZone=1" when launching our app which uses Embree within a child XPC service.  Adding this environment variable to the XPC plist so that it always applies fixes things.

I'll attach two images, before and after.  The windows in the model are modeled as a closed volume that has a fairly small distance between the front and back polygons.  I thought it might be a precision problem, because our old ray casting code used doubles and had no problem with this model.

I'm not sure what other info would be helpful beyond the actual program and model source.  This is the configuration of my Embree build:

Embree Ray Tracing Kernels 2.6.1 (Aug 13 2015)
 Compiler : CLANG 6.1.0 (clang-602.0.53)
 Platform : Mac OS X (64bit)
 CPU      : SandyBridge (GenuineIntel)
 ISA      : SSE SSE2 SSE3 SSSE3 SSE41 SSE42 POPCNT AVX 
 MXCSR    : FTZ=0, DAZ=0
 Config   : Release TBB4.3 TBB_header_interface_8006 TBB_lib_interface_8006 SSE4.1 SSE4.2 AVX AVX2 intersection_filter bufferstride 

[SvenW_Intel]

Could you send us the full output for rtcInit("verbose=2") for both runs. Maybe there is a different traversal kernel selected when using this MallocNanoZone=1 flag. Do you see any artefacts, e.g. Z-fighting when shading with UV-coordinates?

[Gary_B_]

I don't see any difference in debug output when I change MallocNanoZone outside of some minor timing differences.  The artifacts still show up when I set MallocNanoZone=0 for the XPC service and go away for MallocNanoZone=1

Now that you mention it, I do think I see z-fighting when interpolating the surface point from UV-coordinates, but only between adjacent triangles along a shared edge.  I'll need to take care of it if I ever decide to use that approach.

I've attached a single text file with both outputs included within it.

[SvenW_Intel]

Look like this is some precision problem. However, I have no idea why this MallocNanoZone flag changes the precision of some calculations.

You can easily check if the issue is caused by floating point precision by rendering a shot where the camera is much closer to the window. If the bug then disappears, the issue is likely related to floating point precision.