I am using Embree to compute all the intersection along a ray path with a triangular mesh. I do this using a ray filter function that adds every intersection met to a list of intersection. This works fine for most of the cases and using the Pluecker intersection function, the intersection process is watertight, which is a very good thing in my application.
However, when a ray hits an edge between two triangles, sometimes, the ray/triangle intersection function finds two hits, one for each triangle T1 and T2.
My guess is that minUVW >= ulp * (U + V + W) for T1 and maxUVW <= ulp (U + V + W) for T2, which results in both the intersection being valid. In the resulting hits, tfar is around 10²~10³ and u is around 10⁻⁴ for T1 and -10⁻⁴ for T2.
In this specific situation, I could simply chose on of the two intersections, but I need to know whether the intersections are indeed on the same edge, in which case I need to select only one. Otherwise, I would need to keep both the intersections.
My current idea is, for both hits, to select which of the three barycentric coordinates has the lowest absolute value, x and y. Then I would check if x and y are of differents signs, which would make it very likely that the intersections happened to be on the same edge.
However, I'm not sure of the robustness of such a solution. I'm guessing it would not be very efficient around vertices, but I could live with that.
What are your thoughts about this problem?