A bit side question, sorry if it's misplaced. I need to find all polygons intersections in scene with some "gap", e.g. if an edge is closed enough to a poly - count this case as intersection too. A code to check polygon/polygon intersection is not a problem, but which to which polygons I have to check? Can I use Embree BVH data structures for this? If so how?
You could use the User Geometry feature for this. Create some user geometry that implements the proper polygon intersection and returns bounds enlarged by the "gap" to Embree.
Using the User Defined geometry feature you write the ray intersection function for some primitive. Best look at the Embree tutorials to see how this works. You just have to calculate bounding boxes are that slightly larger than your polygons (such that Embree will also try to intersect them is they are some gay away of the ray) and pass your special ray/polygon intersection function to the User Geometry primitive.
Sorry, still unclear. Let me explain what I need with images. On first all walls are "good", poly edges are match in all places of walls contact. On second one the far wall (blue wireframe) is "bad" because there are no such edges on other walls. So I need to triangulate some red and green triangles to create needed edges. There is a some "gap". For example if bluе wall is slightly moved up I still need to break red triangles (if distance to floor is less than gap).
Anyway for potenyially large scene I need a data structure like bvh to detect which polygon breaks which. Thus I want to use Embree bvh - but don't know "how"