I did some code cleanup and testing.

The hole-filling and self-intersection checking algorithms are working correctly.

Known Issues

1. It appears that my octree structure is faulty, simply looping over triangles results in better intersection checks. However, I wont be spending time to debug for now.
2. The hole detection algorithm runs into issues with singular vertices. This causes flipped normals and overlapping triangles to appear. We decided to move on for now because maybe it wont occur much in practice.
3. An issue arises when using mesh self-intersection removal + hole-filling in tandem. Self-intersections are first removed, then hole-filling is used to repair the mesh. But hole-filling itself again introduces new self-intersections.

Moving Forward

We have these options:

1. Possibly detect self-intersecting parts of the mesh (after holefilling), remesh that part using voxelization with OpenVDB.
2. BVH may be used to create a hierarchical structure (finer voxels in parts requiring more detail).
3. We can possibly use the mesh refinement and mesh fairing algorithms suggested by Liepa to further improve the mesh.

from repair.