The advances in the manufacturing of depth sensors and camera technologies, such as LIDAR and RealSense cameras, have brought three-dimensional (3D) applications to the front stage in nearly all domains of the industry. Reconstructed 3D surfaces from multiple images appear in applications ranging from recreational, such as construction of realistic game avatars, to industrial, such as robotic assisted micro surgery as well as biometric measurements in medicine and security. The recovery of a 3D surface from depth information remains a long standing difficult problem in computer vision, one for which no off-the-shelf solution yet exists for objects found in natural scenes. Difficulties in surface reconstruction stem primarily from missing information caused by self-occlusion of complex, non-convex objects, but also emanate from variation in noise levels and illumination.

Facial recognition with 3D

Hardware for three-dimensional surface reconstruction

Depth cameras are equipped with two or more light sensors placed at a small distance apart, and an optional infrared projector. The depth in a scene is computed by triangulation, incorporating the principles of stereo vision. The output depth image provides the distance of all objects in the scene from the sensors’ axis. Commercial depth cameras now offers depth information ranging from 0.2 to 10 m, and provide a much richer data than that of a simple light camera. For real-time depth acquisition, the sensor can be thought of as a scanner, one for which a reliable acquisition of a surface depends on the user’s ability to scan all of its obscured corners. In contrast, when a single depth image is acquired, the problem of surface hole-filling becomes a difficult one. Self-occlusion in complex, non-convex, 3-dimensional objects cannot be reconstructed with accuracy and the missing information needs to be interpolated by algorithmic means.


Solutions for three-dimensional surface reconstruction

Surface reconstruction is a challenging problem, and over the years many solution have been suggested, using methodologies from computer vision and machine learning to complete these missing data. The most straightforward approach utilizes surface interpolation, which gives an inaccurate solution for large gaps in the surface. Other methodologies involve the well-known Poisson surface reconstruction, which requires a dense point cloud of orientation vectors to perform well. Despite this limitation, the Poisson surface reconstruction can be used as a framework to reconstruct surfaces for uniform and non-uniform sampling, a situation which is frequently encountered in natural depth acquisition. In the heart of the algorithm lies a minimization process, which attempts to recover a smooth gradient map from the projected point cloud acquired using the depth sensor.

The ripening domain of machine learning provides alternative means to the Poisson surface reconstruction and it enables to simulate empirical expert knowledge in hole-filling. These methods learn the rules of smooth completion of holes, incorporating knowledge of the observed point cloud. One of the most common pipeline for such learning is one for which the depth image (the so-called 2.5D image) is first encoded using 3D primitives (simple spatial shapes) and it reduces the problem surface completion problem into a lower dimensional space, in which constraints on the smoothness of the reconstructed surface can be fulfilled. The completed space is then decoded back to recover the original scene, only this time with holes filled by a primitive shape, or a combination thereof, to the original scene.

At RSIP Vision we tackle the hard problems in computer vision utilizing state-of-the-art tailor-made solutions for our clients. Our engineers stand in the front line of technology to bring our customers the best, most effective and accurate solution to their projects’ needs. To learn more about RSIP Vision’s activities over the past decades, please visit our projects page. To follow the latest trends in computer vision worldwide, subscribe for free to our monthly magazine, Computer Vision News.

Share The Story