INB-Lunch-Seminar

Time-of-Flight Imaging for 3D Scene Reconstruction in Computer Graphics

Christian Theobalt, head of research group "3D Video and Vision-based Graphics" at the Max-Planck-Institut Informatik,
Professor of Computer Science at Saarland University, Saarbrücken

Optical reconstruction of detailed 3D models of static and dynamic scenes is a traditional problem of computer vision. In recent years, however, the topic has been of increasing importance in the field of computer graphics, as well. Detailed shape and motion models of real world objects or persons are building blocks for the design and rendering of believable virtual worlds. In addition, 3D acquisition technology promises to open up new ways of interacting with a computer, and thus researching its algorithmic foundations is of great importance in computer vision and computer graphics alike.

In our lab, we have been working on development of the next generation of motion capture technology, which we call performance capture technology. The goal is to reconstruct detailed shape, motion and appearance models of dynamic scenes, in particular scenes with human actors wearing arbitrary apparel, from multiple input video streams. I will briefly show examples of this work in my talk. While the approaches we developed so far produce high quality results, they are constrained to operate in a controlled studio environment. Making performance capture technology or even static 3D shape scanning available to every users requires a new type of sensing technology that is easy to use and cheap to produce in high numbers.

In recent years, infrared based depth cameras have been developed that measure dense depth maps at video rate and do not suffer from the lack of robustness that many image-based approaches, like stereo, are subject to. Most of them measure the return time of infrared light that was emitted into the scene (Time-of-Flight) or use an active stereo approach (like the Microsoft Kinect). Unfortunately, these cameras typically have a rather low resolution, and suffer from heavy noise and systematic distortions. I therefore present several approaches that we developed to reduce the noise of these cameras and reconstruct static and dynamic scene geometry at much higher resolutions than the physical camera resolution. I will also show that it is feasible to use ToF cameras as handheld 3D shape scanners if the noise characteristics is properly modeled and exploited for reconstruction. Finally, I will briefly show how a single depth camera can be used for real-time performance capture.