Polyhedral hull online compositing system: Reconstruction and shadowing.

Master's thesis, Cornell University, August 2004.

A fundamental limitation of traditional two-dimensional compositing is that it lacks the spatial information necessary for realistically merging live video with simulated environments. We introduce a novel three-dimensional compositing system that leverages multiple camera viewpoints to generate a geometric model of the foreground object. The foreground object can then be merged with a virtual background environment in three-space. Using the three-dimensional definitions of the foreground and background geometries, we correctly handle occlusions and generate physically-based global illumination effects, including shadows and interreflections. Knowledge of the scene structure also removes the fixed camera constraint of 2D systems, and we allow the viewer to specify an arbitrary camera To demonstrate the feasibility of interactive 3D compositing, we have built a prototype system. Our system consists of four video cameras, four client computers to perform the image processing operations, and a server that executes the reconstruction algorithm, computes the global illumination effects, and generates the final rendered image. Both the reconstruction of the foreground geometry and the scene compositing occur in real-time, allowing our hardware and software system to merge live interactive video content with virtual worlds. This thesis focuses on the geometric reconstruction and shadowing algorithms used in our system. We compute the intersection of the silhouette cones from multiple cameras to calculate a polyhedral hull of the foreground geometry. We implement a shadow generation technique based on the penumbra map algorithm, and by leveraging the computational power of the GPU, we simulate soft shadows in hardware. Our system is capable of reconstructing geometry and casting believable shadows onto the surrounding environment at interactive frame rates.

This paper is available as a PDF file Let04.pdf ( 19M).