DIANA PHILLIPS MAHONEY
Image-based rendering is be coming an increasingly popular method for generating virtual interactive representations of natural scenes quickly and efficiently. Rather than building models from scratch using geometric primitives, image-based techniques use information such as depth and color gleaned from photographs to describe complex 3D environments. But for the many applications that stand to benefit from an image-based approach-specifically those at tempting to offer interactive experiences in photorealistic settings-the limitations of today's graphics hardware are a significant obstacle.
Because of this, researchers at the University of North Carolina at Chapel Hill are developing a rendering engine specifically suited to the image-based approach, bypassing the use of conventional graphics systems altogether.
Called the WarpEngine, the new architecture is designed to produce high-resolution image-based renderings of natural scenes at interactive rates. Developed by graphics researcher Voicu Popescu and his UNC colleagues, the system implements in hardware an image-based technique called 3D warping.
With 3D warping, pixels enhanced with depth and color information are projected, or warped, to a target image. Simply warping depth and color samples, however, does not produce a high-quality result. "The output image needs to be carefully reconstructed from the warped samples," says Popescu. Unfortunately, existing reconstruction algorithms implemented on conventional hardware are either too expensive, jeopardizing interactivity, or so imprecise that the quality of the warped images quickly degrades as the camera moves away from the original view. In contrast, the Warp Engine uses a "forward mapping" algorithm that accurately reconstructs the final image while requiring fewer mathematical operations.
Although conventional graphics hardware can be used for image-based rendering using warping techniques, the computational demands of the technique significantly compromise performance. For example, to achieve a target resolution of 1280 by 1024, the hardware needs to process more than 150 million triangles per second in order to sustain a frame rate of 30hz, which, says Popescu, "is outside of the capabilities of today's highest-end graphics engines."
The WarpEngine has been designed specifically to meet these computational demands. The architecture consists of multiple identical nodes that render the 3D scene in parallel. To do this, it first partitions the reference images into sample tiles, then runs visibility tests on the tiles to determine which ones should be used for the destination image. Next, the tiles are grouped according to the screen region to which they map, and processed by the node responsible for the respective regions. Each node's 256 processors warp and interpolate all of the samples in parallel, then deliver the resulting subsamples to the warp buffer. Additional visibility tests are implemented, notes Popescu, "to ensure that if two samples warp to the same output image pixel, the one that is closest to the viewer is kept."
 |
| Image-based rendering takes flight with the WarpEngine, through which depth and color information taken from photos, in this case of a helicopter, are mapped to create a digital version that's ready for interaction. |
After all of the tiles of a certain screen region are processed, a reconstruction buffer computes the final pixel values.
The rendering process can be implemented using either the retained mode, in which a fixed set of reference images describe an environment, or the immediate mode, in which new reference images are received on the fly.
 |
| "EuroTown at Night" was rendered with the WarpEngine to illustrate the capabilities of the architecture, which was designed specifically to meet image-based challenges. |
The retained mode offers better frame-to-frame coherence, since most of the areas that a user may wander into have been rendered, so their respective reference tiles are already stored in the WarpEngine's local memory. In the immediate mode, frame-to-frame coherence suffers, because the communication bandwidth requirements between the host, which provides the reference image tiles, and the WarpEngine are greatly increased. On the plus side, the immediate mode offers the possibility of exploring dynamic scenes.
The primary advantages to the WarpEngine are its manageable storage and bandwidth requirements, which enable considerably higher performance than that which can be attained using conventional hardware. However, the architecture does have some performance limitations. Specifically, the rendering and update rates that can be achieved are constrained by the number of tiles per second that can be warped and interpolated, and the number of regions per second for which final pixel values can be reconstructed and forwarded to the frame buffer.
Currently, Popescu and his colleagues are considering ways to enhance the WarpEngine rendering capabilities. Among the possibilities are techniques for generating view-dependent effects and for changing the original lighting conditions of the reference images. The researchers are also working on ways to improve the tile-choosing process to limit inconsistencies among samples of the same surface seen in several reference images.
Once the technology has been refined, says Popescu, "we expect that the WarpEngine, coupled with image-based modeling or real-time depth imaging will render images that look truly photorealistic, leading to a dramatically heightened sense of presence for applications like visual simulation and telepresence, and enabling entirely new applications, such as 3D TV."
Though the WarpEngine was developed as a standalone system, says Popescu, "an attractive alternative use for the WarpEngine architectural ideas is in a hybrid geometry/image-based rendering machine that uses images rather than polygonized models."