|Although modeling and animation software-and the images they create-tend to be the stars of the computer graphics industry, 3D scanning plays a quiet but integral role in the origin of many digital models, from props and maquettes to celebrities and stunt actors. Initially, the CG industry had been fairly slow to embrace the technology. Today, however, the use of 3D scanning is prevalent throughout the entertainment realm, and new applications continue to emerge.
Cyber-scanning technology was developed over two decades ago by Cyberware (Monterrey, CA) as a family hobby, the brainchild of a retired aerospace engineer, his artist wife, and their computer programmer son. “My dad wanted to make a kind of sculpting machine that measured the human face accurately and quickly,” says Cyberware vice president Steve Addleman, another son of the inventor. “The device was hooked to a computer-controlled milling machine that would carve an image of what was scanned.” A head scan took 17 seconds, and the resulting carved foam bust could be completed in a few hours.
What started as a small venture aimed at artists and sculptors took a huge leap forward thanks to powerful Silicon Graphics computers, which, for the first time, enabled the scanned data to be rendered as a surface. “You could actually see the data,” recalls Addleman. Soon after, Hollywood came knocking-in the form of Star Trek IV: The Voyage Home in 1986. For a special effects scene in which the heads of the bridge crew rolled out of the fog, a seamless look required computer graphics, and the project began with scans of the Star Trek crew’s heads. After that successful experience, Hollywood became a fairly frequent client of 3D scanning.
The US Air Force was one of the first groups to identify the value of digital scanning outside the entertainment realm. The Air Force worked with physical anthropologists to achieve precise measurements of the human body, to arrive at standards for helmets and suits. To achieve this goal, the military branch provided the money for Cyberware to develop a full-body scanner that could get the job done quickly-“a person can only stand still for 15 to 20 seconds,” Addleman points out.
|Service provider Gentle Giant used its Cyberware systems to acquire scanned data of actor Michael Caine (top of page) for the effects in the Austin Powers movie Goldmember. Image courtesy Gentle Giant.
The resulting full-body scanner comprised a precise motion system and four scan heads mounted onto horizontal arms on tall towers. The new device also scanned color and, per the Air Force’s requirement, was “portable” (large trucks could transport it).
With its focus on designing new 3D scanners, Cyberware transitioned from a service provider to a scanning equipment manufacturer; the company continues to sell customized scanners for a wide range of uses, from archaeology to industrial design and the military. The company’s products include the original head scanners, small- and large-object scanners, and the whole-body scanner, all priced between $20,000 and $400,000.
According to Addleman, there is a reason why Hollywood film-production companies and computer game developers utilize scanning services, rather than purchasing scanners themselves. “The scanners are very productive, so it’s more efficient for clients to buy the time rather than the scanner,” he says. A film, game, or TV commercial can take advantage of the specialized skills of a service bureau’s experienced staff without incurring the cost of the equipment. They can also benefit from the latest technology developments and customized solutions.
Some of the recent advancements in the technology include increased resolution and higher-quality texture maps. Increasing the measured points on a surface provides more detail-for instance, the valley of a wrinkle or the break of a lip. Visual effects clients also want high-quality texture maps with a full range of colors. “The original systems had one color value for every X, Y, Z location, or point,” explains Addleman. “Now, high-res systems have 16 times the number of color values, and they’re located on and in between every X, Y, Z location, which enables the image to bear much closer inspection. But it’s still not enough.”
While Cyberware proved the initial value of 3D scanning, other companies in recent years have fostered the technology’s growth and usage, particularly in the entertainment industry, by offering scanning services.
Upping the ante is researcher Paul Debevec at the Institute for Creative Technologies in Marina del Rey, California, where an image-based lighting technique is being applied to the human face. This process enables the capture of the human face from every direction that light can pass, which results in a perfect-fidelity image of the face. Debevec’s latest application enables the capture of the human face from arbitrary camera viewpoints and in performance, which Sony Pictures Imageworks used to generate digital faces in Spider-Man 2 and Weta Digital used for Naomi Watts’s face in King Kong.
|Below shows actress Jessica Vallot while she is scanned in “Light Stage 2” at USC’s Institute for Creative Technologies. Right is the virtual image of Vallot’s face. Pictures courtesy Paul Debevec.
Debevec says that although typical 3D scanning does indeed result in texture maps for the face and body, it imposes limitations. “If you map a person’s face onto nicely scanned geometry, you’ve wrapped a photo around the face,” he says. “You can only change that by editing out the effects of lighting and then resimulating them, which is difficult. This method captures shading that tells us how the face responds to light, independent of the illumination it’s captured in.”
Founded in 1995, Gentle Giant (Burbank, CA) got its start sculpting physical maquettes for animation studios and toy manufacturers. “We’d often sculpt a perfect likeness [of an object], take it into the studio for approval, and they’d say, ‘Great, but it needs to be 10 percent larger,’’’ recalls Steve Chapman, vice president of technology. “And we’d have to resculpt the entire thing by hand. Then it occurred to us that if we could digitally scan [the model] and then output it on a 3D printer (rapid-prototyping machine), that problem would be solved.”
In 1997, Gentle Giant bought a Cyberware scanner, and soon after, started investing in more 3D scanners and more printers. And, business boomed.
Whereas many scanning service bureaus serve several industries, 99 percent of Gentle Giant’s business is from visual effects facilities serving the motion-picture industry. According to Chapman, Gentle Giant’s work sometimes begins in the preproduction phase of a project, with the creation of designs and sculptures for characters; other times, the group becomes involved in the production phase, scanning actors and props on set.
How the team provides the resulting datasets to the VFX companies depends on the client. “[Industrial Light & Magic], for example, has an intrinsic work flow for dealing with scanned data, so we give them raw data,” says Chapman. “Other companies may not have the employees to process the data from raw points to something animatable, so we’ll do it for them.”
|Gentle Giant provided Sony Pictures Imageworks with digital scans for the studio’s effects work on Spider-Man 2. Image courtesy Sony Pictures. © 2005 Columbia Pictures Industries, Inc. Spider-Man charac
Rhythm & Hues recently had Gentle Giant scan actors, maquettes, and props for The Chronicles of Narnia: The Lion, the Witch and the Wardrobe. “We probably did 100 cyber scans for the film,” recounts Bill Westenhofer, the studio’s visual effects supervisor. “They did a lot of the polygon cleanup, and we got a high-res scan. Then we created a low-res model that’s specific to our needs for animation.
Processing the data is a simple matter of taking a random triangle layout and turning it into ordered polygons. “You ideally want to keep everything,” Chapman says, “but there are ways to make it more manageable, such as using displacement maps instead of polygons to define texture and bump maps.”
According to Chapman, the benefits of a production studio using 3D scanning go far beyond those of the film’s basic completion. “When Warner Bros. creates a Harry Potter, it’s not just creating a movie, but a marketing franchise that includes other ancillary products,” he says. “The same data used to animate Harry for the movie is also used for the video game as well as for the related toys and products.” For a VFX facility, though, the focus is on the job at hand. So it’s hardly surprising then, that even though Rhythm & Hues is a major effects studio, the company has no interest in bringing high-res scanning in-house, comments Westenhofer. “The meat of our work is effects,” he says. “There isn’t enough scanning [needs] for us to amortize the costs.”
Sony has also relied on Gentle Giant for scans the studio used while creating effects in Spider-Man, The Polar Express, Spider-Man 2, and, now, Spider-Man 3. Digital effects supervisor Peter Nofz reports that Gentle Giant usually delivers the data as high-res polygonal meshes in three formats, “to make sure that whomever needs it will have it.” He also requests the raw scan for cross-reference.
Nofz is enthused about what he calls “the next leap” in 3D scanning: instant photographic scans. “These photographic scanners will get the information and provide much more accurate meshes instantaneously,” he says. “I hoped we would see it for Spider-Man 3 but now it’s [likely we’ll have it for] Spider-Man 4.”
Like Cyberware, Belgium-based Eyetronics-founded in 1998-manufactures digital scanners. Infrascan, a body-scanning solution, derives a 3D model from a single image. To accomplish this, a person is placed in the da Vinci pose (standing with legs and arms fully spread), and a pattern is projected onto the subject while a camera takes a snapshot. Meanwhile, software automatically calculates polygonal data based on the pattern deformations on the person. The total scanning process takes less than five seconds, and measurements are calculated within 30 seconds. With the company’s Facesnatcher, designed for capturing face and hair details, two cameras take images while a beam projects multiple patterns onto the subject’s face.
When Eyetronics opened its Redondo Beach, California, office in 2000, vice president of operations Nick Tesi pursued industries as disparate as games and medical, providing systems rather than services. Eyetronics sold its software, which calibrated the X, Y, Z positioning of a projected grid, while users bought their own camera and slide projector. But the number of different cameras made that business model difficult to support, and the company subsequently switched to providing 3D scanning services.
In 2001, Eyetronics built a housing for the actual grid that contained just one camera-currently an 8.3 megapixel Canon EOS 20D that uses a flash that gives a 12-in. depth of field; a secondary flash provides a texture map. “It’s not a black box,” explains Tesi. “The hardware is an open system that allows us to bring in new technology as it becomes available.”
According to most vendors and users, the biggest challenge to using digital scanning is the postprocessing of the data. In the case of Eyetronics, it uses computers sporting 3 ghz processors, Nvidia Quadro FX cards, and 2gb of RAM. The end result is provided to a client as a file in any chosen 3D animation software, such as Autodesk Media and Entertainment’s 3ds Max or Maya format. Delivering a 3D model that has a 4k color map and is ready for animation makes the technology an easy sell to movie studios and game developers.
Eyetronics used its ShapeCam device (inset) to capture the digital data of a pelican for the talking-animal film Racing Stripes. Other recent work includes scans for Charlie and
The three markets that Eyetronics serves are films, games, and broadcast. Recent film work includes Batman Begins, Racing Stripes, and The Legend of Zorro. Tesi notes that Eyetronics has recently seen an increase in business on the computer gaming side as a result of the increased quality demands of next-gen gaming.
Continuing to raise the bar in digital scanning and the technology’s profile in the marketplace, Eyetronics has developed a system for facial performance capture, which was used by Café FX to create the Invisible Man in The League of Extraordinary Gentlemen and by VFX supervisor Jeff Okun to generate a digital stunt double of Tom Cruise in The Last Samurai. Also, the Motion Picture Company utilized the system to create multiples of actor Deep Roy for the oompa loompas in Charlie and the Chocolate Factory (see “Eye Candy,” August 2005, pg. 16).
At Café FX, technical animation supervisor Domenic Di Giorgio recalls that, to create the Invisible Man, the group first tested marker-based face-capture techniques but found they didn’t give the level of detail needed to faithfully portray the actors’ mannerisms. “After testing with a marker-based facial capture system, they ran the same test with an Eyetronics scan, and we saw the level of detail they could achieve,” he says. “It was our only alternative, and Eyetronics worked closely with us to refine its technology to meet our needs.”
Since then, Café FX has used Eyetronics-scanned assets for work on Zathura and Blade: Trinity. “The meshes are clean,” says Di Giorgio. “And they can dial the resolution up or down, which is handy for rigging and animation.”
Another new Eyetronics device captures full-body, low-res scans of real actors to create a large number of digital background actors. Tesi notes that the rotating device takes 1.5 minutes to scan each character, complete with texture maps. “We’re constantly looking for ways to make the technology better and faster, and the results more realistic,” says Tesi.
Fifteen years ago, the National Research Council of Canada developed 3D scanning based on auto-synchronous laser technology. Originally created for museums and scientific applications, the technology also fell into the hands of Helmut Kongl who, in 2001, founded XYZ RGB in Ottawa, Canada. And like many scanning companies, it, too, focused on the entertainment industry.
The company’s first scanning job was for ESC Entertainment’s The Matrix Reloaded and The Matrix Revolutions, but it has only been recently that computer power has reached a point where it can support the newer scanning capabilities. George Borshukov, now a computer graphics supervisor at Electronic Arts, worked with XYZ RGB on the two Matrix movies. “At the time, we got high-res scans of maquettes, and it was really novel-you take risks when you’re doing innovative work,” he says. “Yet, we knew that without detail in our facial scan, we wouldn’t have been able to produce the necessary images.”
Three years ago, XYZ RGB began work on King Kong, which features a digital gorilla (see “Long Live the King,” January 2006, pg. 16). The first model of the ape contained six million polygons; the last one, for Weta Digital, had 28 million polygons (split into four chunks of seven million polys, perfectly registered and spaced). Additional film work includes the hero creatures for The Lord of the Rings: The Return of the King, a CG Halle Berry for Catwoman, and other virtual characters for Aeon Flux and Batman Begins. Movie projects currently in the pipeline for XYZ RGB are Skinwalkers, Super Ex-Girlfriend, and 300.
|Electronic Arts, with the help of XYZ RGB scans, generated these CG models for the computer game Fight Night Round 3. Images courtesy Electronic Arts Worldwide Visualizations Group.
XYZ RGB’s 3D scanning output can be generated as NURBS, polygons, subdivision surfaces, or solids, and at resolutions from video to 2k with what Kongl describes as “near-microscopic resolution.” Though VFX companies were the first clients, more recently, says Kongl, gaming companies have shown stronger interest; work in that genre includes Fight Night Round 3 from Electronic Arts, in addition to titles from Sony and Psygnosis. At Electronic Arts’s Worldwide Visualization Group, Borshukov has contracted XYZ RGB to scan R&D prototypes for advanced gaming techniques. “We get the raw data and then extract normal and displacement maps,” he says, explaining that the iterative R&D process makes raw data imperative. But take scanning in-house? No way, says Borshukov. “It makes a lot of sense for us to keep this expertise out of house so we can focus on day-to-day operations,” he says.
Indeed, the gaming industry was slow to pick up on the technology, but with the next-generation consoles, you’ll be able to push so much data and detail through a pipeline that the developers won’t be able to create that kind of detail by hand anymore, Kongl maintains. Kongl has also seen an increased use of scanning for TV commercials and music videos.
XYZ RGB provides scanned data in a number of formats and resolutions, including 2k, which is ideal for its work on films like King Kong. Image © 2005 Universal
XYZ RGB now uses nine different scanners, such as the very high resolution model, which, at 10 tons, is the in-house system for maquettes and inanimate objects. For off-site work, the company uses a lower-resolution, structured light scanner, which projects a pattern of light (from a 50-watt halogen source) onto the face that is recorded from two points of view and resolved into a 3D shape. PixelLock is the company’s proprietary system for achieving accurate pixel-to-object registration.
Yet, choosing the proper scanner is only part of the solution. “I’ve seen people do exceptional work with medium to low-end scanners because they know the limitations of the technology and get the most out of it,” he says. “I’ve also seen people do the exact opposite. Even the best scanner technology will only get you 70 percent of the way there, and you can make or break any project in that last 30 percent. It’s about process, software, and experience.”
Cyber F/X established its scanning business in Burbank during 1992, first with a head scanner and later with a model scanner for reverse engineering. The original idea behind the business, which was first called Cyberscan, was the creation of miniature bride and groom dolls for atop wedding cakes, says president/founder Dick Cavdek. When the firm’s scanner provider, Cyberware, said it would send Hollywood clients Cyber F/X’s way, Cavdek was skeptical. “Lo and behold, they were serious,” he says.
When the producers of Batman & Robin came calling, Cavdek was ready: He had just designed a milling machine that could hold a block of polyurethane big enough to cut an object the size of a human being. Cyberware scanned Arnold Schwarzenegger (Mr. Freeze), and Cyber F/X created the resulting lifecast model used to design the character’s costumes without the actor having to be present at all the fittings. “Now I scan actors weekly,” Cavdek says. “We’ve worked on hundreds of movies.”
One area of Cyber F/X’s business is to create mannequins for a movie’s wardrobe department, such as those representing Angelina Jolie and Brad Pitt in Mr. & Mrs. Smith. The rest of the work is for visual effects facilities, including Digital Domain, ILM, and Pixar, among others. According to Cavdek, nearly every feature film needs some actor or prop scanned. “People should scan actors as insurance for bond completion,” he says. “Just imagine how much a dataset of Marilyn Monroe or Elvis would be worth today.”
Recently, Cyber F/X purchased an ATOS II from German company GOM for high-resolution and highly textured/colored scanning of static objects. More generally used for industrial applications, the ATOS II uses a 64-bit computer to process the data and can be operated with a notebook computer via FireWire.
Founded in 2004, Realscan 3D relies on a blend of proprietary pipeline and hardware and software developments for its own structured light 3D scanning solution. Founders Joel Thornton and Frank De Marco came from a background in structured light scanning and wanted to create their own solution.
To this end, the Los Angeles-based company offers a portable Realscan bundle that can be assembled in 20 minutes and broken down in 15 minutes, for fast work on a television or movie set. An 8 megapixel color capture system and high-resolution geometry enables accurate data and texture-map acquisition. The Realscan scanning hardware and pipeline eliminates baked-in specular highlights for better capture of difficult materials such as metals and other reflective surfaces.
Recent work using Realscan’s solution includes 11 digitized models for Asylum VFX’s work on Sky High, including those of actors Kurt Russell and Kelly Preston. All the actors and props were captured on set.
|Using its proprietary structured light scanning solution, Realscan 3D generated this CG model of its signature Samurai Girl. Image courtesy Realscan 3D.
“We saved a lot of time in texturing and modeling,” says Asylum VFX visual effects supervisor Mitch Drain. “We not only got an animation-ready model, but full-surface textures as well. Since we have this kind of technology available, there’s no need to model the human form in the likeness of an actor; we could have spent six to 10 weeks just coming up with the textures.”
Other recent jobs include animal digital doubles for Underdog and Air Buddies, Jerry Bruckheimer Films’ Déjà vu and National Treasure, FOX Television’s American Idol, and work for Nintendo and Midway Games, recently scanning Dwayne “The Rock” Johnson for the upcoming SpyHunter: Nowhere To Run title, using its RealCapture System.
Scanning has certainly developed over the last 10 years from an exotic and not-always reliable service to a commodity business. Now, with a range of high-quality scanners to choose from, service bureaus distinguish themselves by their quality of service as well as by the quality of their scans.
So, what does the future hold for 3D scanning? At Gentle Giant, Chapman predicts it will be a meld of scanning and motion capture. “A lot of new VFX-driven movies are having the actors do their performance statically, standing still so just their faces are captured, and then applying that to a moving 3D model,” he says. “The key difference is that you’re not just getting key points of data to drive the model; you’re getting a 3D model for every frame of film with the actor’s precise expression.”
And everyone is looking down the pike to see how to make their services better and faster. Whereas visual effects-driven movies have always propelled scanning services forward, that might change with the next-generation game platforms, as everyone in the scanning service industry knows that the Xbox 360 and PlayStation 3 may create a powerful new trend of 3D scanning for games-a prospect the companies are ready for.
Debra Kaufman is a freelance writer in the entertainment industry. She can be reached at email@example.com.