What's Old is New Again

To create the newborn but elderly Benjamin, Digital Domain immersed Brad Pitt’s performance in an aged version of a digital double. Creating a digital human is difficult enough, but two things made this particular task especially thorny. First, Pitt has one of the most recognizable faces in the world. Second, Digital Domain’s reenactment of Pitt’s performance stars in the first 52 minutes of the film, from the time the character is born until he is full grown. The audience must believe in Digital Domain’s CG character right from the beginning to become absorbed in the story, and must remain convinced.

Preeg jumps in with a laugh, saying, “I was deterred.”

Expressions
Fincher commissioned the first proof-of-concept tests in 2004, but research had begun at Digital Domain in 2002. Once preproduction on this film began, the studio started the process of replicating Pitt’s expressions by painting his face green using the special makeup devised by Mova for its Contour system. For two days, one before principal photography and one after, Pitt bit his lips, puffed his cheeks, looked up, down, right, left, raised an eyebrow, tightened his neck muscles, showed surprise, fear, anger, and so forth, while Mova captured these expressions at 24 frames per second using 28 camera.

Figuring out how to lock the individual frames to a reference pose fell to Digital Domain’s award-winning creative director of software, Doug Roble. “The head motion had to be removed from the motion-captured data, but all we had to work with was the data captured on his face,” says Roble.

Once the artists had a stable face, they began extrapolating Pitt’s individual motions from the data. It wasn’t an automatic process. They often had to make such decisions as, for example, whether Pitt always needed to move his mouth to raise his upper left eyebrow. And, they needed to add data where the Contour system stopped—the outer rim of his eye, the edges of his lips, his ears. At the end, they had a collection of thousands of little motions.

“This is where we move into emotion capture, not motion capture,” Ulbrich says. “We built a library of micro-expressions constructed from bits and pieces of Brad so we could assemble a performance of what Brad really does. We aren’t interpreting his performance. We have [encoded] the subtleties of how Brad’s brain drives his [facial] muscles.”

Light the Match “This was a nonlinear show,” says Steve Preeg, character supervisor. “One of the first things that happened was a collaboration between animation, compositing, and lighting. Our animation reviews were done using [Mental Images'] Mental Ray renders and the HDRs.” To light the head, the artists had to match lighting conditions in such disparate environments as the interior of an old-folks home, a front lawn in broad daylight, and the deck of a tugboat on a stormy sea. “The director and DP did a meticulous job in these locations, and our job was to be true to that,” says lighting supervisor Jonathan Litt. His team worked with the 360-degree HDR photographs taken on the sets, video footage of the small-body actors, and the survey data. “We completely reconstructed the scene,” Litt says, “including all the lighting apparatus, the bounce cards, the scrims, and other diffusers.” Then, rather than using illumination in the spherical HDR images to light the CG face directly, they placed the HDR data on the reconstructed lighting elements. Every shot had its own HDR setup.   “Even though reconstructing the set is one step more than we usually take, we went for it because the head was often moving and, in some cases, the lighting was subtle,” Litt says. “On one hand, we exercised artistic control. On the other, we wanted to start with the realistic falloff in the local environment.” The compositing team prepared the HDR images for lighting by projecting them in Digital Domain’s Nuke onto the 3D set geometry and then re-photographing the environment from the position of the head in the scene. “Once we were happy with the overall feeling of the lighting in previs, we exported the 3D Nuke scene into a form [Autodesk’s] Maya could read, and put each light in the HDR onto a card,” explains compositing supervisor Paul Lambert. “The lighters could tweak the cards and put objects in front of them.” The lighting artists would see a primitive version of the real set around Benjamin, with his digital head and rough hair rendered in Maya. “They had everything of significance affecting lighting on the head, including inert objects that blocked or broke up the light,” Litt says. “The kind of things that would be held by C-stands are floating around the head in the CG version.” The most difficult sequence for the lighters was the church tent revival sequence. On set, the DP had used hundreds of lightbulbs hanging from cords and two beauty light sources. Director David Fincher had filmed the sequence on a windy day. “The whole tent was shaking, and those cords were swinging,” Litt says. “We re-created the entire thing. The cords. Every single light source. There was something about the subtlety of the lights when some cords were swinging and others weren’t, and the changes in the camera angles. Without re-creating it, the shots weren’t up to snuff.” Often, the artists compared the result to lighting on the body actor in the plate, and having footage from all four cameras on set helped them understand how light acted within this shot and others. “[The CG Benjamin’s] face is a different shape, and his skin is different from the actor’s skin, but we became so familiar with the tonality of each actor’s skin, it was still useful,” Litt says. –Barbara Robertson

After all this, though, the crew had a facial animation rig with little bits of Pitt—albeit at his current age, not, as they needed, for the 60-, 70-, and 80-year-old faces. For each of those ages, modelers had created heads and faces from scans of wrinkle-faced maquettes. “The maquettes, built by Rich Baker, were the foundation for our work,” says Barba.

One problem. Baker had based the elderly heads on Pitt’s head, but the small actors playing the young Benjamin Button had smaller heads, and Digital Domain’s heads would need to fit on the small actors’ shoulders. The riggers wanted to give the animators a specific facial animation rig for each age. First, though, they did create a scalable rig for the heads to show Fincher various sizes for the different ages.

“Once he settled on the scales he wanted,” Preeg explains, “we retargeted the motion data, making reasonable guesses for how Benjamin’s jaw skin would move when he is 80 and so on. Then, we created a rig for each age.”

It took nine months to put the captured shapes into deformation rigs that animators could manipulate for each age. “Getting those shapes into the rig was the magic voodoo,” Barba says.
And that was only the beginning. Creating a digital human is not easy.

Timing is Everything
“We discovered that Benjamin not only had to move exactly like Brad does, but without Brad’s timing, he wouldn’t come across as Brad,” Barba says. “There was no turnkey solution to capture both.” That is, to simultaneously capture his expressions and his timing.

The plan was to have Fincher direct Pitt reading Benjamin Button’s lines on a soundstage after Fincher had finished shooting the small-body actors and the rest of the cast on location. The timing Digital Domain needed would come from that reading. “In order to use Mova for that, we would have had to put Brad in green makeup for seven days,” Barba says. “That wasn’t an option. With Brad in green makeup, it was hard for David [Fincher] to discern his performance.” Also, the Mova system wouldn’t mix well with the audio recording. “There are many hard drives,” Barba says, “and fans are whirring.”

Skin Deep To create Benjamin Button's skin, the lighting and shading crew combined characteristics from the three maquettes of Button at age 60, 70, and 80, and photographs of Brad Pitt. The maquettes, created by Rick Baker, provided details, such as age spots, that the artists used for texture maps. The silicon models also provided a starting point for shaders that would gave Benjamin's surface its realistic translucency through the scattering of light in the subsurface, sheen through specular lighting on the surface, and shape through such displacement detail as pores and wrinkles. "There's interplay between the characteristics of the surface," says Jonathan Litt, lighting supervisor. "The displacement affects the subsurface and specular lighting." During the look-development phase, the lighting team captured the surface characteristics of the maquettes using the Light Stage system at the Institute for Creative Technologies' Graphics Lab. This system surrounds a subject-in this case, the maquette-with multiple cameras that fire simultaneously. Usually this happens in changing light, but this time, that wasn't important. "We didn't want to invest too much into matching the maquettes exactly" Litt says. "We wanted to use them as a visual reference." Working from a set of photographs taken on the Light Stage with all 164 lights turned on, the crew at Digital Domain processed the images based on lighting data from the HDR photographs taken on location. Then, compositors projected them onto a basic model in Nuke. The Light Stage images provided the falloff and the intensity change caused by lights at various angles. The HDRs provided the light from the environments the crew needed to match. "Paul [Lambert] applied the lighting from the HDR environment map in a way that correlated to each of the lighting angles from Light Stage, and basically mixed hundreds of those images until the CG head looked like it was lit by the environment map," Litt explains. "The compositors would run this out in Nuke while calibrating the lighting setup for each sequence. That way, they could quickly see a lit preview head based on the Light Stage photos to know if they were in the right ballpark to calibrate the color for each sequence." Because the maquette was plastic, though, the CG model looked plastic, too. "[Light Stage] was a neat tool for development," Litt says. "We wrote shaders to emulate human skin based on combining photographs of Pitt and the Light Stage data from the maquettes, but the Light Stage data wasn't used directly in any final frames." Instead, the CG head used procedural shaders driven by texture maps for the skin and head. "We could light the procedural skin by any light, as if it were the real thing," Litt says. "You could plop the head down anywhere." Although Digital Domain's crew wrote specific shaders for this show, Litt is quick to deflect any special recognition. "I want to give full credit to the amazing work that's happened with skin shading in the last five years," he says. "It was a lot of work nailing down the skin shading, though. We pulled from industry-standard approaches to implement our own flavor in [Mental Images'] Mental Ray and tune how we used our texture maps to drive the shaders. But, this is not a young character with perfectly smooth skin. We had many an iteration tuned to what was happening in wrinkles and creases." –Barbara Robertson

Instead, as Pitt read the lines on the soundstage, Digital Domain captured his performance on video. Later, Image Metrics derived animation curves from that video. “It worked beautifully,” Barba says. “Brad was relaxed and gave us a great performance. We had him scheduled for 10 days, and he finished in seven. He was happy. David was happy.”

Here’s how it worked: Pitt reclined in an ab (exercise) machine on a soundstage, with his knees raised up to align his body. Around him, the crew positioned four Thompson Viper digital cameras that were constantly in sync and trained on his face. “His face was always in the center of the frame,” Barba says. “And cameras set at angles always gave us his profile.”

Although Image Metrics typically uses just one camera, Barba insisted on four. “With one camera, you can’t see the jaw from the profile or subtleties around the eyes and ears,” Barba says. Also, Barba wanted to give the animators all four views later for reference. On a nearby monitor, Pitt could see the edited footage from shots filmed on location. He wore an earbud to listen to the recorded scenes. “He’d look down, watch the footage, then say, ‘Great.’ David would call, ‘Action,’ and he’d do the line,” Barba says. “Then we’d show him the next clip.”

Preeg was impressed: “[Pitt] could watch a 40-shot sequence twice and then perform his lines strictly on the audio cues. I was concerned about his matching the performance months after the shoot to a person’s motion that was done 3000 miles away on set. But, I’ve got to say that Brad’s a lot more than a narrow waist and broad shoulders.”

After the performance capture and dialog session, Fincher inserted his selections of Brad’s performance into the edited footage, putting the selects on the side of that footage.

To turn a performance into motion data, Image Metrics’ custom software usually translates key points from a video onto a standard rig. That wasn’t appropriate for this performance capture, though. “Clearly, their rig wouldn’t do,” Barba says. “So, we built our rig in a way that we could give it to them and they could apply the curves to it.”

And, that was the last ingredient in the performance-capture soup. Image Metrics translated the movement of Pitt’s face into animation curves and applied them to a rig that Digital Domain built using shapes derived from data that Mova had captured during the FACS session Pitt had done earlier.

When the animators got the footage, they could see, attached to the small actors’ bodies, the old faces and heads performing Pitt’s dialog. “It wasn’t a turnkey solution,” Barba says of the performance capture. “It was a starting point for timing. There are shots in the video where Brad Pitt is sitting and staring, and you know he’s thinking. Then you load up the rig. He’s not doing anything, and he looks like a dead guy. That’s where our animators came in.”

Hair He Is When Benjamin Button is first born and very old, he has little hair, but even that sparse hair needed to be modeled, moved dynamically, and rendered. A team at Digital Domain created a new hair system based on Maya's NURBS curves. With the maquettes for reference, the team paid particular attention to creating various ways of using the guide hairs to clump the hair. "Even his hair at the scraggliest has clumping behaviors and friction and electromagnetic effects," digital hair artist Mattias Bergborn says. "It might look good modeled, but it gets fuzzy and indistinct when rendered. So we had to give it structure and find ways to make the light rays take more interesting paths to create shadows and highlights." The hair stylists could set global parameters, control areas with texture maps, and create clumps with guide hairs that attracted other hair; a virtual brush gave the stylists control over CVs within a radius as a stylist stroked it along the guide hair curves. Bergborn worked on Benjamin Button's hair for a year. "David [Fincher] was meticulous," Bergborn says. "He'd want minute changes. We had to create the hair on a shot-by-shot basis." –Barbara Robertson

On Location
To fit the old heads onto the bodies of the small actors filmed during principal photography, the tracking team needed to locate the camera and the actor’s head precisely in 3D space within each shot. They needed to do this for 325 head replacement shots. Each shot averaged 100 frames.

During filming, data integration lead Jesse James Chisholm conducted an on-set survey, measuring the main points, taking still images with a Canon 1DS, and collecting high-dynamic range (HDR) images—360-degree photographs of the set taken at multiple exposures for each lighting setup—for lighting. The body actors wore custom-fitted blue hoodies onto which tracking supervisor Marco Maldonado attached yellow markers in specific patterns.

In addition to Fincher’s A- and B-roll cameras, Maldonado added two Viper cameras as witness cameras on either side of the main camera. All the cameras were synced with time code and were gen-locked so the shutters were in sync as well. “The sync was a new process, but we felt doing that would save us time later,” Maldonado says.

The time code helped the team more easily find matching data streams from the cameras so they could do a visual check to see if the data was in sync. “We relied on that before we went into the calibration process,” Maldonado says, referring to the method for deriving the motion for the cameras.

 “Once the cameras were calibrated, we oriented the dataset to the on-set survey point cloud,” Maldonado continues. “That gave us all four cameras calibrated to the set in orientation, scale, and placement. We could see where they were in relation to, say, the Nolan house and front lawn; we could see in Maya that they were in the right place. Then we did pattern tracking.”

By tracking patterns in the 2D footage, the software helped the crew locate in 3D space the small actor wearing the blue hoodie with the yellow dots. And, that made it possible to replace the person’s hoodie head with the CG head.

“We had 20 or 30 patterns on the actor’s hoodie and face,” Maldonado says. “If you track the same pattern of dots with four cameras, the system triangulates that pattern into 3D space. We could see the 3D locator moving around in the environment.”

Eyes Right One artist on the crew worked full time on the eyes to get the "peeled grape" quality director David Fincher asked for on the surface. "There were some close-up shots in the plate that definitely scared us," says lighting supervisor Jonathan Litt. "So we asked the modelers for an extra touch. Our eye artist, who took high-definition footage of his own eyes, noticed the conjunctive layer, a netting that connects the skin in the corner to the eyeball. So we rigged this layer into the eye. When the eye turns to look in an extreme direction, this layer stretches and extends with the eye." For eye reflections, the lighting artists used a hybrid of rendering and compositing. "We tried to give the compositors as much flexibility as possible," Litt says. "Some eyes have straight 3D-rendered reflections. Often, we needed serious adjustment in compositing for the eyes and the skin. The compositors didn't adjust the lighting, but they balanced the skin tone and oily layer, and other optical effects." –Barbara Robertson

At the end of that process, a tracking artist had a point cloud moving in 3D space in the virtual set. “That was the foundation from which everything started building,” Maldonado says. “We attached the CG head to the point cloud.”

Benjamin Button’s hat complicated the problem of setting the CG head on the actor’s shoulders, especially because the CG head wasn’t exactly the same size as the actor’s head. “When you have a collar, you have a little wiggle room, but the hat has a specific line,” says Roble. “If there’s any motion between the digital head and the hat, you see it. You see the hat moving on top of the head, or the head moving under the hat.”

To help the 21-person tracking team, Roble tweaked the studio’s custom software, Track. “I made it faster,” he says, “definitely faster. And, more accurate.”

Once they had tracked the head onto the clavicle and sometimes under the hat, the tracking artists could look at it from different camera views to assure that the CG object was in the correct place. They also did tracking checks at 2k resolution on large screens.

“We used the alpha channels from the roto team to do a quick composite through Maya or Nuke to help verify that the tracking was good,” says Maldonado. When they were satisfied, the head moved on to animation and lighting.

Tiny Tweaks
“[Fincher] put a lot of effort into making performance selects that timed correctly with the body,” Preeg says, “but in many cases, we had to change Brad’s performance.” In general, the animators tried to match the 3D heads to the motion of the small actors’ heads. Often Pitt had matched that motion during his performance capture, but sometimes his head moved too much—for example, when the small actor was walking. Or, his head tilt wouldn’t correctly match the eye lines. Other times, the animators needed to change a small actor’s head motion.

For example, Benjamin learns how to walk during a sequence that takes place in a revival tent. He’s seven years old, and that’s how Pitt plays him, but he looks 80. “He’s got his head down, and he looks up shyly,” Preeg says, “mostly with his eyes at the top of his eyelids, which is childlike behavior. You want to get that shy behavior, but when you put that motion onto an 80-year-old face, the overhang of his eyelids means he can’t see anything.”

So, the animators started inching up Benjamin’s 3D head and slightly moving his eyelids. “We’d gently nudge the lid one-half millimeter, rotate the head a half degree, and lower the eye a half degree,” Preeg says. “It took iteration after iteration.” 

Another example: “There were shots where Benjamin is smiling and David wanted a bigger smile,” Preeg says. “But when he smiles, his eyes narrow, and his eyes were already narrow. So we’d try to open them, and he’d look like a crazy ax murderer.”

As they worked, the animators could quickly see any of the four views of Pitt shot during the performance-capture session on that frame or in a separate window, and could step through the mesh in Maya. They could lock to the head or the neck, and see the facial motion without any body motion. And, they could query the system to see what shapes were active in the rig at one time.

A Blur and a Final Touch At the end of the pipeline, Benjamin Button arrived at compositing in 60 layers, which the compositors could tweak in Digital Domain's Nuke. They had individual layers for lips, teeth, eye matter, hair, six different skin passes, layers for specular and diffuse lighting, and much more. Because the layers arrived without motion blur, the compositors added it using Vector Blur Z, a motion-blur technique developed by creative director of software Doug Roble and implemented in Nuke. The technique helped reduce rendering time. "Motion blur is interesting because parts of a scene are blurred and parts aren't," Roble says. "It depends on where the geometry is in the scene. It's great if you can calculate it in rendering, but it's expensive because the renderer has to essentially render a whole bunch of sub-frames." To add motion blur in Nuke, Roble had Nvidia's Mental Ray output depth and velocity, as well as color, for each pixel. "The renderer doesn't compute the streak, but it tells us how it's moving, if it's moving," Roble says. "So with that information, we can streak the pixel in Nuke just as the renderer would, but more quickly. And, the Z-depth tells us when part of Benjamin's face goes behind another part so we know not to blur there." Vector Blur Z has made a huge difference at Digital Domain, cutting the time to compute motion blur on a previous film from 16 hours to four on some shots. There's no comparison data for Benjamin Button because the crew always used Nuke on the project. "If we had had to motion-blur Benjamin with the renderer, the times would have gone through the roof," says compositing supervisor Paul Lambert. "Tweaking the amount of motion blur added a slight bit of work in compositing, but it was part of our template, so we just took it in stride." Of all the shots in Benjamin Button, one of the most difficult for the compositors was of Benjamin getting drunk in a bar. For that, in addition to the usual adjustments to fit the somewhat drunken digital head into the scene, they painted Benjamin's face flushed using tools in Nuke that would stick the paint to the face. The shot took 150 iterations before they finished. Usually, though, the work flowed remarkably smoothly. " The lighting renders were very close to the plate, so we rarely had to do overall color corrections," Lambert says. "Daylight took more work than dark, moody lighting because we had to increase the amount of subsurface scattering at certain angles. And, we did have to change things like skin blemishes. But, we knew what to tweak based on previous shots and the environment Benjamin was in." Lambert worked for 18 months on the show and yet, like other members of the crew, claims he never found the work tedious. "Because of the pipeline, it was a really good experience," he says. "It was awesome to see the character come to life." –Barbara Robertson

All told, only eight animators worked on the performance. “We weren’t sculpting the face,” Preeg says. “We were turning on bits of Brad, which allowed us to quickly tweak the emotion with a small number of animators. David didn’t want the animator’s version of Brad. He wanted Brad. We used a minimum number of shapes and let the data work for us. It was a lesson in restraint. The uncanny valley is very steep.”

Barba quickly adds, “Steve [Preeg] and I bought property there. We spent a lot of time there.”

Even though most of the artists who worked on the film lived with Benjamin Button for more than a year, they didn’t find the work tedious. For this crew, god was in the details, not the devil after all.

For example, lighting supervisor Jonathan Litt, an 18-month Benjamin Button veteran, explains: “On the one hand, it was nice because it’s a project limited in scope. On the other hand, the scope of that one thing, a digital human head, is so expansive, there’s almost an infinite amount of detail. Skin. Eyes. Hair. Teeth. Tongue. Gums. Stubble. Eyeglasses. Getting it to sit on the neck. Animation. Tracking. We had to go down to the smallest detail.”

But it was time well spent. Two weeks before the release date, The Curious Case of Benjamin Button received five Golden Globe nominations, including Best Motion Picture, Best Director, and Best Actor. It’s likely to score more, including, perhaps, among potential Oscar nominations, one for visual effects. In any case, if Brad Pitt wins an award for best actor from anyone, you can bet that several hundred people at Digital Domain will be sharing those ear-to-ear grins, too.

The Quest The first film to seriously open people's eyes to the possibility of digital actors starring in films was Final Fantasy in 2001, but the milestone 3D digital stars listed below have pushed computer graphics pilgrims further down the road toward their Holy Grail: A digital human starring in a feature film without restrictions. 2001: The cast of Final Fantasy (Square) 2002: Gollum (Andy Serkis) in The Lord of the Rings: The Two Towers (Weta Digital) 2003: Gollum (Andy Serkis) in The Lord of the Rings: The Return of the King (Weta Digital) 2004: Doc Ock (Alfred Molino) in Spider-Man 2 (Imageworks) 2004: The baby in Lemony Snicket's A Series of Unfortunate Events (Industrial Light & Magic) 2006: Davy Jones (Bill Nighy) in Pirates of the Caribbean: Dead Man's Chest (Industrial Light & Magic) 2007: Davy Jones (Bill Nighy) and Jack Sparrow (Johnny Depp) in Pirates of the Caribbean: At World's End (Industrial Light & Magic) 2008: The aged but young Benjamin (Brad Pitt) in The Curious Case of Benjamin Button (Digital Domain)

Barbara Robertson is an award-winning writer and a contributing editor for Computer Graphics World. She can be reached at BarbaraRR@comcast.net.