Who knew there were mummies in China? Thousands of mummies. Uncounted armies of mummies, in fact, as dashing archaeologist Alex O’Connell (Luke Ford) discovers in The Mummy: Tomb of the Dragon Emperor. Leave it to the reckless son of Rick and Evelyn (Evie) O’Connell of The Mummy and The Mummy Returns fame to awaken the shape-shifting Dragon Emperor Han (Jet Li), the first emperor of Qin, from a 2000-year-old curse, not to mention his 10,000 terra-cotta warriors. It’s more than the young adventurer can handle, so he calls on mum (Maria Bello) and dad (Brendan Fraser) to help him stop the mummified emperor from achieving his goal: global domination.
Directed by Rob Cohen, the third film in Universal Pictures’ Mummy franchise also stars Michelle Yeoh as Zijuan, the wizard responsible for the curse; John Hannah as Evelyn’s brother; and Anthony Wong Chau-Sang as General Yang. Filmed in Montreal, Shanghai, Inner Mongolia, Beijing, and other locations in China, the movie features approximately 1000 visual effects. Digital Domain and Rhythm & Hues, the two major vendors, split the effects work in half, and in doing so, shared many of the shots.
Digital Domain, led by visual effects supervisors Joel Hynek and Matthew Butler, created the emperor in his mummified terra-cotta form, his terra-cotta army (the bad guys) and a second army of mummies (the good guys), and animated huge battles between the two armies. The studio used Autodesk’s Maya, Pixologic’s Zbrush, Headus’s CySlice, Side Effects’ Houdini, Massive’s software, Pixar’s RenderMan, The Foundry’s Nuke, and proprietary tools.
Rhythm & Hues animated the emperor riding through the streets of Shanghai in a carriage pulled by metal horses, created furry yeti creatures, transformed the emperor into a Foo Dog and a three-headed dragon, extended sets and created environments in the Himalayas, and filled a pool with diamonds. Rhythm & Hues used proprietary software for modeling, animation, rendering, and compositing, and Houdini for effects.
As the story unfolds, we learn that in 50 BC, the emperor believes Zijuan will give him immortality, but instead, the immortal wizard turns him into wet clay and bakes him like a ceramic pot in a painfully grotesque manner. “Mud oozes from every pore,” Butler says. “He vomits mud, which is the source material for the terra-cotta. It self-ignites and fires, and leaves him in this solid fashion.”
Digital Domain created 20 different bodies and heads, and then scrambled the combinations to produce 4800 terra-cotta soldiers. New cracking techniques made Emperor Qin’s clay army look believable when the statues marched, fought, and shattered into pieces in Massive-driven battles.For the mud, artists at Digital Domain applied fluid simulations to live-action footage of Li. 3D tracking and warped 2D maps helped the artists match Li’s subtle facial expressions.
“We had a number of fluid simulations that had to track onto him accurately,” says Nordin Rahhali, CG supervisor. “It was one of the most difficult tracking shots I’ve seen in a long time.”
For the fluid simulations, the CG crew used an SPH (Smoothed Particle Hydrodynamics) technique first practiced by the studio to create waves of water for Pirates of the Caribbean: At World’s End.
Rahhali explains: “Traditional fluid sims are done in level sets; you effect fixed positions in space for the particles to run through. SPH is the opposite. It isn’t contained in a level set. [The particles] try to maintain a clump factor, like water fired out of a hose; each particle knows about its neighbors.” Thus, the system behaved like a fluid, yet the VFX artists could still take advantage of particle-system tools to control the mud.
Once burned into terra-cotta, the emperor remains unchanged for 2000 years, like an old clay pot resting on a dusty shelf, until Alex accidentally splashes the right combination of stuff on the statue. Although still made of terra-cotta, the emperor comes to life, and the now-digital emperor stars in the remaining two-thirds of the movie in shots created at Digital Domain and Rhythm & Hues. “The emperor speaks in Chinese and is very expressive,” Hynek says. “We had a team of animators bringing life to him through traditional keyframe animation.”
To transform Li as the emperor into an animated CG terra-cotta statue, Digital Domain created a model, starting with scans and photographs of Li’s body and face. “At first, we created a statue that looked real, but Rob [Cohen] wanted it more stylized, so we took it further,” Hynek says.
The trick was to animate the emperor’s face without losing the quality of the inanimate terra-cotta, even though in reality, the pottery would crack and break.
“In the beginning, Matthew [Butler] and Rob [Cohen] did stress studies to examine how a terra-cotta face would break, examining the most stressful parts of the face and having a team cracking the face along those points,” Hynek notes. “It was physically right. But, it wasn’t very pretty. So we started modifying it.”
Butler explains: “The terra-cotta had to break apart and reform to show facial characteristics and body motion, and still have a dry, brittle feel to it.”
To give Cohen the cracks and fissures that created the appearance of deformation, Digital Domain developed a custom rigid-body dynamics system. “It’s not fully rigid,” Rahhali says, “but it looks rigid and real.”
Digital Domain then sent its model, textures, and cracking system to Rhythm & Hues for use in a sequence that sent the emperor racing at night through the streets of Shanghai in a carriage pulled by bronze horses. Rhythm & Hues kept the look developed at Digital Domain, but to better feed the model into their proprietary system, the facility rebuilt the model within its proprietary Voodoo software and rendered it with its Wren engine. Similarly, the studio based the cracking system on methods and shaders developed at Digital Domain.
“We had developed a similar cracking system within our proprietary software based on compression and expansion maps for the horses,” says Jason Bayever, digital effects supervisor at Rhythm & Hues. “We based our shader cracks for the edges on exactly what Digital Domain gave us. The system they developed was very cool. Crack and heal, crack and heal.”
Cracking-Good Chariot RaceAs the emperor moves, his pottery shell breaks; and then as he moves again, those cracks disappear, but others form. “The geometry heals, and then the shader heals,” Bayever says.
Rhythm & Hues plans to deliver a sketch describing the procedural horse-cracking system at SIGGRAPH. “It essentially uses stretching,” Bayever says. “The horse is rigged like a normal horse, with skin stretching. So when the horse extends his leg, or moves its head, the geometry stretches the skin, and when the surface stretches a lot, a crack happens. Then, as the underneath geometry comes back, the cracks heal.”
The geometry includes pre-scored cracks, but the sophisticated system doesn’t use all the cracks with every movement, nor does it heal every crack. It randomly uses about a quarter or two-thirds of the cracks, and knows which ones it uses. “The different plates remember whether they cracked the last time they moved,” Bayever points out. “If it cracked the last time, it won’t crack this time.”
The Shanghai sequence for which they developed this system starts with the horses and chariot crashing through a window. “We rebuilt the interior set that we shot on greenscreen in China by projecting photographs of the set on the lidar scan, and built an entirely digital exterior,” says Derek Spears, Rhythm & Hues’ visual effects supervisor. “We couldn’t do the action we needed on the stage with the real card.”
Digital Domain programmed 200 actions into the motion tree for each of the five Massive agents, which meant the team captured approximately 1000 motion cycles for the armies, with the help of Giant Studios.
On set, the crew put an engine inside the chariot and motored the prop through a fabricated version of Shanghai, using that engine and sometimes a tow vehicle. “We originally thought we could use real horses,” Spears says. “But halfway through the sequence, Rick [Brendan Fraser] has to sit on one of the horses. The streets were wet, so we had safety concerns, and it became easier to attach digital horses.”
Thus, the postproduction crew fit animated bronze steeds under footage of Fraser riding a mechanical bucking bronco that they rotoscoped and removed. Animators used existing motion-captured data to help create the digital horses’ performances. “[Fraser’s] body language really sold the up and down motion,” Spears says.
As he rides along, the hand-animated terra-cotta emperor inside the chariot reacts to the environment, gives directions to General Yang sitting next to him, and cracks and heals, cracks and heals. “Rob wanted to give the emperor personality,” Spears says.
Emperor Han’s Inner-MummyThat desire to give the emperor a vivid personality resulted in a late addition to the 2000-year-old pottery warrior: an inner-mummy.
“Making the emperor look like a statue imposed limitations on his motion,” Hynek says. “Late in the game, Rob and the studio said, ‘We’re not scared of this thing.’ So, we came up with a way to bend the rules, deform him a little, and do minimal cracking, and he looked good. But by then, Rob had come up with an under-mummy scheme.”
The idea was that when the emperor becomes excited or takes a shot, a part falls off and reveals a burned, desiccated Li imprisoned inside the terra-cotta shell. Hynek says, “We told Rob, ‘OK, we can do this, but you’re going to have to direct the motion capture and approve it on the spot. And, of course, every single one changed. He wanted more anger, or lust, or whatever. So we ended up hand-animating the under-mummy using motion capture as a departure point.”
To create the under-mummy, the team started with the same scan of Li captured earlier to build his terra-cotta shell. “We matched him exactly, and then started stepping away from Li,” says David Hodgins, CG supervisor. “And then we mummified him.”
The team accomplished the burned, desiccated mummy look with shaders and textures. “He had to look like Li, but scary and god-awful,” Hynek says.
To help animators perform the emperor and his under-mummy, the team used motion captured from Li on set during principal photography, with three “witness” HD video cameras in conjunction with the motion-picture camera. In addition, for some late-arriving shots, Digital Domain captured a stunt double for Li.
“We used [on-location] motion capture mainly for interaction with the set,” Hynek explains. “The three-camera setup allowed us to do an accurate job of roto-animating. The integration team here has come up with a way of doing that in an automated or semi-automated way that makes it practical.”
Butler adds: “It’s an old concept—triangulation on a frame-by-frame basis—applied in a modern fashion.”
The team used the system to capture Li’s body and face, and to capture stunt actors playing other synthetic characters, especially the mummies and terra-cotta soldiers that interacted with the humans. “We used it anywhere we wanted to do a 3D reconstruction of motion,” Butler says. “It was really useful in a fight between Rick and Evie and the synthetic creatures. We had stuntmen in green spandex with markers all over their bodies fighting with Rick and Evie. The cameras captured the postures of the green-suited stuntmen during filming. When we replaced them with terra-cotta warriors, the fight looked realistic.”
Cracking Warriors When the emperor comes alive, his terra-cotta fighting warriors come alive, too, and like the emperor, they needed to crack and break apart. But unlike the emperor, they are only shells; they don’t have inner-mummies.
The team at Digital Domain helped director Rob Cohen choreograph the intense battle scenes by previsualizing shots with Massive simulations.The last 20 minutes of the film feature various stages of a battle between thousands of these warriors and the foundation army of traditional-looking desiccated mummies. At one point during the battle, Rick’s old friend Mad Dog Maguire shoots the brittle terra-cotta soldiers from his World War II airplane, and they shatter into bits of clay. “It’s like a massive skeet range,” Butler says.
CG supervisors Hodgins and Rahhali helped develop techniques to crack and shatter the terra-cotta. “We knew that Rob Cohen likes to blow things up, so we would have to not only animate the army, but to shatter it right in [the audience’s] faces,” Rahhali says. “We have some smart people here who basically wrote a custom OTL [operator type library] in Houdini and a level set converter to do that.”
By using the level set, the crew didn’t have to prescore the terra-cotta characters to shatter them. “The OTL would convert the geometry to a level set, which is kind of like a voxel format, based on where the hit happened,” Butler explains. “It would take into account the impact and automatically create a series of intersection objects that dice the level setup into a number of chunks—concentrated bits at the impact point that get larger as the energy disperses out. It was all procedural.”
The effects team then ran those chunks through a rigid-body simulation so they would fall and collide realistically. Sometimes, a terra-cotta warrior might take multiple hits before it finally crumbled onto the floor. “We were like kids in a candy store,” Rahhali says. “We had the tools to shatter these guys in interesting ways, and they kept going.”
Massive BattlesThe “keep going” animation happened through Massive software, which Digital Domain set in motion for both armies in the great battle and for other scenes in the film. For example, the so-called “foundation army” that fights the terra-cotta soldiers climbs into the light from beneath the Great Wall, where they’ve been buried for centuries. To make that possible, Digital Domain took lidar scans of the location, created a CG terrain that matched the plates, and then, essentially, unzipped it, creating a hole from which the foundation army emerged, thanks to the Massive agents.
“We got an extraordinary variation in motion from the Massive soldiers,” Hynek says. “When they come out of the ground, they’re supposed to react to Rick, and they do. Some threaten, some are afraid, some scratch their heads. In the battle where we see groups rush at each other, these guys get into it full on, cutting heads off, skewering with big pikes, jumping on the other guys.”
For the 2500 foundation mummies, Digital Domain modeled 10 different bodies, 20 heads, eight clothing styles, and eight hair styles. For the 4800 terra-cotta soldiers, the studio created 20 different bodies and heads. Massive then scrambled the combinations to produce different-looking characters.
For the motion, Digital Domain programmed five agents and used the out-of-the-box Massive logo guy for one sequence. Each character had approximately 200 actions in its motion tree, which meant the studio captured close to 1000 motion cycles. Giant Studios provided the motion-capture sessions for the armies and the hero characters in Los Angeles and Montreal. Choreographer Michelle Ladd directed the motion capture for the Massive actions.
At Digital Domain, Chad Finnerty led the Massive effort, with James Thornton building many of the agents’ brains. “We had to build something that would hold up under scrutiny in broad daylight,” Finnerty says. “And sometimes we bring the characters right to camera. For the most part, the Massive characters held up, but if we needed to, it was easy to pluck them out and augment them in Maya.”
By the time the plates arrived, the studio had built all the agents, so Finnerty used Massive to previs the battles for Cohen. Rahhali describes the impact: “Until then, the entire battle was in Rob’s head. Chad’s team got an effects edit in front of Rob so he could make decisions and we could get approval for the flow of the battle.”
Because the effects team had a lidar scan of the location where the live-action footage for the battle took place, they could pinpoint Li within the battle even though practical dust obscured the markers. Also, plug-ins sent footstep data, sword hits, and so forth in world space to Houdini so the Houdini team could add dust, terra-cotta bits, and other elements to make the battle more realistic.
During the battle, the Massive agents react to characters and objects created at Rhythm & Hues, including a yeti that charges through the battleground and a World War II airplane. “We built a yeti agent in Massive that played Rhythm & Hues’ animation,” Finnerty says. “Our characters would collide with that agent. And we did the same thing with the plane, making a plane agent that collided with the foundation army and the terra-cotta army.”
Yeti, Dragon, and DiamondsThe sequences with the yeti begin in the Himalayas. “It starts with Rick and Evie O’Connell landing on a digital glacier,” says Spears. For this shot, Rhythm & Hues built the airplane, the surrounding CG mountains, and, using Houdini’s particle effects, added snow to this environment and others during a search for Shangri-la.
“It’s a big, sweeping shot over the Himalayas, which are beautiful 2.5D matte paintings,” says Pauline Duvall, lead compositing TD of the plane-landing sequence. “The CG plane lands and breaks off a big piece of the CG glacier. Then, later during the same sequence, we have shots with the emperor and the yeti, and an avalanche.”
The yeti are furry creatures, nine feet tall, that help the humans fight the emperor. “They’re strong but beautiful, and almost cute and cuddly,” Spears says. The studio created the yeti using tools devised for Narnia.
Once in the Himalayas, the studio added set extensions to live-action shots of the actors in snow gear. “We also projected the existing set back onto itself to repair problems with plates and to replace stunt people and stand-ins,” Spears explains. On set, Epsom salts created snow that the studio matched with Houdini particles to create yeti footprints. A system created within Voodoo put clinging snow on the yeti’s fur.
For lighting, Rhythm & Hues’ proprietary camera system, a six-sided cube, captured HDRI images of the environment. All six cameras fire simultaneously, taking between six and eight bracketed photographs that the studio stitches together in TIFF files. “We have a plug-in that converts the digital files from the cameras into color images, winds them together, and projects them as environment maps,” Spears says.
To create a digital avalanche, which rolls down the mountain onto the set, the studio used a combination of Houdini particles and Felt, a proprietary system that describes volumes. “Rob wanted boulders, chunks, and ice, not just smoke clouds,” Spears says.
At the end of the sequence, the emperor jumps into a pool of supposed eternal youth. Rather than water for this magical pond, Cohen wanted liquid diamonds. “We used a vector field to push around lots of tiny, faceted diamonds,” Spears says. “We might have set a render record in the studio the first time we tried it.”
Rhythm & Hues used tools developed for its digital creatures in Narnia to create the furry, nine-foot-tall yeti. Houdini software helped the effects crew match the yeti’s footprints in the snow; the studio’s proprietary software Voodoo put snow on the yeti’s fur.To move the diamond water, the effects team used the studio’s fluid-sim software and Houdini particles. “We ran the Houdini particles through the motion from the water simulation,” Bayever says. “The particles inherited the motion of the water. Then we attached circles, triangles, and so forth to the particles to get the facets of a diamond. We sent tons of elements to compositing.”
The emperor emerges from the pool as a three-headed dragon, a complex creature with wings and limbs that the animators performed within the studio’s standard animation pipeline. “The difficulty for them was creating the action in a constrained space,” Spears says. In addition to this shape-shift, Rhythm & Hues also transformed the emperor into a Foo Dog that crashes through the opposition army of mummies during the big battle, which Digital Domain masterminded.
Cohen filmed the actors participating in the battle on location, not on greenscreen. “Rob didn’t want huge amounts of greenscreen,” Digital Domain’s Butler says. “In the past, we might have fought more against that, but our optical flow techniques have improved by leaps and bounds, so we can re-contribute motion-blur effects onto roto splines.”
The optical flow software, written by Doug Robles, processes hard-edged roto splines to achieve more optically correct motion-blur effects. “When you draw roto splines for motion-blurred edges, you roto the center of the blur,” Butler explains. “Then you run the optical flow analysis on the plate that has the visual data. The optical-flow analysis creates motion vectors at a pixel level that, using the image, can blur the roto spline shape appropriately to properly motion-blur the shape back into the scene.”
For example, to put Rick and Evie in the foreground with a battle raging in the background, the artists had to place their shapes, rotoscoped from filmed footage, in front of the battle. The optical flow software motion-blurred the edges so the actors didn’t look composited on top.
Toward the end of the battle, when the emperor sees the tide turning against him, he goes into an isolation chamber, summons up the five elements, and places a ball of fire between his hands. The fire on his hands is practical; the fire between his hands is Digital Domain’s CG fire.
Like its predecessors before, this third film in the Mummy franchise uses visual effects to make a fantastical story possible and, equally important, fun to watch…from the martial arts battles between Jet Li as the emperor and the wizard, to the armies of breakable terra-cotta soldiers fighting thousands of mummies. And, from the furry yeti to the enthusiastic, if sometimes bumbling, archaeologists.
“It was cool working with Rob again,” Hynek says. “And the fact that this is a comedy…”
Butler interrupts. “It’s a comedy?”
“Well, comedy-esque,” Hynek says.