All Hands on Deck
Issue: Volume: 30 Issue: 5 (May 2007)

All Hands on Deck

For Pirates of the Caribbean: At World’s End, Industrial Light & Magic gives digital Davy Jones a broader role, adds to his crusty, cursed crew, and battles a spinning CG maelstrom.

Most people in the astonished audiences who watched Pirates of the Caribbean: Dead Man’s Chest last year thought Davy Jones was actor Bill Nighy in makeup. To pull the seaweed over everyone’s eyes, Industrial Light & Magic (ILM) invented iMocap to motion-capture Nighy’s body, and artists at ILM painstakingly interpreted his facial expressions to perform the always-digital, many-tentacled Davy. For these efforts and other effects, Dead Man’s Chest, which captured more than $1 billion in worldwide box-office booty, won a visual effects Oscar.

Fast-forward to this year’s film, Pirates of the Caribbean: At World’s End, the third swashbuckler in a Disney franchise that began life as an amusement park ride. Directed by Gore Verbinski, who also directed the first two Pirates blockbusters (Black Pearl and Dead Man’s Chest), the latest action-adventure brings back Johnny Depp, Orlando Bloom, Keira Knightley, and other members of the cast.

Also returning are Davy Jones (Nighy) and his crew of barnacled buccaneers. This time, though, making a believable digital half human, half sea-creature-encrusted crew was the comparatively easy part. Once again, the studio pushed the state of the visual effects art. In Black Pearl, they put animated skeletons onboard a pirate ship. For Dead Man’s Chest, ILM invented iMocap to create digital Davy Jones and his crew. In World’s End, ILM sent Davy Jones and an even larger crew of digital creatures to battle against live-action actors inside a stormy, half-mile-wide digital maelstrom unlike anything seen before on-screen.

Shiver Me Timbers

For a third time, John Knoll supervised the effects, and Hal Hickel directed the animation, with Charles Gibson overseeing the work of studios outside ILM; the three men each won an Oscar for Dead Man’s Chest and received Oscar nominations for Black Pearl. Total effects shots for World’s End topped 2000, of which ILM created 752. CIS Hollywood, Digital Domain, The Orphanage, Asylum VFX, Evil Eye Pictures, Method Studios, and Luma Pictures contributed the rest.

At ILM, Knoll led a team of nearly 200 artists, about half the number for Dead Man’s Chest, who created the 752 shots in three months less time. Moreover, many of the shots were twice as difficult as those in the previous film. “We’d never done much of this work before,” says Knoll. “There was no precedent to build from. It made the short schedule really nerve-wracking.”

For Davy Jones, Clanker, Hammerhead, Quittance, and the rest of the returning digital Dutchman crew, ILM drew on the models, rigs, iMocap, simulation, and animation processes invented and honed to realize these characters for Dead Man’s Chest. But this time, director
Gore Verbinski upped the action. Davy Jones swings into a larger role. Ten more pirates sign onto the Flying Dutchman crew. A few thousand stone crabs carry out an unusual task. Jack Sparrow takes a moment to pick his brain. And at the end, our heroes swashbuckle with the scalawags in a 20-minute sea battle pitched in a half-mile-wide, fast-spinning maelstrom of CG water.
ILM spiked always-digital Davy Jones’s performance with more motion and emotion for this fi lm
than the last. The cursed captain sword fi ghts, his tentacles fl y into a rage, and he even sheds a tear.

In addition to fluid simulations, in many of the 300 shots in the windy maelstrom sequence, the CG crew and digital doubles—all of which needed cloth and hair sims—fought on CG ships with simulated sails and rigging. “This film is another showcase of simulation,” says Knoll. “In the maelstrom sequence, you can hardly look at a shot without seeing simulation.”

When ILM weighed anchor on the film last summer, the facility’s file servers were pushing 80tb of data around for all the shows in production. At the peak of World’s End, new file servers dealt with 103tb of data for this film alone

Davy Acts Out

First, the characters. “Our big challenge on [Dead Man’s Chest] was getting Davy and his crew debugged, up and running, and smoothly present in the shots,” says Knoll, “so with the technical issues worked out, we could more or less pick up where we left off, and that opened up new creative possibilities.”

Nighy provided Davy Jones’ performance, wearing a new, flexible gray suit made of Lycra for on-set body motion capture with ILM’s iMocap system (see “Yo Ho Ho,” July 2006). He sword fights, he rages, he even sheds a tear. This time, though, because the crew and Verbinski knew the animators could interpret Nighy’s facial performance, he didn’t need “raccoon” makeup around his eyes. (The makeup would have helped the studio remove his real eyes for the previous film if the digital eyes hadn’t worked.)

“It’s a pivotal shot,” says creature model supervisor Geoff Campbell of the tear. To pool water in the crusty sea captain’s eye and slide a tear down his cheek when he blinks took Scott Jones’ simulation, Campbell’s shape animation, Maia Kayser’s animation, and Polly Ing’s rendering skills. “We have a character with tentacles,” Campbell says. “We can’t afford to have one thing break down. The surface tension of the liquid, the complexity of the tear rolling down . . . it takes an awful lot of detail to make this simple thing look like it’s no big deal.”

At the other end of the performance scale, Davy’s sword fighting challenged the rigging team. “In [Dead Man’s Chest], he was mostly talking,” says Scott Jones, character TD. “In this film, we see him spinning around on a yardarm fighting Jack Sparrow, moving fast in a highly windy environment. When characters move quickly, it introduces a whole new complexity.”

As before, 95 percent of the time, a simulation system moved his tentacles. “The system was so smooth that this time Gore [Verbinski] could direct the tentacles to create a performance,” says Jones. But to fit Davy Jones into the stormy physical world of the maelstrom shots, Jones and the other character TDs used more than a dozen simulation meshes to move his coat, seaweed, vest, shirt, and pants, as well. The same held true for the 26 other digital pirates—10 more than in the last film.

In late November, ILM discovered that the 16 creatures crewing on the Flying Dutchman weren’t enough to handle the fighting on two ships during the maelstrom battle. As a result, art director Aaron McBride and other artists submitted designs that resulted in the 10 new characters, and Campbell led a team of three modelers who quickly built the creatures (see “Barnacled Buccaneers,” pg. 22). “Our pipeline for [Dead Man’s Chest] let us borrow parts from different characters and re-sculpt them to come up with new characters,” says Campbell. “So we used that technique for these characters.” The modelers worked in Autodesk’s Maya, ILM’s own Zeno, and Pixologic’s ZBrush.

To capture motion data for these creatures’ performances on location during principal photography, ILM used its iMocap system. Stunt actors performed the new half-human misfits who worked alongside the 16 actors in gray iMocap suits playing the returning Dutchman crew.

Brain-dead Jack

The most interesting new digital creature in At World’s End, however, looks like a blend of Johnny Depp with the creature Wyvern from Dead Man’s Chest. Wyvern is the old pirate that had almost become part of the ship. In this film, Jack Sparrow (Depp) imagines himself becoming part of the Flying Dutchman crew. He wastes away and, like Wyvern, his barnacle-covered body sinks into the ship. When this “Wyvern-Jack” creature pulls away from the ship’s wall, he leaves some of his bodily encrustation behind.

As with the other creatures, ILM used iMocap to capture Johnny Depp’s physical performance; although, because he needed to be dressed normally in some parts of the sequence, rather than putting Depp in the gray suit, ILM attached gray bands to his costume. Animators reproduced his facial expressions by hand. “It was a little trickier than doing Davy Jones,” says Hickel. “Davy was interpretive, but we wanted Wyvern-Jack to look like Johnny Depp. We want people to say, ‘Look what they did to Johnny Depp.’”

Campbell built the Wyvern-Jack model starting with a scan of Depp. “The fun of a character like this is that he leaves part of himself embedded in the wall—his back and the back of his head,” Campbell says. So I filled the inside of his body with coral and rock formations, and that’s what he walks around with.” Campbell also built a separate brain that Jack pulls out of his head.

Jones used flesh simulation to help make that brain look real. The simulation engine, which is part of ILM’s Zeno system, preserves a structure’s volume when a collision object pushes against it. Flesh-simulation artists create a digital tetrahedral “flesh mesh,” that is, cages with springs, that they place inside the volume. Altering the springs’ strength, or stiffness, affects the volume’s squishiness.

 “We have an initial motion,” Jones says, “like Jack pulling his brain out of his skull. Then we tack the brain to, say, his fingers or the palm of his hand with digital glue and declare them as collision objects. When we have our entire setup with collision objects and the flesh mesh following the animation, we run the simulation and you see collisions pressing into the flesh.” In other words, the brain moves in a squishy way when Jack presses his fingers into it.

The character TDs often used the same technique to create the illusion that live-action characters moved digital geometry—a 3D proxy for the live-action character match-moved into a 3D approximation of the live-action scene became the collision object.

In addition to Wyvern-Jack and the new Flying Dutchman characters, modelers also built a stone crab, a creature that cracks out of what looks like a rock. “The model had to be smooth stone and also have all these parts inside—legs, claws, eyes, and head pieces,” says Campbell. Animators performed the hero crabs, but in scenes with thousands of crabs, TDs created procedural animations. “We have a whole series of shots with an avalanche of crabs,” says Knoll. “For those, we used particle simulations. The shots were chaotic enough that we didn’t need to go to the expense of full rigid-body sims. When particles got too close, we did an approximation of collision detection, though, by calculating the bounce vector.”

Look what they did to Johnny Depp! Modelers
turned ILM art director Aaron McBride’s concept
art into a cursed CG Jack Sparrow.

Yo Ho Ho!

The film’s climax is the 20-minute-long maelstrom sequence. Davy Jones’ Flying Dutchman and Jack Sparrow’s Black Pearl sail toward each other with an armada of digital ships behind them. But, a violent maelstrom whirls into shape between them. The ships fire at each other across the half-mile-wide spinning whirlpool, and they try to sail around it, but it’s too large. They sail into it, and as they go deeper and deeper, they spin faster and faster.

“When they get into the narrow portion, their masts crack into each other,” Knoll says. “The crew from each ship swings to the other, and there’s hand-to-hand sword fighting. There’s a mixture of live action and CG characters and ships throughout, all in the environment of the maelstrom.”

The production crew shot the sequence in an aerospace hanger in Palmdale, California, with the actors on full-sized decks of the Flying Dutchman and Black Pearl rocking on gimbals. Surrounding these enormous set pieces was a 65-foot-tall, 270-degree, 650,000-square-foot bluescreen. Sheeting rain hammered the deck.

Later, ILM extracted the live-action actors and decks from filmed sequences, a task made especially difficult because the rain turned the bluescreen into a bluish-gray color. Also, using its iMocap system, the studio translated the performances of actors and stunt actors playing the Flying Dutchman crew into animation data. Then, they fit the actors and decks onto digital ships with digital sails and rigs, filled the ships with digital characters, and sent everything spinning into a huge, stormy, digital maelstrom. 

When Knoll first thought about the maelstrom sequence, he imagined creating a sculpted shape for the funnel-shaped whirlpool. “I thought we wouldn’t need a fluid simulation because it was a stable shape,” he says. “We’d put an ocean-water shader on the shape—the same thing we’ve routinely used for ocean surfaces for years—and generate spray particles.” With that decided, he imagined that the challenge would be in sliding simulated boat wakes into the surface.

Unfortunately, the concept didn’t work. “It was apparent within a few minutes of doing the first test that there wasn’t enough visual complexity,” Knoll adds. “We needed to do a fluid simulation.”

The Briny Deep

ILM has a state-of-the-art fluid-simulation system called PhysBam, developed in association with Stanford University, that the studio has used for such films as Poseidon and Dead Man’s Chest. For this film, they would push PhysBam to new heights—and depths. “We discovered that to get the detail we wanted,” Knoll says, “we needed to do a fluid sim at higher resolution than anything that’s ever been done before—by a wide margin.”

The limiting factors in fluid simulation are memory and calculation speed. “It’s not all that different from how an image works,” says Knoll. “To make a sharp picture, you need lots of pixels. To make a sharp water surface, you need lots of voxels.”

Simulations run inside a three-dimensional bounding box divided into voxels, which are three-dimensional pixels. Because every voxel holds information about the state of its piece of the simulation, the entire voxel grid must fit into memory. “So because each voxel takes memory, and the more voxels the better, you divide the biggest memory you have into voxels,” says Knoll. “But the more voxels you use, the slower your simulation runs.”

To simulate the ocean surrounding the massive Poseidon, ILM and Stanford scientists had devised a way to distribute the simulation among several communicating processors that shared memory and calculated different parts, and this made it possible to simulate larger surfaces than before. But, it still wasn’t enough for World’s End.

Because most of Poseidon’s fluid simulations moved a flat ocean surface, the bounding box could be wide in two dimensions and shallow in the third, like a large, flat box. However, to handle the maelstrom in World’s End, the bounding box needed to be deep. Thus, the memory-determined voxel budget scrunched the bounding box into a tall and comparatively narrow shape.

“This all came to a head in January when we showed Gore [Verbinski] where we were with the fluid sims,” says Knoll. “He said they weren’t acceptable. They didn’t look big enough.” It was a crisis: They were up against computational limits, so the too-small bounding box couldn’t stretch any wider without losing depth or resolution. And, the team had little more than three months to finish the shots.

“We were running on 40 processors,” says Joakim Arnessan, maelstrom water effects supervisor. “The problem is that when you get the calculated data back from the 40 subregions, the engine needs to combine them, so we run into memory problems. And, there comes a point where spreading out the workload doesn’t help. It’s not like 40 separate sims. They interface, they communicate together.”
Actors fi ght CG characters onboard ships that are sometimes digital, sometimes partially real
with digital sails and rigging inside a whirlpool of CG water during a 300-shot battle sequence.

No Quarter Given

Frank Losasso-Petterson, who had helped develop the fluid-simulation engine at Stanford before joining ILM, discovered the solution. “Frank came up with the idea of mashing the funnel flat, but transforming all the force vectors from their original position relative to the shape of the maelstrom,” Knoll says. By compressing the funnel as if they were pushing a collapsible drinking cup flat, they reduced the number of voxels needed in the deep direction, which made more voxels available for the maelstrom’s width. Then, Knoll explains, “to make the simulation think it was still in a warped shape even though it was flat, we transformed the gravity and wind and other force vectors.”

The result? They got four times the spatial resolution—enough size to please Verbinski while retaining enough high-frequency detail to satisfy Knoll. Because each high-resolution simulation could still take three to four days to calculate, Losasso-Petterson first tested parameter settings using medium-resolution simulations, and then ran two or three high-res sims in parallel. And that was just the first step.

The fluid-sim calculations produced a representation of the spinning water surface, velocity data, and two particle sets—one above the surface and one beneath. Arnessan’s team created all the bubbles, splashes, mist, and foam from this data with ILM’s proprietary particle system.

“We generated bubbles in the interior of the water from the particle set beneath, and splashes from the one above,” says Arnessan. The water effects team then emitted spray particles from the splash simulation, and from the spray simulation, generated a mist simulation. When any of the bubbles rose to the surface, or particles from the splashes fell onto the surface, the particle system generated white foam on the surface; the foam particles used the velocity data to move at the correct speed.

“We used our in-house system because it was such a large area to cover with particles, and it almost couldn’t handle it,” says Arnessan. “The idea was to create a full-3D water environment so that someone could put a camera anywhere and then render the surface from the fluid sim and the secondary particle sims.”

Fortunately, in about 250 of the 300 shots, most of the action takes place on deck with the maelstrom visible in the background. For these shots, the crew used a generic simulation of the maelstrom rendered at 3k resolution from 12 different compass points using a wide field of view.

 “We packaged the generic maelstrom with pre-generated datasets almost like a creature,” says Arnessan. “The TDs had the animatics from Gore [Verbinski] and the match-moved camera. They could bring in the datasets, assign materials and render parameters, and hit the render button.”

Compositors could then position the rendered whirl of water behind ships, mist, haze, and cannon smoke in much they same way as they might place a matte painting or 3D cyclorama.

Although the 50 hero shots of ships racing through the spinning water required higher-resolution simulations than the generic maelstrom, they, too, needed 360-degree simulations. “Because everything is spinning, when a camera is on a ship, the environment is moving so fast you’d get the full 360-degree tour around the simulation,” says Arnessan.

For these shots, the crew ran a simulation for the upper maelstrom area, when the ships try to catch each other, and a separate simulation for the narrow area at the bottom when the masts crash together. Because it took so long to put the ships inside the maelstrom simulation, they often sim’d the boats in calm water and added the resulting generic wakes later. “It was hard to see the difference,” says Arnessan. “The generic boat wakes came with their own extra particles above and below, and we used those to simulate the bow spray and bow mist.”

In addition to the fluid simulations and particle splashes, spray, foam, and mist, the water effects team generated water dripping from the boats, fog drifting five to 10 feet off the water surface, and mist clouding the air. Pixar’s RenderMan handled all the blobbies, volumetrics, and voxels.  “For the water surface, we turned the level set, the volume that was simulated, into a RIB file,” Arnessan says. “We essentially tessellated it to a subdivision surface pass that we rendered with foam textures and deep water shaders to add more detail.”

One last problem: In the real world, the water inside a maelstrom spins in an evenly distributed way, and the fluid engine used real physics to simulate that. But this is Hollywood, not reality. “There was a story point where one ship spins around and comes up behind the other ship,” says Arnessan. “So Gore [Verbinski] wanted one part of the simulation to move faster.” To achieve this twist on reality, the artists deformed part of the simulation and ran it with a different velocity. Of course, that meant Arnessan had to simulate all the splashes, spray, bubbles, mist, and foam again, too.

The water effects and simulation teams started nailing the maelstrom’s look around the end of January. In mid-April, Arnessan was generating crest spray for the last shot he would send to TDs who would light it, render it, and send it on to compositing. That shot was of the Flying Dutchman heading into a faster lane—an apt metaphor for this production, which, for several ILM crew members, marks the completion of a five-year stint on three Pirates films.  “It’s bittersweet,” says Hickel. “It will be great to work on something not related to these films, but it was such an enjoyable ride.” Albeit a wild one.“It was scary,” says Knoll. “Two months before we finaled, it didn’t seem possible that we could get everything done in the time remaining.” But the scientists and artists at ILM, many of whom logged long hours during the past few months, battled the seemingly never-ending technical challenges and the whirlwind production schedule all the way to the world’s end . . .  and won.
(At top) Several studios created computer-generated ships for
the fi lm. (At bottom) Morey is one of the movie’s nearly 30 CG
creatures that battle with live-action actors in a maelstrom.

Big Bytes
For Pirates of the Caribbean: At World’s End, ILM needed file servers capable of pushing around 103TB of data. To understand how much data that represents, here’s a comparison: In a New York Times article, Brewster Kahle, who founded the Internet Archive, estimated that the entire Library of Congress holds around 20 million volumes, or 20tb of data. That means shots in World’s End needed more data than five Libraries of Congress. As for rendering, the renderfarm repeatedly clocked 100,000 processing hours per night to produce shots for this film. —BR  

Barnacled Buccaneers
The conceit is that the longer a pirate serves on Davy Jones’ crew, the more sea life overtakes his body, replacing and deforming the cursed human in painful ways. The new creatures—10 news ones, plus

—are more colorful than the 16 returning from Dead Man’s Chest, but, otherwise, they represent the same design goal: creatures that could not have been produced using makeup and prosthetics. Wyvern-Jack—Jack Sparrow (Johnny Depp) imagines himself as a cursed half-human member of the Flying Dutchman crew and becomes, for a short time in the fi lm, a crustacean-covered creature with holes in his head.

—With tube coral growing from all of this character’s surfaces, the only thing left making him recognizable as a human are his teeth. “I tried not to give him specific eyes,” says art director Aaron McBride, “just tubular holes.”

—One of the last characters McBride designed, the corrosive, translucent Jelly has a jellyfish growing out of his head, like a beret. “In areas where he doesn’t have mounds of jellyfish, I gave him smooth features and dark eyes to make it look like he’s deevolving,” says McBride.

—In Dead Man’s Chest, the creature called Quittance had an eel poking out of his stomach. For At World’s End, modelers combined that eel with the legs from another creature to create the aptly named Morey. Quittance’s eel became Morey’s head.

—This hapless sailor is fusing with a manta ray. “He looks like a killer wearing a stocking over his face,” says McBride. “We mashed his human face fl at and gave him silvery eyes with pinpoint pupils, like a psychopath.”

—The red-spotted coloring of a rockfish and spiky fins lining his forearms and growing from the top of his head make this warrior-like creature spew aggression. “I gave him a perpetual, distorted snarl,” says McBride.

—Based on a sea urchin, Quill has spikes growing everywhere—on his face, arms, legs, even through his hat.

—Creature model supervisor Geoff Campbell embedded a clam in this character. When the clam opens, it opens Broondjongen’s entire left side and reveals a withered, dead figure inside.

—Withered, old Herman, encrusted with barnacles and covered with shedding fish scales, has a hapless, toothless look.

—This half-man, half-rotting sea bass is a big, bulky creature with a creepily smooth fishy face.

—A 12-inch-long shrimp lives on the left side of Fauntleroy’s face, and shellfish grow out of his hat. —BR      

Ship Shaped
Although several studios, including Digital Domain and The Orphanage, created sailing ships and junks for World’s End, ILM handled the hero ships, the Black Pearl and the Flying Dutchman, that spin into the maelstrom at the end. The studio also created The Empress, a hero junk ship, and other sailing vessels that head into battle but fl oat in the background.
Modelers built two versions of the Black Pearl, one damaged and one not, and two versions of the Flying Dutchman, one crusted with barnacles and seaweed, and one not. “For the hard surfaces, we stay in Maya, but any time we switch to sculpting work, we do that
in Zeno,” says Bruce Holcomb, hard surface model supervisor. “For the damaged Black Pearl, we had a version that was painted so the compositors could wipe away certain parts of the damaged area.”

In addition, during the battle, modelers had to create ship parts on a shot-by-shot basis. “Let’s say that in one shot Davy is fighting Jack on a yardarm,” says Holcomb. “Then Jack swings onto a rope and lands on another yardarm. We’d go to dailies and John [Knoll] would say that we needed more rigging, more of the ropes that make the sails work. We’d say that wouldn’t really be right, but it didn’t matter.So, we had masts in the background unconnected to anything, and fake masses of rigging. And then the sim guys had to make the ropes fl ail in the wind.”

Animators created the path the boats took and then handed that motion to the simulation artists who rocked the boats, blew wind into the sails, and rattled the rigging. To move the ships, the TDs ran simulations. To keep the boats from tipping over, they surrounded the hulls with fl oats. Once TDs perfected the sinking, rising, and rocking motion, they handed the ships to the water effects team, who simulated the wake based on the boat’s movement.

Then the TDs worked their way up the boat from the heaviest materials to the lightest. “We used rigid body sims to knock the masts around so they’re wiggling,” says Scott Jones, lead creature TD. “When that looked good, we did cloth sims on the sails. Then if we had seaweed, we simulated that, and any ropes and nets the crew climbed on.” At each stage, they got approvals and baked out the simulations
to speed the overall process. Unless characters interacted with the ship rigging and sails, they ran the characters’ sims separately to blast wind through their clothes, beads, the digital double’s hair, and the CG characters’ seaweed and tentacles.

“This film was a sim guy’s nightmare,” says Holcomb. “That was the real test, having everything billowing in the wind. We had some amazing work from the sim department. Remember: Any time you see sails in the maelstrom, they’re CG.” —BR

Barbara Robertson is an award-winning writer and a contributing editor for Computer Graphics World. She can be reached at