Facing The Darkness

So begins the long-foreshadowed confrontation between Harry Osborn (James Franco) and Peter Parker (Tobey Maguire). The pivotal plot turn, simmering over the course of the last two films, unfolds not in a typical face-to-face exchange—as such scenes are wont to do—but in a thrilling nighttime aerial battle through the upper climbs of New York City’s rooftops. With Harry in pursuit aboard his glider and Peter casting weblines in retreat, the two careen around tenement buildings and crash through the glass windows of office high-rises. When Peter’s webline is clipped, he’s sent into a vertiginous free fall before Harry spears him into a brick wall.

Surprisingly, as their bodies collide like rag dolls against brick, steel, and concrete, the audience is almost always watching the real faces of Tobey Maguire and James Franco, thus maintaining all the tension and intimacy of two actors squaring off nose-to-nose. Thanks to a revolutionary fusion of live action and CGI developed by Sony Pictures Imageworks and director Sam Raimi, the same holds true for scenes involving Topher Grace as Venom and Thomas Haden Church as The Sandman. In fact, digital faces rarely usurp the actors’ performances in Spider-Man 3. So, how is this possible, given the fact that stuntmen and digital doubles are obviously behind the superhuman action? The answer lies in face replacement and some of the most horrendously complicated matchmoves ever attempted by a major effects studio.

As the first Spider-Man film set the visual effects standard for the new wave of superhero movies, so too has the third installment raised the bar for those that follow, by using CGI to maintain, rather than to replace, the actors’ connection to the audience throughout all the superhuman fight scenes and acrobatics. It’s fitting that Raimi, a visual virtuoso who has in recent years become more of a minimalist focused on character and performance, would lead superhero CGI toward the goal of broadening, rather than limiting, the actor’s stage.

“Right from the outset, Sam [Raimi] and I agreed that CGI has advanced to the point where it’s so commonplace that it can become boring, even disconcerting, for an audience. So our goal was to use as much live-action photography as possible,” says visual effects supervisor Scott Stokdyk. With that goal, almost every effects shot became a tangled web of CGI, face replacement, matchmoving, and elaborate stunt work courtesy of Go Stunts, resulting in spectacular action sequences that far surpass anything audiences have seen in terms of scope, realism, and intensity.

Under the influence of the alien goo, Peter also starts a new romance with his beautiful lab partner, Gwen Stacy, enraging rival Bugle photographer Eddie Brock, a troubled young man who is obsessed with Stacy. If Sandman and Harry (as the new Goblin) weren’t enough for Spider-Man to wrangle with, the black goo soon finds a new host in Brock, turning him into Venom, a fearsome foe who mirrors Peter’s egomaniacal side. After alienating MJ and the city’s inhabitants, Peter must choose between the aggression-induced power of the black costume and accepting his responsibilities as a superhero. Ultimately, Peter chooses rightly, reuniting with Harry to fight The Sandman and Venom in a climactic battle sequence. “Raimi envisioned it in the spirit and scale of those Avengers comic book covers, where a bunch of superheroes would be flying around with giant monsters in the background,” says Stokdyk.

Virtual Venom

The development of the intelligent symbiont that crawls finger-like across its host—constantly sprouting and receding knucklish tendrils like some ever-changing, amorphous organism—evolved throughout the course of production. “It was the hardest thing to figure out, and for the longest time nobody had any idea what it should look or move like,” says digital effects supervisor Peter Nofz. “Finally, [costume designer] James Acheson produced a conceptual drawing that opened my eyes. It looked like a puddle with little arms coming out, and what I saw in the image was a bird mired in an oil slick—completely covered as it struggles to come out. So, this helped me see that there was definitely a bone structure within the goo.”

Raimi also gave clear directions on the morphology and movement of the alien. “Sam was adamant that it shouldn’t look like something we’re familiar with, like an octopus or a bug, or anything with multiple limbs,” says animation director Spencer Cook. “He wanted it to seem intelligent and guided by some intention and thought process. This entailed a complicated process of making limbs come out when it needed them, for example; if it was traveling on the ground, a limb would emerge, extend, and grab and pull itself forward, then that limb would disappear and another would appear, and then another would emerge from that one, and the previous limb would disappear. It was this perpetually mutable thing that never had any specific form—kind of reminiscent of something, but ultimately unidentifiable and otherworldly.”

Following Raimi’s goal of imbuing the organism’s movements with intelligence, Imageworks ruled out both practical goo and dynamics simulation. “There’s a skeletal form beneath the goo that gives it a kind of motivation and intention for its movement, which had to involve a lot of pulling and grabbing and tacking onto different people in its need to find a host,” says Stokdyk. To give it that kind of motivation, the team, led by VFX animation lead Ryan Laney, developed a three-teamed goo assembly line that began with character animators, who animated the main “hero” goo strands in Autodesk’s Maya using bones and spline IK; proceeded to effects-oriented character animators, who add membranes and filled in the gaps with additional spans between the legs and arms; and ended with effects animators, who refined the goo and added final strands using Side Effects Software’s Houdini. The strands, themselves, were modeled as simple tubes in Maya with subdivision surfaces.

“Early on, we knew that the kind of regular cycle we have at Imageworks for creating a character was not going to work, because there is no such thing as a rig for goo; the goo is a rig per shot and needs to be created over and over,” Nofz says. “So, we decided that instead of building one rig, we would create a tool set to allow animators to build a rig on a shot-by-shot basis.” Imageworks also had to determine which animators were the most technical within the character animation group. They would be assigned to the second team of effects-oriented character animators required to build and change rigs on the fly and work in a different way than the first team. The final team would leave Maya and go into Houdini, whose meta­balls made the software a far better option. This enabled the group to create geometry on a frame-by-frame basis using metaballs and implicit surfaces, making one strand a bit thicker and another a bit thinner, before rendering the final result in Houdini.

“It’s the strangest rig I’ve ever worked with,” says Cook. “We had a set of tools on a goo shelf where we could hit some MEL buttons and create a pod (we called the arms, or the tentacles, pods). For each shot, we’d determine what the goo needed to do, then build up the rig with these pods.”

Harry Versus Peter

The most intricate patchwork of CGI, face replacement, and stunt work happens during the aerial battle between Peter and Harry. As Peter swings around a building, Harry cuts his webline and sends him catapulting through the air. The effect was a joint effort between Imageworks and Go Stunts, which used a special belt rig and cable to lift a stunt performer against a bluescreen. “Go Stunts did the stunts on the last movie, but since then, has made huge improvements in the computer control of their wire work,” says Cook. “Essentially, they set up motion-control systems for high-speed winches that can drive either the actor or the stunt performer around the stage at high speed. I instantly saw its potential for doing things that could have only been done in CG before. Even getting half of a stunt effect practically is better than nothing, because now we have refined ways of transitioning from CG to live action, and vice versa.”

A perfect example of this seamless transitioning occurs in the middle of Peter’s free fall, when Harry hits him in midair, driving him backward into a brick wall. To start the shot, the stunt crew yanked Maguire’s stunt performer against a bluescreen into the path of a hypothetical incoming Harry. “After the stuntman simulated the impact, we added a CG Harry to hit the stuntman, then transitioned from stuntman Peter Parker to CG Peter Parker. Then, we did a whip pan following our CG Parker and hooked into the final part of the shot, which was another stunt performer getting ratcheted on a cable into a building façade,” explains Cook. “So, in addition to the obvious CG background and CG extension in most of the shot, we also had to achieve an exact CG match between the position and lighting of the incoming stunt and that of the outgoing stunt. Essentially, we used CG as a bridge to get between two stunt performances that couldn’t have been done all in one.”

 In addition, the team replaced the faces of the stuntmen with those of the actors by either stealing one of their previous performances or tracking the camera to do a specific motion-control camera move across the actor’s face. “We were almost always using a combination of a live-action face and an animated body. We’d have the actors sit in a chair and the motion-control camera would move around them to get the proper perspective; that photographic information would then be applied back onto the model,” says Cook. “It was a complicated process of first animating the shot, then transferring the information to the camera, before assembling it all in a composite in Inferno.” In addition to Autodesk’s Inferno, Imageworks uses Science-D-Vision’s 3D-Equalizer for motion tracking and matchmoving characters, along with its proprietary Bonzai software for compositing.

Model Heroes

While Raimi wanted to energize the action scenes with as much live action as possible, he also demanded more powerful performances from his digital doubles, as well—a goal that would require resurfacing all of the models to obtain greater resolution, especially in the eye area, the proverbial window to the soul. “The problem we’ve had with scanning people is that the eye area is always very difficult to scan, even though it’s the most important part of the facial anatomy for reading expressions,” says Nofz. “Modelers always struggle with the eye area, and are usually forced to work with photography taken during scanning. So, we asked Gentle Giant Studios to find a scanner that could get us better eye detail. They showed us a Minolta scanner [the Konica Minolta Scanner Vivid 9i] that allowed us to capture details that you normally couldn’t get with regular scanners.”

Using the newly scanned data of Maguire, modeler Kui Han Lee created a black Spidey in Maya that was 10 percent beefier than his red-and-blue counterpart, sporting greater muscle mass in the shoulders, chest, and thighs, to show that the costume had enhanced his speed and strength. For Venom’s physique, which was based on a scan of Topher Grace in costume, modelers took the musculature to a whole new level beyond that of black Spidey.

Koji Morihiro, whose highly unusual rigging of the original Spider-Man has remained untouched throughout all three films, returned to do the rigging on the black Spidey. “Seven years ago, when tasked with rigging the first CG Spidey, Koji told me, ‘I want to rig this character with nothing but clusters [of CVs in Maya]; I don’t want any fancy stuff at all.’ It took him six or seven months, and it’s still one of the most flawless characters we’ve ever built at Imageworks,” says Nofz. “And so every movie, there are people who say it’s time to upgrade this rig, and I keep saying, you know, we’re not going to upgrade this one, ‘cause it’s still good. Koji sat there for days and months to get every cluster just right, grouping and weighting thousands of CVs, and produced what I consider to be a work of art. Very few people on this planet could do it that way, nor have the patience to do it that way.”

While Morihiro used his cluster technique for the black Spidey, character setup TD Erick Miller used a more complicated muscle-based system for Venom. Under the system (which was also applied to Harry, MJ, Flint Marko, and the plain-clothed, maskless Peter Parker), Venom’s skin was attached to muscle-based primitives that expand and contract based on the distance between insertion points. To match Venom’s practical costume exactly, animators then used approximately 200 Maya blendshapes to refine the resulting deformations. To make Marko’s baggy pants ripple and billow in the wind and Peter’s suit jacket and pants flutter and wrinkle during the fight with Harry, Imageworks used Syflex’s cloth simulation tool.

The team also improved the flexibility of the spine control on its bipedal body rig to make the animation more believable. “We’ve had a forward and reverse spine since the first movie, but it was just an either/or control; you could only switch between one or the other. With our new spine rig, we’re able to do a percentage, a blend between the two, and make the division wherever you want along the body,” says Cook.

The Sandman

Of all the effects in the film, the entire Imageworks crew agrees that the Birth of The Sandman sequence was the most difficult. In 2700 frames of uninterrupted CGI, it shows Flint Marko cornered in the center of a spinning mass of sand, the atomization arm whirling around him, turning him into The Sandman, a supervillain who can control every particle in his body as well as incorporate outside sand to build mass and transform himself into a formidable monster.

The sequence was divided into three sections: the first shows Marko trapped in the sand-energizing device and reduced to a mound of sand; the second, done at BUF in France, follows a single sand grain as it penetrates his skin and cellular membranes and crystallizes his own molecules; and the final shot follows a grain of sand into the mound and shows him shaping and forging himself back into his normal self before emerging from the pit as The Sandman. At any time, as many as 50 artists were working on the shot. “It was two minutes of continuous CGI and a nightmare in terms of management and resources,” says digital effects supervisor Ken Hahn. “There were nine heroic action-type shots, as well as a subtle shot in the middle of the sequence in which Marko sees his daughter’s locket and tries to grab it, only to see his hand crumble away as it slips through his fingers.”

Once reduced to an amorphous mass of sand at the center of the atomizer, Marko uses the power of his mind to draw sand particles into his body to reform himself. As he pulls more and more sand into his body, he hones himself like a sculptor, at first swelling into a blob-like creature, with sand flowing up and off his legs, then evolving into a rough-hewn humanoid shape, chiseling away until his tree trunk-sized legs and arms dissolve back into his pants and shirt, and the stripes and colors emerge through the swirling sand texture. “It was like time-lapse photography of a sculpture in progress,” says Hahn.

The team began by doing a 3D matchmove of a live-action plate of Thomas Haden Church performing against a bluescreen, blending him with a subdivision model of the actor based on the Minolta scan. “As we were animating and receiving input from [Raimi], we made new targeted blendshapes [in Maya] on the fly,” says Cook. “There’s a huge amount of emotional content in there, in terms of his thought process as he reaches for his daughter’s locket, and his body language had to reflect that. So, we spent a lot of time trying to come up with the right poses, and shot footage of Thomas [Haden Church] to figure out the acting.”

On top of that, there was a whole other level of effects animation for the sand itself, for which the group worked in conjunction with the sand team to develop. “The movement of the sand is as much a part of his personality as his body language, so there was a lot of creative collaboration between the character animation team and the sand simulation team, to determine where and at what frame the hand should break, or how sand should flow over the locket, for example,” notes Cook.

Douglas Bloom and Jonathan Cohen led a sand team that spent 10 man-years developing custom code that was layered into Houdini. They filmed, studied, and digitally replicated the behaviors of wet sand, dry sand, and sand from all over the country, along with various types of sand substitutes, including ground corncob and walnuts. From this reference, the group developed a base set of behaviors for the sand effects. “Scott [Stokdyk’s] office has 20 or 30 vials of various types of sand, and the one we ended up mimicking in the Birth of The Sandman sequence and throughout the film was one we call Arizona sand; it’s very coarse, very large, very rough in texture, and not so quartz-like and specular; it’s not what you’d find if you went to the Santa Monica beach,” says Hahn.

Imageworks struggled through several early unsuccessful attempts to animate The Sandman. Many times, once the sand simulation was added to the animated geometry, animators found that the added mass drastically altered Marko’s shapes and mannerisms. Other times, the animation seemed too rigid and articulated for a free-flowing sand creature. “At first, the animators were doing animations as if the character had a rigid, articulated skeleton; for instance, as The Sandman arises from the sand bed, he was getting up as if he were a regular bipedal character, with these straight, rigid, solid-like arms coming up and planting themselves on the ground,” says Hahn. “[Raimi], to his credit, realized the problem, and had a conceptual artist draw about 100 of the 2700 frames showing what types of masses he was expecting to see. We used that as a style guide for how the sand should move and collapse and fold in on itself.”

Sand Comet

In another sequence, The Sandman draws sand from the bed of a dump truck, grows to monstrous proportions, and smashes his giant sand fist through the front cab of an armored car, burying two guards. While ground corncob was shoveled into the cab, Imageworks added several layers of digital sand (to spare the guards from asphyxiation). To build Marko’s enlarging body, animators used a combination of Maya’s sculpt deformers, scaleable joints, and blendshapes. While an IK skeleton lay underneath, the goal was to make The Sandman appear to have no skeleton—achieved with collaboration between the sand and animation teams.

To test the sand simulation’s interaction with Marko’s geometry, the team would perform a very crude simulation using a proprietary rigid-body simulator called Spheresim. Developed in C++ by Cohen, Bloom, and Chris Allen, Spheresim uses balls—or models—in the place of the particles, allowing Imageworks to review iterations on a timely basis. “We were rendering hundreds of millions of sand grains. It was so expensive that we needed a quick way to gauge the extent to which the sand would change his body, how many additional models, or sand grains, we would have to add or subtract, or how things would react to all the motions the animation team was adding,” explains Hahn. “If it showed some real problems, then we’d go back to animation and have them make a few adjustments.”

When the test looked right, the low-res simulation would feed into the final Houdini simulation, where the grains would replace the simple balls and be instanced to create an order-of-magnitude more particles. “When we turned to doing the sand grains, we would increase their density just to make sure you never saw sand grains floating astray; you always have some type of object-to-object collision,” says Hahn.

The sand particles follow the momentum of Marko’s movements via force vectors. “We applied vectors for velocity and acceleration on the baseline animation and gave everything the appropriate mass; there’s also friction that’s involved. We would change gravity at times, as well. Finding the right parameters was very challenging because their affect is dependent on the number of particles. For instance, if we had a simulation that had 1000 particles, but upped it to 100,000, those parameters had to be tuned down to make sure the motion that we saw at 1000 particles stayed true at 100,000.”

At the end of the sequence, The Sandman knocks over the police cars and soars away like a sand comet through the street. To create the effect, the team added a fluid simulator and a gas solver to the sand simulator, which added turbulence and convective-type motion. “We would try to carry all these simulations, keeping them somewhat contained within this envelope skin, and let the actions take over,” says Hahn. “We would apply additional forces to make sure everything got pulled back in and never got too far behind. But, if the action was very fast and the inertia too great, some of the sand lost its constraint to the skin and would fall based on gravity, creating aesthetically unpleasing artifacts. Even if it lagged behind like loose clothing, it would look like Jell-O and compromise the strength of the creature. In that case, we’d tone down the velocity vectors. The problem was, the sand wanted to disperse itself like a gas, so we also had a couple of effects guys come up with clever ways of folding [errant sand particles] back into the core body of the motion.”

Nevertheless, before The Sandman would suck the particles back into his body, gusts of sand would scatter across cars and buildings. Imageworks had surveyed people on set every day to gather measurements for the modelers, then used The Pixel Farm’s PFTrack, 2d3’s Boujou, and internal software for matchmoving. To generate the fine layer of atmospheric micro-dust left in the wake of the sand comet, Imageworks used a custom volume renderer. “We could integrate our volume renderer into RenderMan and get the proper illumination and shadow casting, which worked out well because we were able to do raymarching-type effects but on a much faster scale,” says Hahn.  

The climactic superhuman tag team match was shot on location in Chase Plaza in downtown New York, digitally made over to look like a construction site, complete with a massive sand bed. “We learned from the last two movies that it’s best to have a combination of plate photography and CG; we felt miniatures limited our options,” says Stokdyk. In addition to principal photography, the team shot Spidercam plates and pan-and-tile plates, which they could mix with their fully CG version of the plaza.

At the various New York location shoots, the crew shot multiple exposures of a chrome ball for reflection mapping. However, Stokdyk warns that, while the classic HDRI pipeline works for static environments—daytime or nighttime—it’s useless in dynamic lighting environments such as Chase Plaza, where police car lights are cycling red and blue, and klieg lights and searchlights are sweeping the sky. “Because of this, we shot moving film footage of the chrome ball to give us a sense of the dynamic lighting, which our lighting artists then used as reference as they eye-matched the lighting in the plates.”

Parting Words

As the possibility of Raimi doing a fourth Spider-Man film remains uncertain, Cook reflects fondly on his many years working with the famed director. “I worked with Sam as a lead animator on the first movie; this one was a completely new experience for me as the animation director. I would meet with Sam every day on production to develop the characters and their performances.”

A huge Raimi fan from the day he saw Evil Dead, Cook enjoyed being on set, watching the director work with the actors, gleaning insights on directing and storytelling. “One of the interesting things I learned from Sam [Raimi] was to tone things down, to let the audience read into stuff; as animators, I think our natural inclination is to maybe go a little bit too big. You want stuff to read very clearly. In the Birth [of The Sandman] sequence, in particular, every time I would show him a new version, Sam kept telling me, ‘make it more subtle, tone it down, tone down the facial expressions almost to nothing,’ and it turned out to be very effective,” he says.

“I watched him directing the actors, too; and he has a very interesting way of toning things down and letting the scene play in a more natural way without overdoing it. It’s like letting the story play out and not intruding on it,” continues Cook. The other quality Cook and the crew admired in Raimi is his unwavering grace under pressure.

Indeed, the good side wins in Spider-Man 3, both behind the camera and on the screen. 

The most difficult matchmove occurs during the Bell Tower sequence, in which Peter, at the top of the tower, pries the black costume from his body, purging himself of his venomous emotions. As the symbiont drips down the tower, it falls onto Eddie Brock, wrapping itself around him, pulling on his face and clothes as it forms the Venom costume. The transformation is captured in a dramatic 360-degree dolly shot and demanded incredibly precise rotomation of Topher Grace and his clothes.

“To help sell the effect, we attached monofilament lines to the actors’ clothing and face that would really pull the skin and stretch their clothes to give that real-world interaction,” says animation director Spencer Cook. “In post, we would then time our animation to those monofilament pulls.”

For Venom, Imageworks not only had to do complicated matchmoving for the goo, but for face replacement as well. Unlike the black-suited Spidey, Venom is able to reveal his face at will by making the goo retract. Sometimes a digital face lies beneath; in most cases, it’s the real face of Topher Grace.

“We have a lot of shots of the goo receding from Venom’s face to reveal Eddie taunting Peter; for those, we used a photographic element of Topher layered with goo tendrils and complex shapes that would conform to his face,” explains Nofz. “So, if he was talking and moving around, that was a matchmove of the face over which we would animate the goo. Remington [Scott] used the facial rig he helped set up and tracked it by hand, using different sculpt deformers in Maya to get the exact shape of the face on a frame-by-frame basis.”

To help animate the CG faces, animation director Spencer Cook led a team that conducted facial-capture sessions with all the principal actors using Vicon cameras at the House of Moves. “We captured the exact expressions [Cook] felt he needed for a shot,” says digital effects supervisor Peter Nofz. The team acquired face data, textures, and photos of the actors in their costumes, and also scanned their faces with the Minolta scanner after every move. Therefore, they had visual or marker reference as well as geometric reference.

After acquiring the data, Imageworks refined its facial animation interface to make it easier and more intuitive for the animators to select controls and manipulate the blendshapes. “For the first movie, we didn’t have visible controls that you could select; we just used the Maya shelf buttons. On this film, we created visible handles for the controls that animators could grab more interactively,” says Cook. 

Although Imageworks could have captured body data, as they did for Polar Express and Monster House, Nofz and his team chose not to for obvious reasons. “There’s no motion-capturing superhuman beings; a superhero has to go way beyond what a Cirque du Soleil artist can do,” he says. Hence, artists keyframed all the physical performances of Venom, The Sandman, and Spider-Man.

“Nevertheless, I would shoot reference with the animators whenever it wasn’t too dangerous. We would do things like running or jumping, rolling and falling, and use that to get some ideas and mix it in with the keyframe animation,” says Cook. “We also have a big library of real-world stuff—acrobats and gymnasts, and animal documentaries—that we used to make the animation as realistic as possible.” —MM