The Devil's in the Details

Frank Miller’s stark, black-and-white graphic novels are about as far removed from what many people think of as comic books as is the movie Spy Kids from the film Pulp Fiction. But it was Spy Kids director Robert Rodriguez who cajoled a reluctant Miller into letting him make a movie based on the novels.

“Robert made a test film, took it to Miller, and told him, ‘This is my gift to you,’” says Stu Maschwitz, founder of The Orphanage, a visual effects studio. “I hope you will direct a film with me.”

The offer to have Miller co-direct the film ultimately caused Rodriguez to resign from the Director’s Guild, which doesn’t allow co-directors. But the collaboration resulted in Dimension Films’ Sin City, co-directed by Miller and Rodriguez and scheduled for release April 1.

Rather than try to expand one novel or create a new story based on Miller’s concepts, Rodriguez picked three Miller novels-Sin City: The Hard Goodbye, Big Fat Kill, and That Yellow Bastard-and compiled them into one feature-length film. Three visual effects studios worked on the film, each handling all the shots for one novel. Each novel needed around 600 visual effects shots: The film’s every frame is a visual effect.

Shots of actors such as Bruce Willis were captured in Sony HDCAM SR format and converted into stylized black and white to match the graphic style of Frank Miller’s novels.

Actors Bruce Willis, Benicio Del Toro, Josh Hartnett, Clive Owen, Mickey Rourke, Jessica Alba, Jaime King, and others were shot entirely on greenscreen stages. Surrounding them, 96 distinct environments-from cityscapes, to car chases, to room interiors-were created with computer graphics in black and white, and, as in the novels, a little bit of color. By all accounts, the film is as true to Miller’s novels as it could be, in look as well as story.

“It’s a violent, black-and-white triptych with amazing movie stars,” says Stu Maschwitz, who co-founded The Orphanage (San Francisco), which handled shots for That Yellow Bastard. Café FX (Los Angeles) took Big Fat Kill; Hybride (Quebec) worked on Sin City: The Hard Goodbye. All three studios had worked with Rodriguez on Spy Kids.

Throughout the film, selected elements were either rendered in color or had color added in compositing, as in this scene created by Café FX for Big Fat Kill.

“I needed people who knew how Robert worked,” says Keefe Boerner, visual effects producer. “I wasn’t going to a new company to do 600 shots in six APRILs when they didn’t know what they’d be getting into.”

The film’s look was developed at Rodriguez’s Troublemaker Digital in Austin, where a small team of artists typically brainstorm with Rodriguez at the beginning of his films, and then follow the work through completion.

“We picked panels from Miller’s books to test with live action plates that Robert had shot of himself and a few other people on greenscreen,” says Chris Olivier, Troublemaker artist. “Our job was to recreate the panels in 3D and explore different looks to see what would be readable. We went from stark black and white, just like the comic, to the opposite-gray scale. Robert decided neither extreme worked, so we came up with a mix of stylized black and white with selective use of color and details in the background.”

“We wanted the film to look as much like the books as possible,” adds Boerner. “Sometimes the blood is in color, sometimes it’s not. Sometimes it glows white in the shadows.”

Once the Troublemaker team had fashioned a basic look that Rodriguez approved, they picked key shots from the most complex sequences in the three novels and translated those into 3D, working in Softimage XSI and in Mental Images’ Mental Ray. “We built 3D backgrounds, textured them, and lit the scenes, all very quickly to get an overall look, and then Robert would tweak them,” Olivier says. “When he was happy, we sent the models, textures, lighting, QuickTimes, and image files to the vendors for reference.”

Miller’s graphic novels provided the choreography and the storyboards. With few exceptions-car chase scenes, for example-animatics weren’t needed for the visual effects teams. “There was no point,” says Olivier. “The stories flowed so well from panel to panel, they could see what Frank did.”

Rodriguez didn’t require storyboards for filming, either. “The books were the storyboards,” says Boerner. “Robert knew in his head what would be behind the actors and just started filming them on the greenscreen. He knows what can and can’t be done.”

Working with Rodriguez on set at the beginning was Daniel Leduc, visual effects supervisor at Hybride. Rodriguez shot the film in full color and stored it in Sony’s HDCAM SR format. “We’d get output in full resolution and 10-bit color, and we could record it with all the definition. It’s a black-and-white movie, but it has tons of shading. We were able to have 10 bits per channel; 1024 shades of gray per channel.” Having that much dynamic range made it possible to have details in the dark blacks and bright whites, and to create a stylized realism.

“Some scenes are almost photoreal now,” Leduc adds. “The style changed a lot from the beginning to what we’re doing today. The way the film was done is that the three vendors worked separately on their books. Robert wanted to have separate styles. That way, he picked what he liked and didn’t like. Today, I know what the other studios were doing, but in the beginning I didn’t.”

For its part, Hybride delivered 726 shots that included 45 different locations, all virtual sets. “All the backgrounds are CG,” says Phillippe Theroux, 3D supervisor. Some were 2½D matte paintings-textures painted in Adobe’s Photoshop and projected onto simple XSI geometry from camera view; some were full 3D sets. “Occasionally-like in the jail-we used the same set for 20 or 30 shots, but usually, we were changing sets every 10 or 15 shots,” he says.

As with all the scenes in Sin City, actors Rosario Dawson and Michael Clarke Duncan were filmed on a greenscreen stage. The CG backgrounds were added by Café FX.

Although animatics weren’t needed for shot choreography, Hybride’s artists created shot layouts. Sometimes the layouts were quick drawings given to Rodriguez for approval; sometimes the shots were framed using simple geometry in XSI to create the backgrounds, with 2d3’s Boujou and Science.D.Vision’s 3D Equalizer tracking camera movements from the live-action plates.

“People wanted to see the same images as on page 25 in the book, the same icons,” says Theroux. “When the police car jumps into the river, they wanted to see that. But the book is framed vertically, and movies are, of course, horizontal.”

In addition, framing the shots helped keep the production efficient. Models were always built from the camera viewpoint. “When the framing was approved, we’d know what needed to be modeled,” says Theroux.

To further save time, although the scenes were lit using tools from within XSI, sometimes artists would paint highlights in Photoshop and project them onto the models rather than create CG lighting effects. Particle effects-rain, explosions, water, muzzle flashes, and so forth-were handled in Maya.

“Sometimes we made images that Robert felt were too real,” says Theroux. “He would say, ‘If I wanted real, I would have shot a real background.’”

Many shots were rendered in 4k and 6k resolution to allow additional camera moves in art and compositing. Some were rendered in color; others in black and white.

“When we turned the color shots into black and white, they created a richer gray scale image than if we’d rendered in black and white,” says Theroux.

Between 16 and 20 compositors worked on the project, always in gray scale, adding color as needed. “When we have color, it’s an object-the bed, a red dress, an eye,” says Leduc. “It’s a comic book look. Sometimes we have an iconic picture that’s 2-bit definition, sometimes it’s photoreal, other times there’s nonrealistic lighting.”

Getting the right look took trial and error. “We tried to have all the details in back and white with no clipping and without losing the shading in the skin tones,” says Leduc. “It was tough because there are so many adjustments you can make. You can move the gamma-the brightness slider-or the curves. We decided to use curves because you can change the definition in the low blacks and high whites without changing the middle.”

Toward the end of the production, the studio decided the film needed more color. “The easiest thing was to color the blood,” says Leduc. “This isn’t a kid movie-there’s blood everywhere.”

“They tamed it down a little bit,” says David Debner, effects supervisor for Café FX, which handled Big Fat Kill. “No one is really a good guy; there are just different levels of bad people. They cut back on some of the gore we created.”

In addition to supervising effects at the studio, Debner was Café FX’s onset supervisor. “One unique thing for this film was working with the new generation of HD,” he says. “The signal was much cleaner. We used a brand new [Blackmagic Design] DeckLink card that’s 4:4:4 compatible.”

On set, the film crew had two HD monitors. “It was easy for me to see whether the tracking marks would show up on an image-something I can’t see on the video tap monitors used for regular films,” Debner says. “One monitor showed the real color; the other was turned to black and white that a video engineer pushed into high contrast to tell if the lights were too hot in certain areas.

“An advantage of this format,” Debner continues, “Is that it doesn’t clip the data. We work with ILM’s EXR format in floating point TIFF. The high dynamic range will be noticed onscreen in this movie. The brights and darks got close attention.”

As did Hybride, Café FX created interior and exterior backgrounds for live-action shots and entire CG car chases, all in black and white with spots of color. In one car chase sequence guest-directed by Quentin Tarantino (Pulp Fiction, Kill Bill), for example, only the lights from a police motorcycle are in color. “Quentin wanted red and blue colors streaking through the back window,” says Debner

For editing, Café FX used Apple’s Final Cut Pro on Macs; for graphics, PC systems equipped with AMD’s 64-bit Opteron processors running NewTek’s LightWave and Alias’s Maya. “We primarily use LightWave, but some car crash scenes were animated in Maya, in which the programmers wrote a custom rain program,” Debner says.

Programmers at Café FX wrote a custom program in Maya to create the effect of rain dripping on the dinosaur statues in the scene with mercenaries from Big Fat Kill (top). Mickey Rourke and Jaime King say &l

Some scenes were painted in Photoshop. Cameras were tracked with Boujou; and compositing and rotoscoping was handled primarily by Eyeon Software’s Digital Fusion with an assist from Pinnacle Systems’ Commotion.

“It was nice to do one continuous sequence,” says Debner, “rather than isolated shots here and there. It was almost a movie in itself.”

Shots ranged from an urban alleyway to the LaBrea Tar Pits. “That’s where they go to stash Benicio Del Toro’s body,” Debner says. “Some vigilantes come; there are a lot of bullets and an explosion. But the cool part is the environment we did-everything from the grass to the lightning. We have dinosaur statues dripping with rain.”

Although much of the scene was created in black and white, the cars were built in color to better match the live-action cars. It was a fortuitous decision. The studio later asked for some black-and-white cars to be re-rendered in color.

“We basically hit the ground running,” says Everett Burrell, digital effects supervisor who split responsibility for the 600 shots with Jeff Goldman. “We started in July, and had to have a version of every shot done by Thanksgiving so the director could see all his stuff cut together. Then we went back to the beginning and started over again with Robert’s notes.”

Toward the end, Rodriguez began singling out objects he wanted in color. “Sometimes the sky would be blood red,” Burrell says. “And sometimes the blood might be white, or black, or red.” Thus, the studio ended up delivering two color versions for 200 of the approximately 600 shots.

One color that didn’t change was the Yellow Bastard’s sickly mustard-colored pallor. Shots for this third of the film were created at The Orphanage, where visual effects supervisor Maschwitz followed live-action rules to create the exaggerated graphic effects.

“This film was an art direction challenge,” Maschwitz says. “Every shot was unique, powerful, bold. By following the rules of live action-the language of movies-we could make it more graphic. I wanted the combination of live action and CG to feel like the comic book, not have it look like live-action actors were walking in a comic book. It had to look photographed, even if it couldn’t have been photographed.” The photographic look was accomplished through painting, texture details, effects, and lighting.

Roger Gibbon led a matte painting team that included Michael Pangrazio, a longtime matte painter, who was also a creative director on the show. The painters worked in gray scale in Photoshop. “It made them rethink the way they composed an image,” says Gibbon. “As a matte painter, you try to create the illusion of depth and shape, and when you reduce a painting to black and white you have only light and dark to lead the eye. We constructed the look through the placement of patches of light and the way in which those areas reacted to people passing in front of them.”

Once the paintings moved into compositing, the details disappeared-and then, sometimes, reappeared: The compositors turned the gray-scale paintings into high-contrast black-and-white graphic images. However, when a shadow, for example, moved across a blown out white area, the detail was revealed. Similarly, hints of details appear in shadowed areas.

That Yellow Bastard played by actor Nick Stahl, here with Jessica Alba, achieved his sickly pallor in this black-and-white scene thanks to visual effects artists at The Orphanage.

“We worked hard not to throw away the information,” says Ryan Tudhope, associate visual effects supervisor. “If we pushed the graphic look too far, it didn’t work. We stayed grounded in reality.”

In one shot, for example, actor Bruce Willis runs out of the woods toward a barn. In Miller’s novel, the barn is black against a white sky. But the shot takes place at night. “We couldn’t suddenly have a bright white background with black shadows,” says Tudhope. “Everyone in the theater would be blinded. It wouldn’t work. So we thought about how we would have achieved the shot if we had been on a set or on location. We gave the barn an atmospheric glow.”

For this shot and throughout the film, the crew used photometric lighting-volumetric lights modeled after real-world lights. “We didn’t use CG cheats,” says Maschwitz. “We even emulated the artifacts you see in films-the way you sometimes see big light flares, for example. They’re totally obvious and totally fake, but they’re part of the language of movies.”

One thing that helped Maschwitz and the team create lighting effects such as these is an artifact of the film’s storyline. “It doesn’t pay to optimize unless an environment is rendered often,” says Tudhope. “So we turned on all the bells and whistles and really pushed SplutterFish’s Brazil.”

For example, in a night shot, two shady characters filmed on a treadmill walk (once they’re composited into the CG background) past a row of seedy motel doors toward the camera. Maschwitz had the lighters add tiny aperature flares as bare bulbs flash into the scene. Also in this shot, a CG truck parked in the background reflects snow from the ground in a hubcap. The snow was baking soda that the crew photographed; the truck was rendered with Brazil. In many shots, however, the snow was created with particles in Side Effects’ Houdini and rendered with subsurface scattering in Brazil.

In this all-CG shot from That Yellow Bastard, the car fishtails into the frame and trails an almost subliminal flash of red from the taillights. Created at The Orphanage, the car was modeled in 3ds max, animated i

For example, during one of many car chases in snowstorms, a CG Chevy fishtails into the frame. CG supervisor Rodrigo Teixeira created the snow by generating 500,000 particles in a box around the car. The box moved with the camera; the camera was tied to the car. The car had a kind of force field around it to keep the particles from penetrating. To make all this efficient, Teixeira recycled particles rather than regenerating them. “Whenever the points hit the ground, instead of dying, they pop back to the top,” he says.

The car was created in Autodesk Media and Entertainment’s 3ds max, animated in Maya and 3ds max, and rendered in Brazil. As it does a 180 into the frame, you see a flash of red from the taillight, and then it stops and fills the frame. It’s a shot the crew is particularly proud of. “We nailed the CG problem of broken reflections and refractions,” says Maschwitz. “Even though it’s in black and white, it’s the closest thing to photoreal that we’ve done.”

Barbara Robertson is an award-winning journalist and a contributing editor for Computer Graphics World.