On Director Saschka Unseld's Twitter page is a photograph of a dark-blue umbrella lying upside down on a rain-soaked street next to a white fire hydrant. The umbrella's exposed ribs, bent at awkward angles, look like spidery legs trying to lift up the crumpled canopy body. You could almost imagine the nearby fire hydrant with dots for eyes, nozzle nose, and ears on each side looking on.
When Unseld took that picture, he was a layout artist - a cinematographer - at Pixar Animation Studios. Now, he is the highly praised director of Pixar's short film "The Blue Umbrella." Released with Disney/Pixar's feature Monsters University, the six-minute, 40-second film has sent ripples through the CG community for its unique style.
The story is simple. The film opens on a busy, sunny city street, and then one raindrop falls. And another. The music begins and we see smiles in the hints of faces on objects that come alive in the city - a concrete outlet cover in the street, a gutter, a mailbox.
With the rain come umbrellas. The camera looks down on a sea of gray umbrellas in which one blue umbrella stands out as it bounces down the street. A red umbrella happens alongside. Blue flirts with Red. But, their owners move in different directions. Desperate not to lose Red, Blue uses a gush of wind to break loose, and he soars into the air above the street. When Blue falls, the city helps him survive, but still, he lands in the gutter. Then, magically, his now soaking-wet owner finds him, and as Blue and his owner try to recover, Red and her owner appear and lend a hand. At the end, Blue and Red lean toward each other, as do their owners, and look in a window.
A PHOTOREALISTIC CG CITY comes alive in Pixar’s stylish short animated film “Blue Umbrella.”
As with many creative successes, the film juxtaposes contradictions. It's an animated film shot like a documentary, which makes it feel real. Simple characters with cartoon faces - the blue and red umbrellas - perform within a rendered CG city that looks real. So real, in fact, viewers often ask if the CG film is live action with 2D morphs.
Animated objects in the city become a supporting cast, a kind of Greek chorus that comes alive in the rain and interacts with the blue umbrella. Yet, although the best description of the film is "photoreal," thanks to the rainy atmosphere, the short animated film has an abstract, almost painterly quality.
"The Blue Umbrella" is a triumph of 3D modeling, animation, cinematography, lighting, and rendering. And, of course, story.
Perfect Pitch
"I didn't think about the visual style when I figured out the story," Unseld says. "I just wanted the heart and soul of the film. When you pitch, you have to tell the complete story. The only visuals I had were four collages I had made from photos to underline main story points - a sea of umbrellas with a tiny blue umbrella and one with a red umbrella in the middle. The umbrella in the gutter. And, the happy ending."
At Pixar, anyone can pitch short film ideas. It's a two-round process. First, the hopeful director talks through three complete story ideas with a panel of Pixar veterans in the development department who, in Unseld's experience, included Directors Peter Sohn and Pete Docter, Story Supervisor Jason Katz, and others. At a second meeting, successful candidates, such as Unseld, incorporate notes from the first round in preparation for a presentation to Pixar's Chief Creative Officer John Lasseter.
"They loved 'The Blue Umbrella,'" Unseld says. "They didn't have any notes for changes. So, I made some changes. They said, 'Naw. The first time felt more relaxed.' I was overthinking it."
Unseld still hadn't created visuals beyond the four collages, so during his pitch session with John Lasseter, Lasseter asked how the film might look visually. Unseld calls his answer the "subconscious" part of his pitch.
"I hadn't given it much thought," Unseld says. "But, at the end of my pitch, I said, 'Oh, by the way, I did this test that might be interesting to look at. It might be one way to bring the city to life.'"
Sometime earlier, the then layout and camera artist had shot short video clips of faces he had seen on building facades, mailboxes, fire hydrants, and so forth in San Francisco.
"I thought I could make a music video," Unseld says. "So, I did a 3D track of the footage in [Autodesk's] Maya. Then I did a 2D morph to animate the faces to the lyrics of a Sarah Jaffe song. Because the flat morph was tracked to the camera movement, it had a magical quality; every time I showed it to people, I'd see a sparkle in their eyes as the city comes to life. They [the Pixar directors] said, 'It would be so great to have that magic from you, from thinking the city is real to it being alive.'"
But for "The Blue Umbrella," Unseld wouldn't be able to work with 2D morphs of live-action footage. Shutting down several city blocks for filming, hiring dozens of extras and drivers, and, most importantly, controlling the rain, was not an option.
Thus, creating magical animated faces in what first appear to be static objects depended on creating a realistic, believable CG city.
Magical Realism
Computer Graphics Pioneer and Pixar Cofounder Alvy Ray Smith once stated that "reality is just a convenient measure of complexity." It fell to "The Blue Umbrella" production crew to create that complexity.
"We had a very high bar," says Chris Burrows, supervising technical director. "Saschka [Unseld] was really interested in the photorealistic look, so everything fell under that umbrella, so to speak. It required scene complexity in textures and in the geometry itself."
Burrows describes a wide shot in which Blue floats through the air above a city stretching into infinity as the camera pans from left to right. "That shot has a million and a half pieces of geometry in it," Burrows says. "A million and a half individual models. Most, though, had 400,000."
The team started with a set that was six city blocks long and had six intersections, built in rough, low resolution. "We used that to find our locations," Unseld says. "Once we placed the action and the camera and decided how shallow the depth of field would be, we gave it a rough lighting pass and detailed it out more. We worked on everything in layers to not have the work spiral out of control."
To determine which buildings needed geometric detail and which could be textured cubes, the team borrowed a working method from visual effects studios.
THE BLUE AND RED STARS of the film stand out among the sea of gray umbrellas.
"We implemented shot dailies," Burrows says. "That was a new process for Pixar. Every morning the entire crew would look at the shots at whatever state they were in. That provided the context for making decisions. The level of detail was essentially a manual process after that, but one of the things we did with "The Blue Umbrella" was test out [The Foundry's] Katana." With Katana, the artists could look at the scene and reduce complexity where possible before sending it to the renderer.
"The first time we raytraced a scene, we ran out of memory on a 64gb machine," Burrows says. "So we did a lot of work to optimize and lighten up the geometry."
The animated faces, however, remained geometrically complex. "We see 14," Unseld says. "And there are another three or four that don't have active roles, like the building we called Anna that still has a face but isn't featured. You can discover it if you watch the film a couple times."
Unseld gave each a name - the triplet awnings, the traffic lights, the church, and so forth. "Some are high-energy creatures," Unseld says. "Some are in their '90s and have jaggy movement. The mailbox is Carsten, the rainspout is Greta, and the electric outlet cover, the first one you see, is Lisa. It felt right to name them."
An outlet cover on the street near Unseld's home in San Francisco inspired the face he calls "Lisa." In the film, the magic begins when she winks.
"Lisa probably had 20,000 [polygonal] faces," Burrows says. "Saschka went out with a tape measure and took high-resolution photographs of the concrete plate in San Francisco. We had to model that dense geometry and then put high-resolution textures on top of that. The others weren't that complex, but we definitely pushed the bounds."
For textures, the shading artists manipulated photographs and then mixed in procedural and painted textures. "That brought a richness to the detail," Burrows says. "It would be hard to paint nice concrete by hand, but we didn't want it to look like 'here's a photo, there's a photo.'"
Scene Complexity
Unaware of the magic surrounding them, crowds of people under the sea of gray umbrellas walked on the sidewalks, and cars traveled through the rain-soaked streets, all of which filled the city with more detail and energy. "The city feels alive," Unseld says.
Animation cycles applied to the people and cars kept them moving, and pre-canned cloth simulations for the clothes on those close to camera added to the visual complexity. "We realized that in the wider shots, we had people down the street who would hold up in close-up shots," Burrows says. "They even had buttons on their jackets. But, we didn't need to model clothes on people at a distance. We just painted the skin to look like they were wearing clothes. When they get even farther, you just see umbrellas rocking back and forth down the street."
To move the people, the crew used the crowd pipeline developed for Monsters University. Animators created walk cycles and drew paths on the sidewalks; the crowd system placed the digital people on the paths and applied the animated cycles.
"We cached the walk cycles into our internal file format, and then the simulator would pick cycles to match certain speeds," Burrows says. The crowds didn't need to be agents with an artificial intelligence-based simulation. They only walked, stopped, or stood.
For the cars, the crew modified models created for the film Cars 2. "We removed the eyes and mouths, and stretched their chunky, thick proportions into more normal cars," Burrows says. "We also removed most of the undercarriage."
THE EFFECTS ARTISTS CREATED and rendered specific types of rain to heighten the mood: (top) sharp rain for a brutal scene and (bottom) softer, more mysterious rain.
Similarly, they used a simplified version of the Cars 2 driving system. "Animators could have a car travel on the road, and the system would automatically lock the tires to the ground," Burrows says. "So, we modeled potholes into the street surface. All the animators had to do was keyframe the car from one end of the street to the other, and the car would bounce a little up and down. We laid out an animation cycle for three city blocks that lasted for 2,000 frames, and re-used it. You couldn't tell."
Although it sometimes seems the city, the crowds, and the cars extend for miles, the crew managed that illusion using the complex but smaller six-block set they started with. "We had a few long, looking-down-the-avenue camera scenes," Burrows says. "We found that once we added atmosphere, six blocks with some matte-paint extensions were enough.
Facing the Rain
For those long shots of the city, lighting artists added depth-based fog, and for shots of Blue flying through the air, effects artists created fluid simulations and rendered volumetric clouds.
But always, there was rain. Once the first raindrop hits Lisa at the beginning of the film and the music begins, it never stops raining. The rain sets the mood.
"I come from Hamburg [Germany], where it rains a lot, so I feel closely attached to rain," Unseld says. "The rain, to a certain extent, is the third main character." In fact, as with the mailbox, the electrical outlet cover, and other faces in the city, Unseld named the rain. Not with anthropomorphic names, but with words that describe emotions.
"Saschka had really clear ideas about what he wanted the rain to look like," Burrows says. "So, we needed to get the rain right from a storytelling standpoint. He had a chart listing very specific looks." The rain might heighten a love scene, create mystery, or be as fierce as a gladiator. Unseld's rain-driven moods ranged from boring to stormy.
"When they [Blue and Red] meet, I wanted soft, hazy, romantic rain," Unseld says. "When the [blue] umbrella falls into traffic, I wanted the rain to be brutal, harsh, uncomfortable. We did a lot of work on the rain."
To create the rain, the effects artists used Side Effects Software's Houdini's particle and fluid simulations. "For the hero shots, like when the rain hits the mailbox, we'd run high-resolution fluid simulations to get a splash element," Burrows says. "But for a majority of the rain shots, 75 to 80 percent, we had a library of rain and splashes on the ground they could just run. One thing we did that was neat was to have splashes that don't correspond to any rain hitting. When you put the shots together, you can't tell."
BLUE MOVES FROM a cold, green environment into the warm yellow colors surrounding Red.
For the falling rain, rather than replacing particles from the simulation with curves at render time, the artists did the replacement before rendering and saved the simulations to a cached file format. "We cached animated curves, not animated particles," Burrows says. "By doing this further upstream, we could modify attributes on the curves in Katana."
For example, Unseld described one type of rain as "gladiator" rain, referencing the slow shutter speeds used for the film Gladiator. To create that look, artists working in Katana reduced the number of points on the curves to shorten them and make the rain sharper. For another look, they might rotate the curves to be at 45-degree angles.
"For some of the emotional transitions, we did hero rain box simulations," Burrows says. "For example, to time the rain with the action when Blue struggles to not go into the subway station, we created a hero rain sim and put the rain through it. But for the most part, we're running curves through the air."
The artists changed the look of the rain in other ways, as well. To heighten a romantic scene, for example, the artists toyed with motion blur. "We played a lot with motion blur," Unseld says. "Sometimes, we faked the length of motion blur to create a softer, smoother, more romantic feel."
Burrows explains the process. "We used a shader trick," he says. "We rendered the curve as a ribbon and faded in transparency at the top and bottom. Then, to make the rain look softer, we made that transparent region longer."
On screen, the artists working in Katana would see what Burrows describes as "lots of little lines." "What you eventually see depends on the render, so the effects artists would adjust the lines in Katana and then poke the render. They might tweak the length, the opacity, or the falloff, and then render the shot. Sometimes the effects department might package the rain and give it to the lighters. When you're dealing with the aesthetic look of rain, you can't evaluate it until it's lit and in the scene. So, it was an iterative process, which is partly why we did shot dailies."
Because all shots in production rendered every night, the longest anyone had to wait was the next day, but usually the feedback was faster. "We quickly found that the way the rain was lit had a big impact," Burrows says. "And, just as in live action, the rain looks different when shot with long or short shutter speeds. Because photoreal was the goal, we decided to use the physically plausible shading developed for Monsters University. When we have materials that respond to light the way they would in reality, we could put a streetlight at the right intensity and be 85 percent of the way there."
Physically Plausible
The materials in the city and water-slick environment provided a perfect palette for the physically plausible shading and raytraced rendering techniques. "Saschka said he didn't want just photoreal," Burrows says. "He always said 'photoreal' can mean many things, that a documentary is photoreal and so are the films by Wong Kar-wai. He wanted the film to look photographed and was heavily influenced by the photographer Saul Leiter."
The new lighting rigs developed at Pixar for Monsters University lent themselves well toward creating the layered look of Leiter's photographs in which abstraction and precision share a frame. To bathe the images in "The Blue Umbrella" in a neon glow, for example, the lighting artists used a rectangular shape that functioned like the "window" light used in Monsters University (see "Back to School," July/August 2013).
"Saschka wanted saturated glows coming from off-camera," Burrows says. "So we created renders of neon signs and textures the size of the signs, and then put those on a rectangular light. Because we used physically plausible illumination, those textures acted like HDRI domes."
Similarly, the lighting artists used a hemispheric dome light for the sky, with different gradient maps providing colors for various scenes. In the beginning of the film, a gradient map representing the sky transitioned from pink at the bottom to dark blue at the top. Later, gradient maps turned the sky gray.
"Saschka had a clear vision of what he wanted from an emotional standpoint," Burrows explains. "When Blue and Red are together, the world is bright and yellow. When they are separated, the world becomes green and cold."
Within a scene, practical light sources in the city motivated the type of lights the artists selected from the lighting kit. "We have a sphere light with a radius you can define," Burrows says. "And, because the lighting is physically plausible, as the light gets bigger, it gets brighter. For streetlights, 10 centimeters gave us the appropriate softness of shadows that a real streetlight would give."
The artists could also attach "barn doors" to the sphere lights to constrain the light to a particular region and even create shafts of light.
"Of course, all these lights contributed to the indirect illumination," Burrows says. "For global illumination, we turned on two bounces. A fill light would hit a sidewalk and bounce onto the building next to it."
THE SIMPLE CARTOON FACES created surprisingly challenging rigging problems.
The raytraced lighting and complex geometry could have easily caused unworkable render times. "Our goal was to make sure the lighters could launch a render at night and get results in the morning," Burrows says. "That amounted to about four to five hours per frame. So, we had to do a fair amount of work to be careful about which objects were hit by what lights. If you don't keep an eye on all the geometry, raytracing and global illumination can become expensive."
Lastly, to achieve the photographic look that Unseld wanted, the artists refined the images in DI. "We ended up doing a lot of color grading in [The Foundry's] Nuke to get a filmic look and to vignette-out the edges, which is also where we did depth of field," Burrows says. "We did not render depth of field in-camera. Doing all this in Nuke is unusual for Pixar."
For Burrows and the rest of the production crew, the praise raining down on the work has been worth the effort. "One of my prouder moments was after an internal screening," Burrows says. "A Pixar employee who probably should have known better asked, 'So when did you guys do the shoot?'"
Singing in the Rain
Even with all the detail and magic in the background, the simple bright umbrella stars of the film keep viewers focused on their performances. "If you create a look you haven't done before, you don't know how distracting it will be in the end and how readable," Unseld says. "The lighters did an amazing job making the shots look real and with the lights and colors on the surface of the umbrellas so you could read the faces. The picture is so full of detail we needed to make sure the face pops out enough."
During the design phase, Unseld and the team resolved the issue of whether the umbrella canopy would be the character's head or body. "When we made the face big, the canopy would look like a head," he says. "When we made it small, it felt like the body. We ended up with something in the middle. And, because we wanted to have the face stay between two ribs of the umbrella, we faked the front gaps bigger."
The last steps of the design phase were animation tests, and for this, Unseld brought Animator Christopher Chua onto the team. "Chris drew 2D faces on top of our CG umbrella to help us figure out what the faces looked like," Unseld says. "He helped us explore how much expression and detail we could get from these two dots and lines. The problem with CG is that you can't animate before you rig, but the rig takes so much work. And, you have to know what the face should look like before you rig. How big, how small, how readable."
"So much was in the face," Unseld adds. "It was an interesting challenge. The faces are cartoony, so it was clear how to animate them. The challenge was in the rigging. We needed such clean shapes, circles, and lines. 'Simple' sounds simple, but it isn't. So much lies in where to put the apex of the curve and how it tapers off to the end. With a pencil, you have control. But, if you have values to adjust, you want to adjust everything. You end up with a thousand controls, and that's hard to handle for animators. So, we had a lot of back and forth."
The animators also needed to create performances for the entire umbrella and the hero characters beneath. In doing so, they had to decide who controlled the movement.
"We had to be sure you would never get suspicious of the movement," Unseld says. "So, as long as the owner holds the umbrella, the owner controls the overall movement, the big movements. But, to a certain extent in animation, you want to move the character. [Blue] had no arms or legs, but we didn't want the audience to feel like this is a character with no arms and legs."
Thus, if Blue's canopy body moved on its own while still held by the owner, the viewer needed to attribute that to the wind or someone bumping into him. That helped when Blue meets Red and in the beginning of the storm, but we see their reactions largely in the facial performances.
A second challenge was the relationship between the faces and the surfaces beneath. "That movement was a nether world," Unseld says. "A cartoony face can make fast moves, but the umbrella needed to feel real. So, the question was, Do we sim the cloth or not sim the cloth? We ended up simulating the cloth only in a handful of shots when Blue lies in the gutter. There, we needed to feel the canopy sagging and hanging down. Otherwise, the animators controlled the movement of the cloth. We were afraid the simulation would move it too much and the face would swim around."
For the gutter shots, the effects artists used FizT, Pixar's in-house cloth simulation software. "Most of those shots were animated by hand," Burrows says, "But, there's a part where Blue is lying in the gutter, rocking back and forth. For that, we ran the cloth simulation and blended it into the hand animation. We also did some cloth sims on the hero humans."
The hero humans are the two owners of the blue and red umbrellas, and while subtle, the story is about them, too. "Even though we don't see them much, they needed to move and act," Unseld says. "Mostly, they're defined by their Wellington boots. Everyone else wears shoes. And because their gloves and boots are the color of their umbrellas. At the end, we slightly shift the attention onto them, to their story line. We needed those details for the payoff in the end."
For the crew, the payoff is in the accolades and experience. For Pixar, it is, as always, the chance to experiment in short form with ideas that might make their way into feature production. "We used global illumination on Monsters University," Burrows says. "So, that's here to stay. We're moving toward using Katana on a more full-time basis, and what we learned on this short film has had a strong impact for features. And, the shot dailies concept was intriguing to the feature-film group. They are experimenting with a more iterative workflow."
As for the style…will we see a photorealistic animated feature someday in which elements in a city are characters like Lisa the outlet cover, Greta the downspout, the triplet awnings, Carsten the mailbox, and the others who brought the city alive in this film? It's hard to predict. But if another potential director happens to take a photograph that inspires new kinds of characters, whether at Pixar or somewhere else, thanks to "The Blue Umbrella" crew, we know that magic is possible. CGW
Barbara Robertson is an award-winning writer and a contributing editor for CGW. She can be reached at BarbaraRR@comcast.net.