Starlight Theater
Issue: Volume 37 Issue 6: (Nov/Dec 2014)

Starlight Theater

It's rare that a visual effects team affects both the art of filmmaking and pure science while making a movie. And it's even more rare when the team turns to past, nearly forgotten scientific papers and filmmaking techniques to move science and filmmaking forward. But all that is true for the work by Double Negative (DNeg) artists who helped create Director Christopher Nolan's film Interstellar.

DNeg Co-founder Paul Franklin was visual effects supervisor on Interstellar. It is Franklin's fifth film as visual effects supervisor for Nolan, and he has received awards for the previous four. He received Oscar and BAFTA awards for Inception, BAFTA nominations for all three of Nolan's Batman films, and an Oscar nomination for The Dark Knight.

Franklin and Nolan's latest collaboration, Interstellar, tells the story of explorers and scientists looking for a way to save humanity living on an environmentally devastated Earth. When a wormhole is discovered, a few become space travelers who look for another habitable planet. The film follows their journey. The all-star cast includes Matthew McConaughey, Anne Hathaway, Jessica Chastain, Michael Caine, John Lithgow, and Matt Damon.

"We brought that journey to life," Franklin says, "the trip through the wormhole and near a black hole."

The Legendary Pictures production, distributed by Paramount, is a science-fiction film, but Executive Producer Kip Thorne, Feynman Professor Emeritus of Theoretical Physics at the California Institute of Technology (Caltech), and one of the world's leading experts on the astrophysical implications of Einstein's general theory of relativity, insisted on scientific accuracy.

"The first thing that told me this film would be different from shows in the past was that it was a science-fiction film we wanted to ground in real science," Franklin says. "We had a fantastic collaborator in Kip Thorne. He's an expert in space-time and black holes. In the '70s, he worked out that wormholes are theoretically possible within Einstein's theory. These things are stock-in-trade now in sci-fi films. But Kip pointed out that none are accurate representations of physics. We wanted to show them as realistically as possible - if these things exist, what they would look like."

Franklin enlisted the help of DNeg Chief Scientist Oliver James. "Paul sent me an outline of the challenges for Interstellar in May 2013," James recalls. "Among them were the wormholes and a black hole. This was something we hadn't done before, and I had no idea how to do it. My background is physics, but this was way beyond anything I'd done. Paul said Kip Thorne wanted it done properly, and he was willing to help. So, I had a Skype call with him, and he sent a paper describing how a photon would move around a black hole. I stared at the paper and my mind was like a Ping-Pong ball. I went antisocial for a week getting my head around it."

James realized DNeg would need a new rendering system to create scientifically accurate effects around the wormholes and black holes.

Bending Light

"No commercial renderer handles curved light rays," James says. "They always seem to be doing straight lines. But when you have a large, gravitational body like a black hole, light moves in curves. The gravity is so strong, it warps space-time and acts like a giant lens that bends the light rays around. We wanted to see the starlight warping and reflecting around the black hole from a spaceship moving toward it."

VFX artists projected outer space images on screens during filming for the actors, director, cinematographer, and crew.

James knew that if he could tell Thorne precisely what equations he needed, Thorne would be able to provide them. They both realized they were speaking the same language.

"Oliver has superb training in optics, better than mine," Thorne says. "But his training is not as good as mine in relativity. So, it was a great match. And, he had fabulous software already for dealing with optics, raytracing, and light beam propagation in flat space-time. We needed to carry this over to curved space-time. The bending of light beams produces the effects."

To generate equations that governed the propagation of light beams and the orientation of their elliptical shapes through curved space-time, Thorne discovered and adapted work from an obscure pair of papers written in the 1970s by Physicists Rob Roeder and Serge Pineault.  He checked the equations using Mathematica and sent them to James in the form of documents. 

The process of refining the equations and implementing them extended over several months.

"I needed a document control system," Thorne says, "because there were a lot of documents and a lot of iterations. The documents were like highly technical papers in a journal but aimed only for me and Oliver to communicate."

"Kip is such an expert," James says. "When he explains something, it's simple and clean, so I had to do the same thing - to ask very clearly what I wanted. He'd think about it and a week later a paper would appear in my e-mail box, maybe 10 pages long, detailing exactly the answers to the questions I had asked. He must have written a dozen of these papers."

"Once we had traced the path of light rays," Thorne recalls, "and were getting nice images, Oliver said, 'Now we have to do light beams.' That's what was necessary to get the high-resolution images he needed. Computing how a light beam changes, including its gradually changing shape and orientation, is a far more complex thing to do. I don't think physicists have done that. He had warned me that was where we were going, but it was still a surprise and a challenge. I knew how in principle, but in practice, the equations were complex."

"Kip put in an amazing amount of work and time to develop the equations," James says. "From his equations, I could generate a proof of concept. I wrote a bit of C++ code given a camera position, the position of the black hole, and the texture of the stars. It was crude, but it showed the idea could work. Our team at DNeg then implemented the equations in a new suite of rendering tools that accurately traced light rays around the black hole." They created software capable of producing the ultra-high-resolution, rapidly changing images needed for the film.

One surprising result of their work was a new discovery, a new structure on the edge of the black hole (see "Revealing a Mystery," page 12). The other was that once James showed Director Nolan the work, Nolan wanted the actors to see the images on stage.

"The images were so compelling, we didn't need to add any science-fiction fantasy overlay to make them exciting," Franklin says. "The science gave us a pure, stark beauty. The images were strong enough to use in the movie."

Bringing Outer Space In

Chris Nolan is known for wanting to film reality in-camera. But, if part of the film takes place in outer space, the only way to do that would be to bring outer space imagery onto the set.

"In a majority of visual effects films, actors work on sets with greenscreens," Franklin says. "Over the past couple of films with Chris [Nolan], we've moved away from that. For Inception, we relied on rotoscoping. We felt having real people in real locations gave better results photographically because we didn't compromise the lighting with a giant greenscreen. Chris wanted to take that idea further by projecting images on giant screens with the actors and the sets in front."

The projections would give the cast and on-set crew a sense of where the spaceship was in the vast universe.

"So, I did a lot of research into digital projectors," Franklin says. "I needed to know the amount of light output we could get. I discovered that as long as we didn't try to project into too large an area at one time, if we used two projectors carefully aligned to double up on each other, and if we shot on 500 ASA stock, with the cooperation of the brilliant cinematographer, we could shoot the interiors of the spacecraft with low-level lighting and get a decent exposure ratio with what was outside."

Franklin believed, though, that the researchers, TDs, and artists at DNeg wouldn't have gone far enough toward creating the graphics by the time they started shooting, so he would need to replace the projected images with final graphics later. Still, he thought, the rough images would help inform the photography on-set anyway. The reality was different.

"I found that the guys were making such fantastic progress in developing the renders, we could provide advanced graphics early," he says. "I didn't plan this when we went into the movie, but at the end, I realized we didn't use a single greenscreen for all the space shots."

The production crew built the main sets on huge stages with 100-foot ceilings that had been used for The Wizard of Oz in the 1930s. The set for the spaceship Endurance sat on a seesaw gimbal 250 feet long. Projected on 300-by-80-foot screens outside the spaceship were the images created at DNeg using James' implementation of Thorne's equations.

"We shot the images all in-camera with IMAX cameras," Franklin says. "Later, we fixed some artifacts, and in some, we replaced the in-camera images with newer, better material. But a significant amount of the images shot on the set made it into the movie. I was blown away by how it looked. The unexpected thing is that the cast responded to these images."

During filming, when the actors looked out the window of the spaceship, they could see and react to light bending around a black hole.

"They all said how much they appreciated having these images," Franklin says. "I think it allowed them to get into the awe and majesty and the scale of the concepts we were dealing with in the film. That was fascinating."

To make this possible, the images being projected had to be precisely aligned with the set. The crew moved the Barco projectors into position on forklifts; each projector weighed approximately 600 pounds.

Time Travel

To wrangle the digital projectors, Franklin used the Los Angeles company Background Images, which specialized in on-set projections for live events.

"We made it clear that we would have 15 minutes to set up the projectors for the next shot," Franklin says. "They thought we were kidding. They usually took a week. But they rose to the challenge and gave us a fantastic result. For one of the final scenes, we had 17 or 18 projectors running simultaneously on the set, all being moved on a setup-by-setup basis."

In that third act, Matthew McConaughey as Cooper, the hero of the film, enters an exotic space built for him by a mysterious power, a space in which the fourth dimension, time, has turned into a physical dimension.

"Chris [Nolan] wanted to give the set life with events flowing backward and forward," Franklin says. "We projected animated maps that precisely conformed to different parts of the set, which was very complicated."

"Matthew is floating in a void surrounded by horizontal lines, the stretched-out timelines from objects in his past," Franklin continues. "You'll see flickering information racing along those lines. We projected light patterns onto the lines that resolved into objects at certain points in the set. It was quite abstract. It was extraordinary. There was a sense that at one level, Inception was an art house movie. This takes that idea further. For me, it's the most powerful scene in the movie."

Sometimes, Franklin and his crew created new images on-set to put onto the projectors before the camera rolled. The visual effects artists were, in fact, creating content for the film on the stage.

"The real revelation for me was that the digital projectors are powerful, flexible, and robust enough for this work," Franklin says. "Front projection is as old as filmmaking, but it went away with digital compositing. The lead time on getting prints made and projectors set up was so cumbersome, it was easier to do it in post. This process says that's no longer true. And it was more powerful. The interaction of light from the images on the screens bouncing off the actors' space helmets - all those sorts of things we got free, in-camera, at a level of detail we'd never get by compositing images digitally. The light reflected through the helmets and refracted in more complex ways than I would have imagined and would never have put in a comp."

That flexibility, that effect, suggests that this isn't the last time we'll see Nolan and Franklin using image projection as backgrounds on-set.

"We certainly will use it again," Franklin says.

Planetary Environments

In addition to the outer space environments, artists at DNeg created and extended location shots for two main planets.

"We shot the water planet on location in Iceland for a week," Franklin says. "During that sequence, giant waves, a mountain of water, swept everything away. We had great reference from the location, but the wave was down to hard-core simulation work."

Robot props were puppeteered by actor Bill Irwin, but when the robots ran, they were CG.

Simulation artists used deformers and added surface detail through time-consuming simulations to
produce the majesty and weight. Compositors then blended a full-size model of the spacecraft, used as a prop, into the wave.

The second planet is made from frozen vapor. "We put an IMAX camera on the nose of a Learjet and flew from LA to Louisiana, skimming the edges of clouds," Franklin says. "It was so awe-inspiring. We were under-cranking, running at 12 frames per second. When we ran it at 24, we added a delicate matte painting on top to suggest water vapor in the clouds."

There are also two robots in the film, Tars and Case.

"They aren't humanoid in any way," Franklin says. "They're giant slabs divided into four blocks with three sets of pivots that can switch on and off to create configurations. Chris, as ever, wanted to make it as real as possible, so the special effects team built a 'puppet' that weighed 200 pounds, and Chris shackled [actor] Bill Irwin to the back. We had it in the lagoon in Iceland, on the glaciers, and in all the sets. On its own, it looks like a subzero refrigerator, but Bill managed to give it real character."

Revealing a Mystery

DNeg Chief Scientist Oliver James and Executive Producer and Physicist Kip Thorne worked together to create scientifically accurate visualizations of curved light around a black hole. And through this virtual telescope, they uncovered a mystery.

The process of creating the renderer extended over several months. Thorne would send James equations that James turned into rendered images.

"We iterated back and forth," Thorne says. "He would see things that looked strange and send a film clip so I could look at it."

One film clip showed an unusual effect.

"We saw a weird structure near the edge of the black hole," Thorne says. "I was quite sure it had to be an error. But, the error never went away."

He describes the 'error,' which became a discovery: "Let's say you have a black hole in front of a star field, like a night sky," he says. "The black hole gravitational lens is like a wavy mirror in a fun house. That's been seen and known for years -

it goes back to Einstein himself nearly a century ago. But, what we had never seen before was a complex fingerprint-like structure that we discovered on the edge, really quite beautiful, really worthy of a collaboration between a physicist and an artist. It changed in surprising ways as the camera moved around the black hole."

So, they pushed it further.

"There's this property of a black hole, which is spin," James says. "The faster it spins, the more distorted the light field becomes. It would be too confusing to show that in the film, but Kip is interested in extreme ends. The renderer we had built is like a virtual telescope, so I said I'd push it to the extreme and make images for him. When the spins got incredibly high, 99.9 percent of the maximum spin a black hole could have theoretically, the distortions became more and more extreme."

"It produces a beautiful pattern," Thorne says. "I think it's exciting in that I believe there are some interesting mathematics to be sorted out. There has been a lot of work by other physicists to study gravitational lensing by black holes where the source of light is near and the camera is far away on Earth. For science fiction, you want the camera close and the source of light far away, or sometimes close. There has been little work where the camera is close."

Thorne continues: "It turned out that there is this little discovery where no one has explored. It's not profound. But, it is a mystery. It's highly enjoyable. This is truly a fun mystery that I expect others will sort out. But, not in the next few weeks. What's going on in there is too complex. The mathematics associated with that pattern may turn out to be rather neat."

-Barbara Robertson

However, in zero gravity when a robot ran, or water wheeled through the lake, it became a CG double.

"The animation was informed by what Bill and the stunt performer did, though," Franklin says.

In keeping with the goal of having everything real, Nolan also made wide use of miniatures for the spacecraft, relying on New Deal Studios to create the models - a 15th-scale miniature 25 feet long of the Endeavor, and a fifth-scale model of the Departure. In the foreground, he had full-sized props. Nolan piloted a 50-foot-long Ranger positioned on a motion base himself.

"In the end, the vast majority of spacecraft are miniatures," Franklin says. "They were pretty spectacular. We added the digital backgrounds, but the miniatures gave us a tactile reality that makes this film look unique. I don't think anyone has done anything like this for a number of years."

One of the themes of this film is time travel, and to create the effects, the crew traveled back in time to reprise past techniques, but with state-of-the-art tools and techniques.

"The interesting thing I learned is to never assume anything has been settled in the way we do these things," Franklin says. "Stuff from the past works well and shouldn't be ignored as we move to digital tools. At the same time, advanced rendering offers new possibilities and takes us to places we couldn't have imagined a few years ago. The most satisfying thing for me, though, is that through making a sci-fi film, we discovered new science."