Wētā FX played a vital role in bringing the thrilling events of the new season from the page to the screen, including the massive snow-covered infected hordes that descend upon Jackson in Episode 2. Working closely with showrunners Craig Mazin and Neil Druckmann, the team crafted terrifying new infected creatures, built an extensive CG environment surrounding Jackson, developed a new compositing-based weather control system, and much more. VFX supervisor Nick Epstein and animation supervisor Dennis Yoo shared an inside look into bringing Season 2’s epic visuals to life.
CGW: How did you set up your workflow for this season?
Nick Epstein (VFX Supervisor): The initial thing is to really understand what the creative goals are, and then to set up the processes that are going to make your team successful in reaching those goals. You want to do that before things get really gnarly and you hit that critical mass of shot production, so you're not just solving the same problem over and over again. You can do it in a much more structured way.
We went through the previs, so I had some idea about how epic and challenging this was all going to be. But honestly it wasn't until day two or three when I was on set with Alex [Wang], who's the production visual effects supervisor. In the morning it was blazing sunshine, by lunchtime it was pouring with rain, and by the evening it was snowing. It was like that I'd say two thirds of the shoot, where we had these really crazy weather conditions to deal with. So that cued me into some processes that we would need later on.
Then also seeing what was being shot. It was fairly close to our previs, but it's only when you see the plate come through that it really dawns on you that this is what we're up against. You get an idea of what you're facing, and then you begin to break it down with the team.
Dennis Yoo (Animation Supervisor): One of the things that I thought we needed to prepare for was the dogs. I had experience working in dog movies before, and they're one of harder animals to do. Because everyone knows what a dog's supposed to move like, and everyone knows how they're supposed to act. So they're one of those animals that you can't get wrong.
Then some of the things that I thought I had a handle on because we did it in Season One, like the crowds. We'd get these plates, and they'd be blank. You have a title card going: massive horde. We started filling them out, and they started looking like marathon runners. They were a bit tame. Even though we did the motion capture, two or three individuals bumping into each other does not make a massive horde look like an organic entity itself. I've never had to deal with thousands of people bumping into each other and moving around.
It had to feel chaotic, so we tried a few different approaches to achieve that. One was a Massive simulation, but the tricky part is that Massive doesn't handle collisions all that well. So we layered in Ragdoll, did a lot of motion editing and a lot of keyframe animation for bespoke animation. So It was almost like a mix of all the principles of motion in one shot, and there were several shots like that. Some things we couldn't really plan for; we just knew they’d be tough, like those top-down shots with all the bodies running over each other. But we just tackled them as they came.
Nick Epstein: The layers got peeled off and revealed to us. There needs to be a mound of hundreds of Infected, which all need to move around—and then that needs to influence the motion of the infected that run over them. Oh, and by the way, there's explosions going off and they all need to be on fire. It’s layer after layer of difficulty. You start getting into it with Alex and with Craig [Mazin], and the reality of the vision crystallizes. This is actually what we're up against.
CGW: Can you tell us about the process of creating the full CG environment around Jackson?
Nick Epstein: On the environment side, it’s important to note that Distillery built Jackson itself. The set was incredibly impressive. It was four fully realized blocks, plus the wall, plus the guard tower, and so on. We inherited their buildings and their mountains behind, which we definitely appreciated.
Everything looking out in front of Jackson, that was our creation. The Last of Us always tries to shoot a plate and tries to realize everything practically. So there was an impressive amount of space. But essentially, when you're looking out beyond the gate, you're just met with a massive green screen. We did have some mountains in the distance, and they were somewhat at the correct distance. You could judge atmospheric conditions and so on.
We built that environment based on a bunch of tilesets that were shot. Combinations—some in Kananaskis [Alberta], some in Whistler [British Columbia]—and they all had to conform to what was established at the end of Season One when they approached Jackson. We had some pretty good reference, but it was a little bit freeform. So that was a bit of exploration. I think the environment team did a really great job building all of that.
In the shots where the lookout guy sees the horde off in the distance through the binoculars, it's important to refer that back to some sort of reality. So they went out and shot these plates in Kananaskis in somewhat representative conditions. They were lucky on those days. Then we had to transpose that into our depth space outside of Jackson, line that all up. Ultimately it served as a really good reference as to what our horde should look like running through a blizzard looking out from Jackson.
I really appreciated the grounded approach they took to photography. Even if a shot's that’s going to end up being full CG, they go out and shoot a plate. In 99 percent of cases, there would be a plate for us to refer to. That's one of the reasons The Last of Us looks so good, because they set this very high reality bar that you have to hit. Even though some of the work obviously is fantastical by its nature, it's all grounded within some level of reality.
CGW: Tell us the new compositing-based weather control system you built for this season.
Nick Epstein: That came about because of my experience on set. The weather and the shooting conditions were all over the place. We obviously had to conform it to the storm as it blows through Episode 2. It's almost like a character in itself. It's metaphorical, and it's basically as much a part of the story as any of the other characters involved.
So we knew these wildly varying weather conditions had to be conformed to this very specific time of day, this very specific blizzard feel. That meant we needed a very robust way of doing that. We knew we'd have to execute hundreds of shots, so it needed to be something that wasn't going to be explored on a shot-by-shot basis.
It was the brainchild of one of our comp supes, Toby [Wiesner]. We'd been looking at some machine learning techniques to extract depth mats from monocular photography. Also, it gave us segmentation. We had LiDAR, which was captured on set, and we would convert that to deep data. We would then remap these depth passes based on that deep data, which gave us a very cohesive and controllable depth space. So within that, we could insert our CG horde. We could insert any sort of weather we wanted.
That would give the compositing artists a very good basis to start with. All we'd ask them to do is really identify the wind direction in the plate, which again could be varying. There could be no wind. You'd look at surrounding shots if that was the case. The system would basically populate a number of particle systems—all based in Nuke—which would be the blowing snow, a number of volumetric elements which had been pre-generated by effects. This system would lay them out based on depth from camera and conform all of that to the wind direction and the selected density. Basically, you'd have a fairly convincing weather system inserted into any plate.
Then where it was a little bit flickery or the segmentation was a bit rough, that's where we would employ roto to then go in and actually fine-tune all of the edges and things like that. What that meant for animation is also that we were able to give them a pretty good presentation. So even at an early stage, animation was able to insert their work in and show it to the client with a very good representation of what the end shot would be.
CGW: Did you utilize any other software tools for the horde work?
Dennis Yoo: We used Massive quite a bit. Massive is great in the background, but we rarely put it in front of the screen anymore. We actually had to start transferring that motion into our real puppets so that we could have simulation on top of that. That made things a lot more heavy, but we had processes in place. On top of that, we used Ragdoll, which is simulated dynamics on top of things where you can add influences to them. It gets a little goofy when you add too much, but it's just that subtlety that you need.
Then from there we use motion edit on top, using Nuance. That's a dedicated animation software that deals mainly with FK puppets. And then Maya, where the animators like to make new motion, bespoke motion, and things in that nature. With Nuance we tend to preserve motion, but we can bump things around and move things. It was a combination of all of them. Not everyone can use all the tools—they usually specialize. So there's a lot of jumping back and forth between different motion departments. I'm impressed with how we did it, to be honest. It was real. We all worked quite gracefully.
Nick Epstein: It was a very interesting mix of disciplines on the motion side and also on the effects side as well, the simulation side. Crowd work sometimes conjures up images of compromises or cutting corners because there are so many of them. You know you're not really going to see them, so you don't need to simulate the cloth or the hair, or you can get away with coarser simulations. We ended up with the highest number of bipeds that we'd ever simulated at Wētā in terms of cloth and hair. If you look right towards the back of the horde, I think it really made a big difference. You'd see these profile breaks where the cloth would sort of billow in the wind and the hair would flap around and so on. I think that adds this level of realism that you would expect to observe in a plate, so we should have it in our CG as well.
In terms of software, Loki is our in-house proprietary solver. You could call it a universal solver. It's more like a framework of lots of solvers that talk to each other. So you can have a cloth solve that also has air influence. It can be influenced by water and vice versa. You can have a cloth solve and, in our case, a snow solve interacting with each other. So for things like that creepy clicker head where it's rotating into the ground, we'd be able to simulate the snow and then have that also simulate the cloth and the hair at the same time. I think that just adds to it a bit. It gives you much more of a cohesive result because you're simulating all of these things at the same time.
Dennis Yoo: For the explosion shots, the crowd’s running through and it's on an angle. There's no place to hide in our plates, so there was no place to hide in our shots either. It wasn't dynamic fire lighting where you get these dark contrasts. It was all diffused and all daylight. On set they shot it practically, and there's no shockwave or anything. Just people running through flaming balls of fire. Later on, Craig said they needed to be falling, flying, and exploding. So we’re not matching the plate here at all. That went on to using simulation to get explosions, and keyframing to handle some of the bigger explosions.
A lot of the time people go, “Just do motion capture.” But we're under the same restrictions in motion capture as any sort of stunt work. You can't hurt people capturing motion. They're not actually stepping on each other. They're not getting blown apart. We have to add that later.
In this small section, animators are going in and getting people blown up and flying away. Then in another section we have motion editors grounding everything, making sure that their foot contacts are working. Making sure they're not running through each other and bumping into each other individually. Then Massive in the background on top of that, and they're running on top of each other. It was daunting.
Nick Epstein: Basically, it's a whole heap of simulations and traditional motion work that all needs to cohesively work together, and there's no magic bullet for that. There's no software that will just give you that. So it needs to also be a predictable process that you can go through, and your director or your showrunner can direct you through it. It was a lot of back and forth, a lot of exploratory work. Most shots have 90% CG horde, 10% stunt horde and you can’t tell what’s CG and what’s not.
CGW: Can you tell us about creating the massive fireproof bloater?
Nick Epstein: He was new for Season 2. We as a facility built the Season 1 bloater, but they wanted something larger, more imposing. Tommy's trying to rally everyone to defend the town, but things are really bad—and then it gets even worse. You hear this bloater roar in the distance, and then Tommy looks for him and sees him. So that initial reveal had to be like, “Oh shit, it's a really big bloater, too.” We put a new head crown on him. His cordyceps design went through a few iterations with Craig and Neil [Druckmann]. Obviously this is Neil's creation, and you're showing him designs for a bloater that he's never seen before. So it was cool to get his feedback and make sure we incorporated all of his ideas.
We ended up with this more armor-plated bloater. He was further complicated by the frost layer. We scattered lots and lots of ice and snow on him, which geometrically made him a little bit more challenging. The ice was properly refractive and so on. All of that had to be accounted for in shading and rendering. I think we learned some things from Season 1 in terms of how we designed him.
Dennis Yoo: In Season 1 we had cordyceps around his groin area. If it was real-life action, he wouldn't be able to move his leg—it would be in the way. So there's a lot of crunching going on, and luckily the lighting helped us there. We could hide a lot of things. So in this season, knowing that we can't hide anything, now we’re very adamant on where we put these cordyceps and how they're not going to hinder his movement. He needs to be able to walk and run.
We started the simulation on top of him quite early. He wasn't just all cordyceps, he was muscle in there as well, and fat. So I thought we got that jiggle and volume on him working quite well. In Season 1 we had issues only because we started a little bit later. So starting early, we got to have more iterations and see things in lighting. I thought his muscle systems and his legs were working great.
Nick Epstein: We had to get that balance. You want some movement to him, but you also don't want him to feel like he's super jiggly everywhere. They really wanted to differentiate the areas, so some of the cordyceps plates needed to be quite rigid. Then you've got these big slabs in his leg which would move up and down appropriately with his footfalls and so on.
Dennis Yoo: That shot of him breaking through the wall, that was one of the shots that we planned for. The deeper you think about how wood splinters, the crazier it gets. My job was just trying to figure out how does this bloater break a log like that? In the previs he just punches, and the thing explodes. But Craig was adamant that no, he's not a superhero—he's grounded in real physics. So it was a hard process trying to find the right performance, the right energy. He was punching logs, but he couldn't blow them apart.
Nick Epstein: There's a nice sort of almost humanity, right? When he manages to get through, he actually has to take a pause because he's exhausted from punching through that wall. It was a really nice touch because at one point, he was human. He gets tired like the rest of us, even if it takes a bit more to get him tired.
Dennis Yoo: Our motion was always grounded. We try to use as much motion capture as we can, and then enhance on top. We had [Isaac] Ike Hamon, who was our motion stunt coordinator at Wētā FX. He was a Season 1 bloater as well, so they already were attuned to his acting. I think we used a couple of actors, but every time I showed his motion, Craig would be like, “Yeah, that's him.” So we knew exactly how he moved. It was great having him.
One of the things that we were struggling with was, how does he break through the wall? Instead of me iterating on motion capture, which takes a little bit of time, I just opted to film a bunch of footage with Ike. Holding a bunch of pads and him banging on it, just to get the feel of what sort of energy he wanted. I gave Craig a bunch of takes, and he was very appreciative of that. He picked little segments that he liked, and then from there we did that acting again with the motion capture suit. So that definitely helped us find that performance.