Pre-production in Real Time
Issue: Edition 2 2020

Pre-production in Real Time

Some might consider the pre-production work Dan May and his Painting Practice company do as "gameplay." And, in a sense, it is.

Painting Practice, a West London design studio May founded more than a decade ago, can be described as a previsualization/VFX facility, offering services such as previs, animation, VFX, production design, concept art, and digital matte painting. It has recently garnered attention for its design and visualization work for the large sets and complex action sequences on HBO's His Dark Materials. And they did so using real-time technology by implementing Unreal Technology's Unreal Engine.

His Dark Materials is an HBO epic fantasy drama series produced by Bad Wolf Studios and based on the book series of the same name by Philip Pullman, the first of which, "Northern Lights," was the basis for the feature film The Golden Compass (see "Animal Control," CGW, December 2007). The story introduces Lyra, an orphan, who lives in a parallel universe where science, theology, and magic are entwined, and where everyone has a personal daemon, a manifestation of his or her soul in the form of an animal.

His Dark Materials
A look at scene lighting for Trollesund.

The feature film was ambitious, yet the television series is even more so in terms of its scope and scale, with a plethora of creatures from witches to armored polar bears, and mythical cities and vast landscapes created by CG artists at Framestore (see "Crafting Daemons," at the end of this feature). Before that happened, though, Painting Practice provided real-time mock-ups of various sequences that enabled the director and producers to make decisions concerning the series' complicated sequences.

"Unreal was a great way of getting people to physically be in a space that was not even conceived," says May, creative director at Painting Practice and one of the company's co-founders.

As Joel Collins, executive producer and production designer on the show, notes, "These are very expensive sets, and on a show like His Dark Materials, where you're building an entire town for one episode, you've got to make really good decisions that are practical, financial, and creative. What Painting Practice's team created gave us an ability to make critical judgments that meant we could get it right on the edge of affordable and, absolutely, on the edge of shootable, in the sense there was no fat on what we built."

Painting Practice originally was set up to facilitate production design, whether physical or digital, but as more greenscreen sets were being incorporated, the company's work on previs became more plentiful, though it still does quite a bit of animation as well. "Primarily we're first-in-the-door, working for producers, directors, or showrunners on the vision of the show," says May. "We often help get them off the ground, and then we work through the traditional pre-production art department space, previs, and then in the last few years, with Black Mirror and now with His Dark Materials, we're very much hand-in-hand with editorial and the effects-room postvis, which is obviously a quite complicated and involved process because it involves spending money on things that might not even make it into the cut, but has to be good enough for people to make informed decisions about."

In some instances, the company will do final shots, especially if it is an environment and within the company's strengths, whereby it will do final matte paintings, as they did on the Netflix dystopian sci-fi anthology series Black Mirror.

"We're not looking to be a post house," says May. "We're purely there to serve and help facilitate productions or post houses to get them through design or tricky problems, whether it's an animation problem or [involving] previs storytelling."

Embracing Real Time

According to May, he had been looking at VR and game engines for some time as way to explore a space. He used the real-time technology to build the night club for "San Junipero" (Season 3 Episode 4) of Black Mirror. "Rather than just doing a few visuals and white-card modeling, I went the extra mile and built a VR experience that the director and producers were able to look at and experience firsthand," he says.

May began exploring this medium further, examining the use of game engines beyond gaming applications.

"His Dark Materials is full of complicated challenges. The show has a good budget, but with the ambition of the show, that budget is quickly stretched. So, you have to come up with creative solutions and find ways to resolve situations, and this is often the technology that can do it for you," May says.

His Dark Materials
Painting Practice uses real-time tech to aid design and vis for His Dark Materials.

His Dark Materials
A look at a smoke-filled scene of Bolvangar.

While filmmakers like James Cameron and Jon Favreau have been using virtual cameras to scout locations for some time now, the process up to now has been out of the cost range for television. But, Unreal has opened up that access to far more people.

Nevertheless, the integration of Unreal Engine into the process called for a retooling of Painting Practice's pipeline. When the company began its work with the engine, it had difficulty getting the different software to communicate, whereas now there are more scripts and plug-ins available to make the process easier and more automated.

For the most part, Maxon's Cinema 4D serves as Painting Practice's backbone - "It's great for previs and postvis, for effects work," notes May. In addition, the facility employs the Adobe Creative Suite, along with Adobe After Effects, and Premiere for editing and Pixologic's ZBrush for complex environment and character designs. The group also uses for collaboration as well as delivery, archiving, and referencing, along with Quixel products (Bridge and Mixer) and PureRef, freeware for making giant art boards.

Worlds and Characters

In essence, Painting Practice's Concept team for His Dark Materials on Season 1 (Season 2's release date is still pending) worked with the environments and creatures teams as well as with the production designer to help determine what the worlds and characters would look like. They would then review that with Framestore and Russell Dodgson, VFX supervisor.

"Sometimes the directors wouldn't come on board until late in the process, and we'd need to plan a lot of the sequences. So, it was my job, really, to have a go at making those sequences, with various outcomes and different tone levels," May explains. "Then when the directors would start, we would show them a bunch of things. Sometimes they would like the ideas or the principle, or they wouldn't necessarily like all the cameras or the speed or tone. It would be a hit or miss, but a lot of the time we got it quite close, and then we'd make the changes. But, it is really important and essential that the directors have creative ownership of the sequences so they can shoot some of that previs; otherwise, [the work] becomes null and void."

As May points out, there were many instances on Season 1 where the work paid off and was used shot-for-shot, and other times when it was used but was filmed a bit differently although the elements and beats were the same; alas, there were even times when everything changed, particularly when there were script alterations.

His Dark Materials
Unreal Engine enables the director and showrunners to plan out scenes.

His Dark Materials
A visual comparison of work in Unreal (left) vs. broadcast footage (right).

One of the particularly challenging sets Painting Practice had to design for His Dark Materials is Trollesund, the main port of Lapland in the world of Lyra Belacqua, the story's heroine. For this, the team made a scan of a quarry using photogrammetry drones and used it to create a high-resolution model to import into Unreal Engine. To this, the group added additional models and textures to complete the real-time mock-up.

The previs enabled the director and producers to explore the conceptual environment and make key decisions concerning the scale of the buildings and the set. The previs also aided in the storytelling aspect, so even though the area was called Trollesund Port, not all the action had to take place dockside, providing more flexibility for constructing the set, which was then extended digitally.

His Dark Materials
An overview of Trollesund by Painting Practice.

The environments are complicated, but a good deal of the series' budget from the start went toward the complexity of the daemons, which were photoreal, not stylized. After all, the characters are a big part of the story.

"His Dark Materials and Black Mirror have an air of fantasy and science fiction [respectively], but there's also a reality, a grounding, and that's the space where we fit well," says May. "And, they have a raft of very complicated, weird, and wonderful challenges that we aim to solve."

Another critical piece of previs the company provided was for a fight between two of the series' armored polar bears. Episode 7, called "The Fight to the Death," involves mortal combat between Lyra's friend, Iorek Byrnison, and the usurping king Iofur Raknison. According to Jamie Childs, who directed the episode, being able to experiment with a virtual camera on a set that did not yet exist enabled him to explore the best camera angles to maximize the scene's believability.

"I wanted to walk around and look at the bear fight because I wanted that fight to feel like it was a real fight being shot, not a CG fight," says Childs. "I could actually do that with Unreal. I could move around that room physically, get the camera in the position I wanted, and see on my camera monitor what was going on, and record those shots and cut it together - and that was really freeing for a director."

His Dark Materials
A sample of a video plan involving the characters in the popular series.

According to Childs, the sensation reminded him of his early days learning his craft, when he would go out and just film things to try them out, letting his creativity find the essence of the story. "I didn't really think that previs would help me do that side of things," he says. "I thought it might help me with technically [determining] where to put a camera and things like that, but it actually made me go, 'Right, I don't need to worry about any of the noise; I can actually just go and create something.'"

In the last episode of Season 1, "Betrayal," a great battle plays out with exploding airships firing at the bears and Lyra. The Painting Practice team dropped the director and DP into the virtual environment, some of which "never saw the light of day due to flooding [at the real location]," recalls May. Initially, that scene "blew the VFX budget out of the water because we were having so much fun," he adds. "Then we pared it back to what we could then shoot, although it was a painful process to create something that was really interesting and fun and quite beyond our wildest dreams, then smash it into pieces in a reality check."

His Dark Materials
The previs saves valuable shot-planning time on the complex production.

One of May's favorite and most successful sequences occurred in Season 1 Episode 4, "Armour," as Iorek retrieves his armor. "We planned out this town and built it in the game engine, and could visit it virtually inside the studio, rather than having to drive an hour and a half into the Welsh mountains," says May. "And we were able to make complicated and very cool animations of the bear running around and trashing the town. We were able to get in there and discover shots with the director [Otto Bathurst] in the various sessions. We would make multiple versions of the sequence, do an edit, and then have different versions of the edit, with different endings. We ended up with something that could be shot, was achievable, on budget, and Framestore was happy with the volume of shots. It was a collaboration among all the partners that pulled off well."

Crafting Daemons

Framestore played a key creative and collaborative role in the worldbuilding process for His Dark Materials, delivering all of the first season’s VFX and served as a key creative advisor from the earliest stages of production.

This saw Framestore build a production-side visual effects team at Wolf Studios in Cardiff led by senior VFX supervisor Russell Dodgson, on-set VFX supervisor Rob Duncan, and VFX executive producer James Whitlam. They provided guidance, advice, and input as to how best to bring Philip Pullman’s unique world to life – from feedback on the production notes to creating the show’s lush environments and developing the nuanced relationships between the actors and their daemons. 

The result is one of the most ambitious feats of storytelling yet seen on TV, with over 2,000 shots of spectacular CG and animation work bringing the richly-imagined universe and its colorful denizens to life. “His Dark Materials demanded an extraordinarily high VFX baseline,” says Dodgson.

Crafting a visual spectacle equal to Pullman’s beloved source material drew on a wide range of talent, and presented challenges in terms of scope, scale, and storytelling. “Every human character having his or her own daemon means, theoretically, every shot that features an actor could also require VFX,” explains Dodgson. “This level of work and detail is unparalleled when it comes to television – the sheer volume of shots requiring the inclusion of a pine marten, a hawk, a monkey, or a snow leopard meant our lighters, animators, and CG artists really had their work cut out for them.”

Framestore created more than 50 distinctive daemons as well as the bear-like panserbjørn, effectively bolstering the show’s overall cast with its CGI creations. To ensure emotional, realistic interactions between human actors and fully-CG characters, first-pass takes were conducted with puppets, which helped provide visual and interactional cues. The puppets were then removed from set, and clean plates featuring just the actors in situ were filmed. It was then up to Framestore’s artists to conjure forth the show’s distinctive daemons.

“On the one hand, there was the challenge of creating so many photoreal creatures,” says Dodgson of the process. “You’re building each one of these characters from the ground up, which means crafting their skeletons and musculature, and that’s before you even get on to the intricacies of feathers, fur, eyes, and claws.” Bringing these characters to life was not just a question of volume, however. “It was obviously important to imbue each daemon with its own personality,” says Dodgson. “A deeper concern, however, was how they then interacted with their humans, and how their gestures and behavior augmented, reflected, or concealed what their human was thinking or feeling.”

The scope and scale of Season 1 of the show meant Framestore drew on skills from across the entire company, with the work split between offices in London and Montreal. “It was a team effort on a grand scale,” says Fiona Walkinshaw, Framestore’s global managing director, film.“We’re no stranger to huge projects and tight deadlines, but His Dark Materials really feels like a landmark in terms of both the quantity and quality of VFX shots TV audiences witnessed. It’s a perfect example of how film and high-end TV are converging: There’s a heightened demand from clients and audiences alike for VFX work that would look as good in your living room as it would at the IMAX.”

As well as allowing the company’s artists to flex their creative and technical muscles, the work also tapped another of Framestore’s core skills: the ability to get inside a piece of existing IP and breathe life into complex creatures, characters, and settings that exist in the hearts and minds of a great many loyal fans. 

“This isn’t a world we’re building from scratch – it’s one that millions of readers feel deeply invested in,” Framestore CEO William Sargent points out.

His Dark Materials

Plan V

Soon interest piqued concerning Painting Practice's proprietary tools and Unreal Engine setup. The company received funding that helped push further development of the tools, and after some fits and starts, the visualization/effects company released Plan V. A new software app based on Unreal Engine, Plan V is a bespoke studio environment that enables artists, producers, directors, and others involved in a production crew to experiment with lenses, storyboards, previs, and more.

With a simplified workflow and a user--friendly interface, Plan V is designed to enable less technically inclined users to interact in a high-quality 3D environment so they can design worlds, sets, and scenes for films, television, advertising, and games. The tool also supports local and remote collaboration.

"You can get an amazing visual look out of [Unreal Engine] very quickly, and there's an incredible amount of assets that are now available [for it], so it makes building environments super quick. But expecting directors and writers and others who are not necessarily technically-minded to pick it up is a bit of a tall task," says May. Plan V makes viewing the Unreal Engine scenes simple and affordable.

Moreover, as crews continue to work remotely for whatever reason - be it pandemics, travel costs, personal or professional obligation - tools like this will become even more valuable.

As May points out, this is such a fast-paced industry, and it is more important than ever to keep relevant and keep moving forward.

His Dark Materials
Using Painting Practice’s Plan V.

As of this writing, work is in progress for Season 2 of His Dark Materials. Season 1 was a learning phase of sorts, as the group became familiar with the virtual production tools and cameras as well as the operation of the engine itself and the resulting collaboration with the vendors. On the upcoming Season 2, the group pushed the usage further, creating a virtual space where the actors could better understand the sight lines and so forth. And as the seasons progress, the technology will become even more useful, May predicts, in handling the increased fantasy elements and pushing the storytelling.

"A lot of Season 1 was just trying to get things to work and discovering cool ways of working and making some great sequences. In many ways, we made great things in previs that we probably couldn't make for real, ultimately. But doing those techniques and finding discoveries were still useful," says May. "Even though we were limited by our imaginations in Unreal, we had budgets, health and safety, and a million other things to consider. You have things in previs that we probably couldn't make for real, ultimately. But doing those techniques and finding discoveries were still useful," says May. "Even though we were limited by our imaginations in Unreal, we had budgets, health and safety, and a million other things to consider. You have to get realistic about what's achievable in terms of time and money."

In addition to His Dark Materials, May and Painting Practice are using the virtual pre-production process on other shows, which are under NDA at the moment.

"I think we're still in the early stages about how it's going to be used," May adds.

Meanwhile, the series recently received two BAFTA awards, including one for visual effects.

Karen Moltenbrey is the chief editor of CGW.