Augmented Reality, My Dear Watson
July 16, 2021

Augmented Reality, My Dear Watson

When “rip” is used in a script, it’s regularly a job for the computer graphics team. Sometimes that’s a matter of creating dynamic tears in a costume’s cloth simulation. Other times, it could be slashing the skin of a 3D creature during an epic battle. 
But what if you’re asked to visualize something as abstract as a rip in the space-time continuum? How about one that needs to have its very own story arc? And what if you need to do all this while collaborating with a tentpole production team during the COVID-19 pandemic?

Most of us wouldn’t know where to start. For the artists behind NVIZ, however, it was just a new challenge they had to solve while working on Netflix’s supernatural crime series, The Irregulars

Turning a Set Piece into a Character
An eight-part series inspired by Sherlock Holmes, The Irregulars follows a gang of troubled street teens in Victorian London who work with Holmes and Watson to solve crimes.

On the show, “The Rip” isn’t just a magical effect. It’s an essential part of the story: a tear between the natural and supernatural worlds that functions as a character in its own right. 

“The Rip continuously expands and contracts to reflect key turning points throughout the series,” says Eduardo Schmidek, Virtual Camera Operator at NVIZ. “These include when the woman Sherlock loves sacrifices herself to the Rip in order to save the world; or when the show’s villain, The Linen Man, plans to absorb the Rip’s powers to become a god.”

Each of the Rip’s many movements – which affected both environment lighting and talent interactions in the final shot – couldn’t be accurately depicted through concept art alone. Instead, extensive previsualization and AR virtual production from NVIZ was required to give the on-set team full control over the final result. 

Better Lighting with Augmented Reality VP
First, NVIZ artists worked with director Joss Agnew, DP Nick Dance and VFX Supervisor Richard Briscoe to generate extensive CG previz sequences involving the Rip, via remote interactive sessions in Unreal Engine’s Editor mode. 

The team then leveraged Ncam’s camera tracking system, together with NVIZ’s proprietary AR solution built with Ncam’s open-source AR Suite plug-in for Unreal Engine, to overlay the CG lookdev and animation from previz on top of footage captured by the camera on set. 

Because they could see a real-time preview of how the Rip would affect lighting, characters and environments as it moved, Agnew and Dance could easily make creative decisions on elements such as framing or camera rigs during principal photography rather than having to rely on imagination alone. 

“Ncam was very helpful for integrating the Rip into the environment the way it was required,” Schmidek says. “With Unreal and our proprietary augmented reality tools, we were able to display the Rip floating on the set, even for shots without green screens.”

This was incredibly useful for visualizing a sequence where the Rip opens inside a dark cave, illuminating the environment with an electric blue glow. “In the story, the Rip grows, causing the cave to be further bathed in the changing blue light as the scene progresses,” Schmidek continues. “Without AR, you would just see a plain disk and soft light. With AR, we could fully understand and prep the lighting on set, and the effects that were based around it.” 

A Safer, Faster Production 
The carefully planned workflow also enabled The Irregulars team to be far more efficient during principal photography, helping them budget how much CG the series would need and showcase the overall pace and rhythm of the scenes before post production. 

This was crucial to ensure the series could logistically be completed despite COVID-19, with minimal crew and on-set hours. By the time the project was over, the NVIZ team had used AR powered by Ncam tracking to deliver 23 CGI-heavy shots from just four days of shooting – far more than would have been possible with a traditional workflow.

“Planning each sequence meant there was far less time wasted setting up different camera rigs,” Schmidek explains. “For example, there were some scenes looking down at the Rip from above and it looks like a thin membrane. Being able to see the AR version of the shot was important to not just ensure lighting was accurate, but also to place the camera at the right angle.” 

The workflow was such a success that the NVIZ team sent the composited AR versions of the shot straight to post production. This allowed final post and VFX teams – including Double Negative and UPP – to work from the Rip’s lookdev as inspiration, speeding up their delivery.

“At the end of the show, everyone was very happy,” remembers Schmidek. “It was emotional. We went from not knowing whether production could continue, to being able to deliver stunning AR effects that everyone on set could see in real time. And we did it faster than ever.”