Some storybook-themed television series made their debut last season, following in the success of other shows whose premises were lifted from the pages of popular rhymes and tales. These live-action productions are filmed on locations but are enhanced by a range of visual effects - at times the sets and backdrops rely on CGI, while in other instances, characters emerge that are fully or partially digital. And, all of them feature VFX that drive the story magic.
Once Upon A Time, a popular fantasy/drama, centers on well-known story characters who are cursed to live normal everyday lives, but plot lines often bounce between their "real-world" and fairy-tale selves.
This past April, the Once Upon A Time spin-off, Once Upon A Time in Wonderland, closed the book on its final chapter following a 13-episode run that began last fall. The series was based on the "Alice in Wonderland" premise, but like the parent show, the action takes place in present-day bizarre Wonderland with flashbacks to the equally bizarre Wonderland before a curse took hold.
Merlin, a fantasy/adventure loosely based on the story of King Arthur, is set in an earlier period of Camelot when Merlin and Prince Arthur are just getting a feel for their power - Arthur as he learns how to become the future king, and Merlin as he begins to master magic.
Grimm, on the other hand, takes viewers into the darker world of the German Brothers Grimm, where characters inspired by the tales and writers' imaginations appear in modern-day society as creatures that shape-shift from human to make-believe creatures that often bring death and destruction.
The mystery/drama Sleepy Hollow is based on the classic Washington Irving story of the Headless Horseman, only with Ichabod Crane as a Revolutionary War hero who finds himself in present-day Sleepy Hollow. There, he and a female detective battle the Headless Horseman, who is determined to bring about the end of days, while Crane tries to stop him.
CHAPTER 1
Once upon a time…
…There lived an evil queen who cast an awful curse on the fairy-tale characters who lived in a great kingdom near the Enchanted Forest. The spell washed over the land and transported the inhabitants to a far different world where they lived without their fairy-tale memories. Before the spell could take hold, though, the valiant Prince Charming and his loving wife Snow White secreted away their newborn princess, Emma, through a magical wardrobe portal, which, unbeknownst to them, also sent the infant to the real world.
Baby Emma grew up and began living life on the edge. With little to offer, she gave up her baby son for adoption. Meanwhile, the king, queen, and subjects began life anew in Storybrooke, Maine, as ordinary folks - that is, until a curious young boy led his birth mother (Emma) to the small town where he lived with a powerful woman who ruled in the real world as mayor but as the Evil Queen in the fairy-tale realm.
This unlikely princess, the key to the fairy-tale characters' happily ever after, eventually learns and accepts her role as savior, and continues to battle evil both near and far while trying to provide the happy ending these characters, and her son, deserve.
That is the premise for the television series Once Upon A Time, a popular fantasy/drama that appeared in 2011 on ABC and will continue this fall. Each season brings new adventure, as the core characters regain their past memories and as various new characters magically arrive in Storybrooke. Nearly every episode starts out in the real world, but flashbacks and secondary plots unfold in the fairy-tale realm, as well. The Evil Queen, Rumplestiltskin, Snow White, Prince Charming, Belle, the Seven Dwarves, Grandma, and Red Riding Hood are among the regular cast. Others have been brought on for longer-term plots, including Captain Hook, Peter Pan and the Lost Boys, and the Wicked Witch of the West, while a slew of others made limited appearances.
Principal photography occurs in Vancouver, British Columbia, where the nearby village of Steveston serves as the fictional "real-world" Storybrooke, Maine. That location in the Pacific Northwest provides a lush locale for the woods in nearby Storybrooke and in the fantastical Enchanted Forest.
In addition, there are a number of practical sets employed on soundstages. Both the inside and outside locations are often supplemented with visual effects, created by Zoic Studios, with Andrew Orloff, co-founder and head of episodic television, serving as overall VFX supervisor on the series.
Zoic and Orloff have been involved in this giant fairy-tale mishmash from the start. While episodic shows can be challenging in terms of budgets and time, neither the studio nor Orloff are strangers to this demanding workflow. Typically, the VFX requirements for each episode are revealed approximately two and a half weeks prior to the live-action shoot. That is when Once Upon A Time's Production Designer Michael Joy and art department provide the concept art for the virtual sets, props, creatures, and most of the magical effects. The VFX artists, approximately 70 in all, start with these initial designs, flesh them out, and optimize them for real-time playback on set during filming - the key to creating the vast kingdom, the real and fantastical worlds, and plethora of effects.
Zoic's work on the series spans the gamut of visual effects, although the majority of the work, by far, are the backdrops - "everything you would see in a Hollywood fantasy movie: virtual backgrounds shot on greenscreen with everything composited into them, exterior and interior environments, fantasy environments, realistic New York City environments, and all shot on a greenscreen stage," explains Orloff. One particularly complex environment was Captain Hook's ship, which appeared at night, during the day, at sea, in a storm, and even sailing across the sky. "The goal is to make the CG environments realistic, even when they are fantastical," he adds.
In order to handle this large volume of landscapes and architecture, Zoic uses ZEUS (Zoic Environmental Unification System). This virtual production system, developed in partnership with Lightcraft Technology, provides real-time camera tracking and rendering of the virtual environments on set, enabling the DP, cast, and crew on a greenscreen stage to see the virtual environments during filming, and gives the director a rough composite of the actors on set tracked against the digital background.
"The highly detailed environments and imaginative characters of the series' fairy-tale world have challenged us to expand and fine-tune both our ZEUS and animation pipelines for television," Orloff says.
ZEUS was developed to handle the large amount of virtual sets in the TV show V, but with all the subsequent television work Zoic does, including Once Upon A Time, the setup has proven itself vital in the production processes. ZEUS comprises various components, including the Lightcraft virtual set-tracking system with InterSense motion tracking, Shotgun's shot management software, Autodesk's Maya 3D modeling software, and The Foundry's Nuke compositing program at the core. Tracking markers mounted to the ceiling of the soundstage, along with equipment mounted to cameras, provide the data necessary for ZEUS.
On the front end, Zoic's proprietary iPad application lets the director and creatives examine the set library so animatics can be built for previewing the virtual environments. "They can take measurements and make storyboards, comments, e-mail frames, and more," says Orloff. "We also write a bunch of proprietary tools that take the camera information from Lightcraft and put it into our tracking pipeline so we can get through a large volume of shots faster than ever before. We have scripts that integrate and manage the assets, interface with editorial, and export the assets for automatic delivery."
The volume of shots and complexity of the editorial demands are extremely high, so on the back end, the pipeline automates a good deal of the process, so the artists are creating rather than managing, contends Orloff.
Enchanting Work
With a grueling schedule and limited budget, Once Upon A Time's production and visual effects teams must establish a balance between using practical and virtual sets. For example, on one episode, the set consisted solely of a table and columns; ZEUS delivered the dark walls and glowing fires that set the mood for a tense scene. In another, it was used to build a ballroom wedding scene, extending practical columns, adding stained-glass windows, and filling the room with digital guests. And that was just the beginning.
In addition to the digital sets, the show uses a substantial number of matte paintings - in fact, three full-time matte painters alone work on the production, using Adobe's Photoshop along with Maya to give the scenes parallax.
Backdrops and sets aside, Zoic also has its hands full with the show's magical effects, such as portals, smoke, fireballs, freezing and barrier spells, locking and unlocking spells, and disappearances. Environmental effects, such as clouds, tornadoes, lightning, and storms, are plentiful, too. "We do a lot of the particle effects and magic designs," says Orloff. "Those are the second most popular effects in the series and are in just about every show." For transformations, the group often uses digital scans of the actors - for instance, to create an adult version of Pinocchio, once the actor was scanned, the artists generated a wooden texture, then rotoscoped the texture and matchmoved it onto the character in the shots.
For this past season, flying monkeys were the rage and a vital story element to the Oz theme of several episodes. Another challenge this season was creating the ghost of Cora, the Evil Queen's mother. The director wanted her to perform like an actor yet react like a spirit, so the crew made a combination real/CG character. Creating the hybrid required filming the actor's reactions and mapping them onto the CG character so that her face was real but the rest of her body was not, and then cutting that combination character with actual reactions of the actor in costume. The end result was a transparent ghost that was active, powerful, and scary, says Orloff.
Other CG characters included a whale, werewolf, wolf pack, Medusa, Ursula, ghosts, fairies, dragons, a giant turtle, and even a dementor-like character. "We have done a lot of diverse character work for Once Upon A Time," Orloff notes, "bugs, snakes, critters on land and on water…the list goes on."
Hand-in-hand with the magic effects are the physical reactions to those magic effects. "Often we are augmenting or cleaning up and adding to physical effects," says Orloff. "For instance, the Blue Fairy is a combination of a real and CG character - an actor suspended on wires with CG wings. We do everything we can to ground the effects in reality."
An average show contains between 300 and 400 visual effects shots, and those can go as high as 700 to 750.
Some of the more memorable work on the show, in Orloff's opinion, occurred in Season 2. The first sequence was with Pinocchio, where he and Gepetto are on a stormy ocean and are swallowed by a whale. For this scene, the person playing Gepetto was filmed on a practical raft in a water tank, while Pinocchio and the whale were CG characters. Practical wind and wave effects provided turbulence, while Zoic added CG water that was a simulation created by Fusion CI Studios using Next Limit's RealFlow to extend the water environment. "That was so incredibly challenging from a logistical and an animation standpoint," Orloff says, noting that Pinocchio was mocapped but the whale was animated by hand.
Another complex scene had Peter Pan's shadow, a virtual character, grabbing a live character (on wires) and dragging the person as they flew through the CG streets of Victorian London. "We had to build an entire period London that was huge in scale to accommodate the fly-through," Orloff says. "Planning was really critical to get the shots we wanted because we had a minor up on the rig and a stunt situation, along with all the CG." Animatics and a lot of concept work helped the group perfect the timing of this wild ride over the chimneys and rooftops, with Big Ben and the Thames visible in the background.
To build these and other environments, and to create the characters and effects, the artists use Maya and Nuke, part of the ZEUS pipeline. Some procedural modeling is done, but the majority is block and tackle modeling with assets pulled from the image library built over the years. Scenes are rendered with V-Ray from Chaos Group. "We generate a set of layered EXRs that have a bunch of different passes, including all the separate lighting passes," explains Orloff. "With that, we can make lighting adjustments in Nuke without having to go back to the CGI. That is something Mike Romney, head of pipelines here at Zoic, made happen by working directly with Chaos to ensure we would have all our passes put together with proprietary Nuke scripts that can read and combine those passes automatically."
Texturing is done using Pixologic's ZBrush, with Autodesk's Mudbox used from time to time along with Photoshop.
Visiting far-off lands and dealing with a wide range of creatures and characters is a weekly occurrence on Once Upon A Time, and there's very little that the series' creators can throw at Orloff that surprises him. "We let the creative drive the process, and we are always ready," he adds.
CHAPTER 2
Alice: And how many hours a day did you do lessons?
The Mock Turtle: Ten hours the first day, nine the next, and so on.
Alice: What a curious plan!
The Gryphon: That's the reason they're called lessons, because they lessen from day to day.
- From "Alice in Wonderland"
The lessons may have lessened for the Gryphon, but they certainly did not for the VFX artists at Zoic who juggled the visual effects in the Once Upon A Time spin-off, Once Upon A Time in Wonderland, with Andrew Orloff once again in the role of overall VFX supervisor.
The single-season, 13-episode Wonderland - which takes place in present-day Wonderland with flashbacks to pre-cursed Wonderland - contains countless virtual sets. In fact, Orloff estimates that more than half of each episode uses virtual environments, and here, like in the original series, the ZEUS system proved itself in spades. The Wonderland environments are far more fantastical and improbable, exactly what is expected down the rabbit hole. There are castles, but for this show, they are in the mountains, on an island, and they even float in the sky. The Queen of Heart's abode is metallic and made of chess pieces, leading one reviewer to call it "splendid."
"There is a huge amount of virtual set work and design work in the show - the exteriors and interiors are very stylistic and unusual, exactly what is expected in this universe. We had an outdoor environment that was a boiling sea below a cliff, where half the sky was day and half was night," Orloff describes.
Whereas in Once Upon A Time characters will enter a forested outdoor environment, in Wonderland, they enter a mushroom forest that just cannot be filmed in a real setting. "You have to make some type of augmentation or virtual set for almost every environment," says Orloff. "Nearly everything in Wonderland has to be touched to some degree."
There are a number of CG characters, including the White Rabbit (who is a piece of mastery) that regularly acts within a real and digital environment, as does the CG Cheshire Cat, which is shown climbing a tree. A team of 15 animators worked on that sequence alone, collaborating with editorial to make sure they had the proper action for the cat, especially since there were live actors interacting with the CG feline within the virtual environments.
"We got one piece of concept art from production, and we created the Cheshire Cat from that one piece of flat artwork," says Orloff. The group modeled the cat in 3D using Autodesk's Maya, and initially used a combination of Joe Alter's Shave and a Haircut and Yeti, a fur plug-in from Peregrine Labs, to generate a feral version of the animal with wild, wild hair to match its wild, wild personality. As the show progressed, however, the studio began to solely use Yeti because it integrated better within the ZEUS pipeline.
Unlike the parent show, Wonderland's effects are outrageous rather than realistic. But like the parent show, they ground the characters into the story world and support the plot, no mater how outrageous. Or how difficult.
March Hare: Start at the beginning.
Mad Hatter: Yes, yes. And when you come to the end...
[chuckles]
Mad Hatter: STOP. See?
CHAPTER 3
A king can only work with his best tools."
― T.H. White, "The Once and Future King"
Unlike the time-shifting that occurs in Once Upon A Time and Wonderland, the plots that unfold in Merlin stay rooted in Arthurian times. Alas, after many adventures through digital environments featuring magical CG dragons and creatures, and fighting dark magic from sources inside and outside the castle walls, it came time for the young king, his queen, and his faithful magician-companion Merlin to say farewell. So after five seasons, the BBC series turned its last page at the end of last year. But, not before Merlin revealed his magical powers to Arthur, just as The Mill and, later, Vine had been doing throughout each episode.
Merlin relied on subtle visual effects magic to tell the story of this powerful magician who managed to hide his capabilities from Arthur and those around him until the final episode. As such, the crew dealt with an average of 50 to 60 shots per episode, many of which are fireballs and enchantment-related VFX. However, as Merlin began to grow up and learn more magic, the effects had to progress as well, so during the last season, the work became more complex. And, Vine's Michael Illingworth, VFX supervisor for the series, was ready to perform his magic, alongside Merlin.
Illingworth founded Vine in 2007, and a group of five there produced matte paintings for Season 4. Most of the exterior castle shots were of an actual castle in Wales, with CG backdrops and extensions to make it more Arthurian. Also, a castle in France was often used for larger on-set shots, requiring the team to digitally replace modern fixtures for medieval ones.
"We did mattes of the French castle and made nighttime Camelot matte paintings while populating the grounds around the castle with medieval houses, churches, and such to give the impression the castle was the center of the city," Illingworth explains. By Season 5, the artists - numbering close to 15 - were making full-CG, matte-painted environments. Illingworth estimates that the group handled up to 10 matte-painted shots per episode.
While many believe that the interior castle shots were filmed on location somewhere in England or Ireland, they were actually shot greenscreen in a Cardiff, Wales, studio. The shots were done in a greenscreen "box" with a few props, and the CG artists then filled the box with a stunning computer-generated environment.
Of course, there were still the signature Merlin eye effects (glows), but during Season 5, the team expanded its role and created some complex CG characters, as well. The most difficult was the Euchdag/Diamair, a fully digital humanoid that was semi-translucent with a glow effect, had a lot of dialog, and whose face appeared in close-ups. Actor Josette Simon, who voiced the character, also provided the body movements and facial reference. On set, she was dressed head to toe in LED lights, so her hands and arms would emit "an interesting interactive light when she interacted with the characters," Illingworth explains. Later, the actor was replaced with the CG character.
Simon's facial performance was achieved using a facial capture and imaging technique from Dimensional Imaging (DI3D), which captures the entire face rather than just markers applied to the face. With this technique, an array of cameras encircling Simon's face captured every nuanced move, including her mouth movement for lip sync. "TDI3D captured nine separate video feeds of the character and used triangulation to get a 3D mesh of the full performance, including cheek twitches and eye blinks," says Illingworth. "It's not a traditional method of motion capture, but one that's been used in video games, and I thought it would be useful for us." In fact, Vine is using the process again for work it is doing on Atlantis.
Starting with Season 1, Merlin has featured a CG dragon created by The Mill, "a great asset we inherited," says Illingworth. The Mill's creature had been lit with Mental Images' Mental Ray but had to be developed to work with Vine's (Solid Angle) Arnold-based pipeline and shaders for the shots in the last season. At the end of Season 4, Vine got its turn to create one of these mythical creatures from scratch, modeling and animating a full-CG baby dragon that later turned into an adolescent.
Martin Rexard designed Aithusa, which was then brought in-house at Vine, where it was textured and rigged for animation. The model was designed in Pixologic's ZBrush and retopologized in Pilgway's 3D-Coat to get a leveled density in the mesh that would work better for rigging. The group then built the rigging in Autodesk's Maya and added cloth dynamics to the creature's wings. Textures were done in Adobe's Photoshop and The Foundry's Mari, with displacements for the scales done in Autodesk's Mudbox. "She had a rough life and was a battered soul," says Illingworth of the scarred creature.
With dragons come fire, which was mostly practical and composited into the scenes with The Foundry's Nuke. But in the final episode, when Merlin finally reveals his magic to Arthur, it is done around a campfire as Merlin conjures up the shape of a dragon from CG embers. For this, a short animation cycle by Sally Goldberg was produced in Side Effects' Houdini. She made layers of particles whereby the embers lifting from the fire fused together to form the dragon shape, then fell apart and floated off.
And just like that, Merlin showed off his magic skills with a subtle display, but one that required quite intense work and finesse by the VFX team.
CHAPTER 4
There once was a man who lived a life so strange, it had to be true. Only he could see what no one else can-the darkness inside…the real monster within…and he's the one who must stop them. This is his calling. This is his duty. This is the life of a Grimm."
Each episode of the police drama Grimm, which began airing in 2011, starts with a quote that relates to the plot and usually to the wesen creature featured in that particular story. In the series, these characters exist in the modern world and walk among us masked as ordinary people - until they suddenly transform into the fabled creature they are. In this series, the characters are typically grotesque and violent, and do not live happily ever after once the Grimm, a police homicide detective, gets hold of them.
The show is set in Portland and shot in and around the city, while a good deal of the visual effects work is done there, as well, with three vendors sharing the load, including Inhance Digital (Los Angeles), which works on nearly every episode in some capacity, with about every third one being all theirs.
In the beginning, Inhance did a variety of effects for the show - a lot of invisible things, such as set extensions, monitor burns, digital backdrops - but none that involved creatures, the main crux of the work. "We made it our business to learn creatures, and now we are doing a lot of them," says Eddie Robison, Inhance visual effects department supervisor. (See "In-hancing Grimm" in the July/August 2013 issue of CGW.)
The creatures in Grimm just don't suddenly appear in their full-monster splendor. Rather, they morph from human (actor) form into wesen, requiring digital assistance during those transitions. Sometimes the creatures use practical makeup and prosthetics, other times they are CG or a blend of the two. "When the actor wears makeup, we re-create that makeup in CG and our shots are used to transition the actor to the makeup," explains Robison. "The actor will have tracking markers on his or her face, and we will morph the person using all CG for four to five shots, bridging to the next series with the person in makeup." The artists also facilitate transitions back to human form in the same way.
That process is used in approximately half the wesen transformations. The other half entails no makeup on the actor whatsoever; the wesen character is completely CG. This process is mostly used for the really outrageous characters or the furry ones, such as Rosalee, a fox-like character, and Monroe, a werewolf. "When you see [Rosalee] as a Fuchsbau, she is all-digital," notes Robison. (Rosalee and Monroe, as well as other main characters that debuted early on and have recurring roles, are handled by some of the other facilities also working on the show.)
The group begins the process by acquiring reference photos of the actor, then completing a sculpt in Pixologic's ZBrush. "We do a head model, pushing and pulling on it to match it to the actor," says Robison, adding that photographic textures are then used for a convincing CG model of the actor's head. "It is only seen on screen for a split-second while the person is morphing," he adds.
At Inhance, Autodesk's Maya is used in the base character pipeline. In that software, the artists will add scales, fur, slime, whatever. The actors in the scene are tracked using Andersson Technologies' SynthEyes. The track is brought into the background in Maya and the character is attached. Rendering, meanwhile, is done in Chaos Group's V-Ray and compositing in Eyeon's Fusion. The fur is generated with Joe Alter's Shave and a Haircut.
While Robison has seen his fair share of unique and strange creatures, one that stands out in particular is the Aswang, lifted from Filipino folklore and featured in an episode this past spring titled "Mommy Dearest." The scary creature haunts pregnant women and uses its long, pointed tongue to extrude the fetus's amniotic fluid.
"I am going off to a house and entering it like a snake…I will devour their babes and make their hearts ache."
In its human form, the monster turned out to be a seemingly harmless elderly woman (the pregnant lady's mother-in-law). In its wesen form, it was a complex CG character designed, modeled, rigged, animated, textured, lit, and rendered by a handful of artists at Inhance. "In the past, there were scenes where we just transformed the actor, which is what Grimm is all about. This character, however, is fully CG and was animated in scenes scaling a tree outside the victim's bedroom and climbing on the ceilings inside," explains Robison.
Earlier in the season, the team created a Manticore, with the head of a lion and a tail of a scorpion. In this case, Inhance Artists Matt Lefferts and Andy Lewis used the CG Manticore head only for the four transformation shots and four others in which the actors weren't shot in the makeup, while effects makeup covered the rest. However, the scorpion tail was completely CG in every shot it appeared - nearly 35 shots when all was said and done.
That was just one of many monsters Inhance and the other studios wrangled for Grimm. Indeed, there is a wide variation of creatures in the show, and they are not hidden - "It is always done in a medium to tight shot in full light in all its HD glory, so nothing is left to the imagination," says Robison. "It all happens right there in camera."
"He stripped off his skin and tossed it into the fire and he was in human form again."
CHAPTER 5
From the listless repose of the place, and the peculiar character of its inhabitants, who are descendants from the original Dutch settlers, this sequestered glen has long been known by name of Sleepy Hollow…. A drowsy, dreamy influence seems to hang over the land, and to pervade the very atmosphere.
-Washington Irving, "The Legend of Sleepy Hollow"
In Washington Irving's classic, Sleepy Hollow is haunted by spectre, including the Headless Horseman, believed to be the ghost of a Hessian soldier from Colonial times. In the Sleepy Hollow television series launched last fall, the Headless Horseman is resurrected in modern times, as is Ichabod Crane, in a re-imagining of the story.
In this dark tale, Ichabod Crane is a Revolutionary War hero who was wounded in battle (but not before beheading the Hessian who dealt the battlefield blow). Crane awakes underground more than two centuries later in present-day Sleepy Hollow, as strange happenings start to occur. Crane and Detective Abbie Mills find themselves battling otherworldly forces, including the old adversary - one of the Four Horsemen of the Apocalypse.
So, not surprising, one of the signature visual effects of this live-action drama is the removal of the Headless Horseman's noggin, which also requires the creation of a CG collar and digital wardrobe fixes resulting from occlusions by the stunt actor's head in the shots.
Last year, work on the series was split among Pixomondo and Synaptic VFX, with some work by FuseFX, all tasked with producing feature-level effects on a television budget and timeframe. Currently, the show is posting its first episodes for the fall season, with Pixomondo back on the horse, so to speak, and thus far Origin Digital Studios riding alongside. "We do a little bit of everything, and each week there is something we didn't do the week before," says Jason Zimmerman, overall visual effects supervisor for Sleepy Hollow.
The number of VFX shots for a Sleepy Hollow episode varies: 30 to 40 at the low end, with the average being close to 100. Two particular episodes involving the capture and interrogation of the Horseman topped out at 700. The tool sets used to get the work done include Autodesk's 3ds Max and Maya, NewTek's LightWave, Thinkbox Software's Krakatoa, Sitni Sati's FumeFX plug-in for the CG work, and The Foundry's Nuke and Adobe's After Effects for compositing.
The visual effects work includes a good deal of roto/paint as well as particle simulations, such as dirt, dust, crumbling rocks, explosions, and liquids. In addition, there are set extensions, along with lots of fog and atmospheric effects that help give the show a spooky feel. The steed ridden by the Headless Horseman and his cohorts are real but have digital eye treatment for the red glow. For the most part, Pixomondo handled the simulations, simple to complex.
As Zimmerman explains, the show is shot in Wilmington, North Carolina, which serves as present-day Sleepy Hollow as imagined for the series. The southern location offers the crew historical structures for flashbacks to Colonial times, and there are at least a few in each episode. While most of the period sets are practical, at times CG set extensions are required. For example, in one episode, the artists at Pixomondo added a bell tower with lantern lights for the opening shot depicting Paul Revere's ride. The team also populated the shot to make the area look like an actual town, not a set. For the opening of another episode, Synaptic generated a large water simulation for the re-creation of Colonial Boston Harbor and augmented it with digital matte paintings.
There is a range of creepy characters that appear in the series, but most of them use prosthetics. "We don't do full-CG character work for the series," Zimmerman says. At times, though, CG is used to augment their appearance. For instance, Police Officer Andy Brooks, who became a follower of the Horseman, is often seen with a backward-facing head resulting from his death. Mostly this is achieved with a prosthetic and CG combination.
"We start with a practical prosthetic head that is on backward, and we map his face back onto that," says Zimmerman. "We usually do as much practical work as we can before turning to digital solutions. That is the look of the show. We are there to help production tell the story they want to tell."
Zimmerman explains that the VFX group tries to work with the various departments when they reach a point where a CG solution would be better than a practical one.
When it comes time for the Horseman to go headless, that task is split between two of the vendors, depending on the overall work load and shot count in the episode. It involves painting out a green hood worn by the stuntman and matchmoving a CG collar and neck stump. Often, it entails much more, Zimmerman points out. "We actually have a full body scan and sometimes have to replace arms, shoulders, and more depending on the angle of the shot," he says. Artists have to complete the scene by tracking in a clean plate created with what was occluded by the head before it was removed. Next, the artists solve the camera and track it to match the plate. Making this more difficult was the fact that the character is always in motion, whether riding on his horse or swinging an ax.
Synaptic also handled a vine animation in the finale, as well as a scene with swarming locusts. That compelling sequence involved Officer Brooks (John Cho), during which he enters a tunnel and is overcome by a swarm of locusts, which encase him in a cocoon. On set, flexible fabric helped sell the various stages of the cocoon, although the CG artists had to connect the stages and sequence them together. "You have the locusts swarming around his legs and wrapping his legs, progressing up his body and ultimately pinning him to the wall and covering his face in a large cocoon," Zimmerman says. "It involved some fun technical stuff to figure out the swarming and dynamics of the cocoon."
According to former Synaptic CG Supervisor Eric Hance, the locust sequence was "process rich." After getting the general idea of the sequence from Zimmerman, Synaptic started developing the key elements: the locust flocking and dynamic web wrapping. "Our team collaborated with The LightWave 3D group, which added functionality to their existing particle/flocking system to get the behavior we needed," he explains. "As we proceeded, we built and animated hero bugs and variations. During the R&D phase, we also rigged dynamic webbing in Maya to travel through the air and then properly stick to our digital version of the actor and the set."
Once Synaptic received the plates, the digital artists carefully matched the actor's performance and the set in a spatially correct way, so that webs could properly interact with John [Cho] and the set around him. After they got the technical underpinnings in place, the crew created an in-house edit of the sequence that would help give them creative direction. "We studied the patterns of real locusts and re-created what we thought were the most terrifying, disgusting parts," says Hance. They then continued to refine the performance of the bugs and the density of the webbing coming from the bugs and constraining the actor. The artists also created additional web mesh elements in 3ds Max to help achieve the final density of the cocoon around Brooks.
At the end of last season, a new character named War - an empty set of armor - was introduced. The armor was real, but the person inside had to be digitally removed and a sword of fire added in his grasp.
"With a character, we take a practical approach first and then see what we need to do to help," says Zimmerman of VFX's role. For instance, the team helped transition a character from a regular girl into a demon child for an episode. "In that case, we did a full CG scan of the girl in regular form and in the demon prosthetic version, and translated between the two looks." In her possessed form, CG limbs were added as she flailed wildly. Her face was also digitally replaced with contorted features during the transition and was practical thereafter.
Last season, Detective Brooks released Moloch's acolytes, who were sent to do the Horseman's bidding. Zimmerman, along with Alex Kurtzman (executive producer), Len Wiseman (executive producer), and Mark Goffman (executive producer/showrunner), were looking for a unique death scene involving the acolytes. "They just didn't want them to simply fall over or disappear," says Zimmerman. "So we worked together and came up with a shattering, breaking effect when they are hit by a bullet, so it makes them look as if they were made of rock." That sequence, handled by Pixomondo, required a particular simulation to achieve the final look.
In fact, the shattering effect was used a number of times in the first season. One instance happens when Abbie and Crane step through a portal - there is a forward shattering effect (achieved with a particle effect) that invokes an open door leading into long, monastic-like halls with lots of atmosphere and ancient aesthetics. "The rift in reality is a 3D environment that is ultimately the characters' door to the other world," says Zimmerman.
Pixomondo's Nhatphong Tran describes the work. "The shattering effect consists of two separate simulations, one being the cracking and the second being the blast. For the cracking, we made a hand-drawn crack map from reference photography that was shaped in Photoshop to match concept drawings of the portal. This was used to generate cracks that form around the actual portal with the [RayFire Studios'] RayFire plug-in." To animate the appearance of cracks, L-systems combined with conventional rotoshapes were used. To create the actual blast, the inner part of the portal was procedurally shattered using RayFire. Approximated geometries were used as soft force fields to deflect the virtual shards from hitting the actors. Additional hand-animated shards were placed in the scene for direct interaction with the actors.
The CG work was done with 3ds Max. The shards were rendered with two different shader setups using Chaos Group's V-Ray. "When the shards are still sticking in the wall, a glass shader is applied. Once they blast out, the shader switches to an opaque diffuse shader with reflective properties of glass that has the plate statically projected," Tran explains.
The hall was modeled as a really long hallway. At the end of the hallway, the artists placed an area light to blend into the other realm. "A lot of lights were placed between the pillars to produce atmospheric lights. Additional smoke and atmosphere were added in comp," notes Tran.
The most challenging aspect of the VFX work in Sleepy Hollow is not necessarily the characters or the set extensions, but rather the schedule: Typically, there were just 10 days from when the plates are received to final delivery, though this year there will be more time. Another challenge: figuring out the best way to approach some of the effects. "We play to the strengths of the vendors," says Zimmerman, noting the work is done by feature film-level artists, so the expectation is higher than for the average television series. "We have to figure out a way to make sure we deliver on that level week after week."
And, in each episode there is something different. "This week we might have to wrap someone up in vines and drop them into a grave, or come up with a CG portal. There's never a dull moment," Zimmerman says. "There's always something new and fun to figure out, and as a visual effects supervisor, that is why I started doing this, so I can have a good time with it."
No doubt, audiences can also expect something new and fun in each episode of Season 2, as new demons, plagues, characters, and even a haunted house find their way to Sleepy Hollow.