SquareZero composited the body of an impersonator with the rotoscoped head of singer Frank Sinatra from film footage to create a hologram of the star.
|
When singer/actor Frank Sinatra died in May 1998, fans the world over mourned his passing. Born to Italian immigrants, this kid from the blue-collar, working-class city of Hoboken, New Jersey, eventually would turn into a superstar whose nicknames included The Chairman of the Board, The Voice, and ’Ol Blue Eyes. A member of the so-called Rat Pack, Sinatra became a quintessential legend of radio and Hollywood.
It’s no surprise that countless people—young and old alike—were big fans of Sinatra’s music and movies. Among them is UK native Simon Cowell, music executive, television producer and personality, and entrepreneur, who is best known as the highly critical judge on the reality TV show American Idol. So when organizers began to make plans for a 50th birthday bash for Cowell at a facility in England, they were determined to have Sinatra make an appearance. Of course, it would be impossible to have the famed crooner appear in the flesh, but having him appear digitally, in a hologram, well?…?that had possibilities.
|
“The party organizers wanted the hologram to look 100 percent real, as if Sinatra himself had stepped onto the stage for a private performance,” says Vicky Godfrey, director of SquareZero, a London design, animation, and production facility that brought the singer back to digital life.
The most challenging and time-consuming part of the project was locating source material for the hologram. Banana Split, which commissioned the project, wanted the holographic singer to belt out “Happy Birthday,” although no footage could be found of the real Sinatra warbling that tune. In fact, the selection of possible songs was extremely narrow due to the limitations of the project.
Foremost, the footage had to be continuous and filmed with a single locked-off camera. “The holograms do not work well if you have edited source material; you just can’t do anything with it,” says Godfrey. “The hologram has to look as if a real person is standing in front of you, not [clips of] an edited show. It has to be as real as real can be.”
Moreover, the footage had to be full-body, to make it appear as if Frank himself was making the guest appearance. Otherwise, it would appear as a floating head or partial body, which was obviously undesirable.
Godfrey spent nearly three weeks just tracking down footage that would work for the hologram after digital magic was applied. “We couldn’t find any footage of Frank singing ‘Happy Birthday.’ I suggested that the client think about having Marilyn Monroe, but Frank Sinatra is Simon Cowell’s idol, and it had to be Frank Sinatra,” she says. However, the available public archive footage of Sinatra singing was filmed in close-up (head-and-shoulders shots) for films. The best possible footage was eventually found—albeit from Sinatra’s own private collection that had never been in the public domain.
“That is why it was shot with a single camera and not edited. It was for his own use,” Godfrey says.
Godfrey spent weeks negotiating with Sinatra’s estate, and eventually received permission to use the footage for this project; there would be no encore. And, the estate gave SquareZero two choices: footage that used a single-camera feed of the crooner singing Pennies from Heaven or Learning the Blues. SquareZero chose the former, though it still had limitations: Sinatra walks toward the camera and then back, showing him, at most, from his knees upward. His lower body is missing.
|
|
|
Continuous single-camera footage of the entire singer was needed for the project. When that could not be located, digital techniques were required to achieve a head-to-toe image. All told, 24,000 frames of Sinatra’s head were roto’d from film footage within Apple’s Shake, and comp’d onto newly acquired footage of a stand-in using The Foundry’s Nuke and Shake.
|
The Project Gets Legs
To make the footage work, the crew at SquareZero had to give Sinatra legs and a lower torso, onto which they planned to fuse the rotoscoped upper body from the original footage. The needed body parts would come from a body double. When SquareZero revealed its plan to the client, Banana Split decided to ditch the project, thinking it required too much work.
Then, just two weeks before the party, the client changed its mind, asking SquareZero to resume the project. At this point, Godfrey and her group closely examined the footage again only to realize that the original plan of compositing the stand-in’s footage from the knees down would not work. “After reviewing the footage from Pennies from Heaven, we realized that we had to cut Frank out at the neck level, not the knees,” says Godfrey. “If you are trying to connect [the images] at the knees, you have to be in perfect sync. It’s really difficult. But with the head, as long as you have the same movement with the body double, you can get away with a little more.”
So, suddenly after getting the green light on a Thursday and the contract on Friday, and revamping its strategy, SquareZero found itself fighting the clock. The group located a body double, a Frank Sinatra tribute performer who was the same height as the star. That was important, explains Godfrey, because the head had to have the same proportions as the comp’d body.
On the following Monday, the team filmed the body double at the studio. To ensure that the double’s movement was perfectly in sync with Sinatra’s, a low-resolution version of the film footage was projected onto a screen, producing a mirror image for the performer to follow while he practiced, concentrating on the body position and motion.
Yet, getting the necessary high-resolution footage, from which Sinatra’s head would be rotoscoped, proved difficult again, this time on another level. “There were gigs, and gigs, and gigs of it, and it all had to be sent via FTP from Los Angeles,” says Godfrey, noting that the file took at least 24 hours to upload. Then came the download. “We would start downloading the file, and it would crash. And then we would have to start again. And again. This process started on Friday, and we didn’t get the high-res footage until the following Thursday. It took nearly a week!”
Creating the Man, the Myth
While the crew struggled with the files, a DOP filmed the body double, who was dressed in a suit from the late ’50s/early ’60s, to coincide with the period when the archive footage of the younger Frank was made. Key to the shoot was the lighting: It had to be just right so that the performer’s dark suit would still show up against the black backdrop that was used in the hologram. The lighting also had to match the lighting in the original footage. However, Godfrey notes, that lighting was not conducive to what was required in the hologram rig—an issue that the director of photography eventually resolved.
“At this point, we had only eight days left before the party, and still had to do viewings with the clients,” says Godfrey.
Head of animation Olly Tyler and compositor Gabriel Sitjas worked nonstop over the weekend to rotoscope 24,000 frames of Sinatra from the original footage. Not only did they have to separate out his head, but the roto—which was done using Apple’s Shake—also included his eyes, neck, mouth, hat, and hat brim within the cut. “We cut out the stage set from the house [where the original footage was filmed], so we just had Frank standing there,” describes Godfrey.
Within the original footage, Sinatra moves around somewhat, mainly in a front-to-back pattern. This action resulted in the size of his head becoming larger and smaller in camera. Lead compositor Jonson Jewell had to counteract this movement, or size differential, so that the singer’s head remained at the same size and in the same position by stabilizing it and keeping it locked on that one plane.
“With the holographic projection system, when a person walks back and forth, they get stretched in an odd way, and you end up with an element of distortion at the show that we wouldn’t have had any control over,” says Godfrey. “We just didn’t want to risk that, so we kept him relatively still.”
Jewell then worked around the clock mainly in Nuke (The Foundry) and a bit in Shake to convincingly attach the cutout head footage to the newly filmed but headless body double. This task was especially tedious: The slightest mismatch of the head and body would show up badly around the neck area. So Jewell tracked the body double’s neck area and applied that to the head using Shake and Nuke, making minor readjustments every half second throughout the song.
Animated shapes were used to patch up areas around the neck and collar, some of them used to create artificial shadows and lighting, which in turn gave the head the correct-looking volume and depth. At one point in the sequence, where the body movement and the head just would not match up, the team created a short morph sequence using a different section of Sinatra’s head footage, which fit the body better.
Once the head was locked onto the body, the team used Shake to colorize Sinatra’s face (the archive footage was black and white). They also utilized the Furnace tools that ship with Nuke to de-grain the footage and restore film damage inherent in the footage. The finished image was then added to a black background.
Although Godfrey thought that keeping Sinatra in black and white during his hologram appearance helped situate him more in the era from which he came, the client opted for “living” color. “They wanted him to look as real as possible, as if he just came back from the dead,” she explains.
During Sinatra’s recent stage appearance, his image starts out in a picture frame with the original footage, and then the footage “steps” out of the six-meter frame, all the background disappears, and the holographic image, which is in color, performs the song. Two projectors provided the necessary brightness on stage.
For the playout system, a Musion Eyeliner 3D holographic projection system, which required an HD static full-body camera shot of Sinatra on a solid black background, was employed. The system uses a technology similar to Pepper’s ghost, an illusionary technique that makes a 2D image appear solid, in 3D, though to do so, the object, or in this case, the character, has to have some slight movement.
Blast from the Past
While SquareZero and countless other facilities have composited heads onto body doubles in the past for commercials and other projects, Godfrey says there is a big leap from doing those sorts of projects to one like this. “You have more flexibility and can hide things behind an edit. Here, we couldn’t do that,” she says. “The hologram was life-size and appeared right in front of the audience’s eyes, so we couldn’t do the typical cheats that are done for TV and film. And, the shot had to be one take, continuous.”
SquareZero had been asked to bring other entertainers back to life, including Miles Davis and Freddie Mercury, but the necessary element, Godfrey says, is in locating footage of an entire song captured with a single camera. “What gets archived is just the edit from the multiple cameras, not the footage from each camera,” she explains. “We were lucky to have found [the Sinatra] footage.”
Perhaps soon we will see more celebrities who are no longer alive performing digitally. For Sinatra, though, this was a very special one-time appearance.