Issue: Volume: 24 Issue: 8 (August 2001)

Digital Diva



By Gregory Peter Panos

performance animation

Ray Kurzweil is a world-renowned scientist, inventor, and techno-evangelist. But one role he has never played-until now-is that of female rock star. Using motion capture and numerous other computer graphics technologies, Kurzweil is able to become Ramona, a real-time virtual chanteuse who helps deliver his technological message.

Ramona's genesis began at last year's Siggraph conference, where Kurzweil was introduced to the latest performance animation, digital puppetry, and 3D human scanning and modeling technologies. Kurzweil pondered how these tools could be used to present his concepts of "life in virtual reality" in future speaking engagements. Eventually, he decided to create a traveling road show starring a 3D singing, dancing, female alter ego with realistic moves and facial expressions. This real-time computer-generated performance, complete with backup dancers, would be projected onto a large screen behind Kurzweil as he performed.

Helping with the high-profile project, which was developed in Kurzweil's lab outside of Boston, was a team of engineers and artists, as well as a number of companies. Among these were Kleiser-Walczak Construction Company (KWCC), known for its "synthespian" film and television characters, motion-capture vendor Ascension Technologies, graphics card maker 3Dlabs, system developer Compaq, animation software vendor Kaydara, and 3D scanning system manufacturer Cyberware. In all, more than 50 people collaborated in the development of Ramona.
Ray Kurzweil (far left) becomes CG rock singer Ramona (above), while his daughter Amy (left) performs the movements of Ramona's backup dancers. (Images courtesy Ray Kurzweil.)




Kurzweil found among his own staff a woman to serve as the basis of Ramona's form. She was digitally scanned at Cyberware in Monterey, California, and the resulting data was imported into Kaydara's Filmbox, then tweaked and animated. A skintight wardrobe was chosen for the photorealistic Ramona, both to emphasize her proportions, and also because tight clothing presented less of an animation challenge than loose-fitting garments.

Using a speech-input feature in Filmbox, the team could control real-time facial animation data, allowing Kurzweil to perform Ramona's singing while on stage . To accomplish this, many 3D facial expressions from Ramona's live model were acquired during the scanning sessions. The resulting facial deformation targets were integrated with the speech-phoneme recognition engine in Filmbox, allowing Ramona's face to morph in real time into the expressive singing personality envisioned by Kurzweil.

With a proprietary vocal-shifting tool created by colleagues in the music business, Kurzweil also was able to change his voice-in real time-into one appropriate for Ramona. Finally, the group used Ascension's Wireless MotionStar magnetic system to translate Kurzweil's body movements and Filmbox's voice interpretation features to turn his facial motions on stage into those of Ramona. Amy, Kurzweil's daughter, performed the movements of Ramona's backup dancers, controlling three synchronous line-dancing characters who were created from information guru Richard Saul Wurman's 3D body scan data, then composited over a changing variety of virtual background locations.

Ramona's first scheduled appearance was at Wurman's annual TED (Technology, Entertainment, and Design) conference in Monterey. The audience responded to Ramona's debut with enthusiasm, although there was at least one visible technical complication. Other devices in the theater interfered with the magnetic motion-capture system's sensors, causing Ramona's image to undulate. Kurzweil's short-term solution was to dance up a storm so that Ramona never stopped gyrating long enough for anyone to notice.

Ramona is scheduled for a number of future performances, but Kurzweil has other plans for her as well. One aspect of Ramona's incarnation is as "conversational avatar in residence" on Kurz weil's new Web site, kurzweilai.net. As new technologies evolve, raising the bar of believability for Ramona, her possibilities will expand. Will she be able to compete with other cyber personalities such as Kyoko Date? Only her alter ego, Kurzweil, knows for sure.

Gregory Peter Panos is an inventor, writer, and evangelist in the fields of human simulation, virtual reality, and wearable technology. He has twice served as elected chairman of Siggraph-Los Angeles and can be reached at panos@siggraph.org.




KEY TOOLS: Filmbox, Kaydara (www.kaydara.com); MotionStar, Ascension Technologies (www.ascension-tech.com)