Imagine, if you will, being able to converse and interact with a compelling digital character or group of characters one-on-one, in a private setting. More specifically, having an actual two-way, progressive conversation and experience based on information that is being exchanged in real time, as opposed to the character providing a string of "canned" responses that barely relate to the discussion or activity. Also, imagine that the digital character can carry on that two-way chat/experience as part of an entertaining and engaging variety/talk show - a show hosted by the character, with you as the featured guest.
Thanks to a number of technological innovations and a great deal of creativity and imagination, a small crew of engineers and artists has brought this experience to life with The Winston Show.
After approximately a year and a half of development, The Winston Show made its debut recently - not on cable television, but via an iPad app geared for young children. The "show" is hosted by the comedic character Winston and his sidekick Ellington and comprises a number of activities personalized and unique for each user based on the human responses. Currently, there are five sketches, housed within separate "studios" on the show's virtual lot: Win with Winston, where you can test your knowledge on a quiz show; Fireside Chats, where you can talk to Winston about your favorite topics; You Vs., where you can challenge "colorful" characters; the Costume Department, where you can dress up and play backstage; and the Writer's Room, where you can create your own tale and watch it come to life on the page. There is also a backstory that introduces the cast of characters and their unique world.
Stage 1
The "producer" of The Winston Show is a new company called ToyTalk, cofounded by CEO Oren Jacob and CTO Martin Reddy, with an office in San Francisco. While their names may not be immediately recognizable, they were some of the top talent from Pixar, where Jacob served as CTO and Reddy as lead software engineer prior to that he specialized in AI. They started ToyTalk nearly two years ago and received funding through various sources, including Greylock Partners, Charles River Ventures, and others. After creating a prototype based on technology they devised, they were able to pique the interest of a Pixar coworker - enough to lure him to their new venture.
Bobby Podesta, director of The Winston Show, comes with an impressive pedigree, as do Jacob and Reddy: Podesta served as animator and story artist on a number of Pixar films, directing animator on Cars, and supervising animator on
Toy Story 3 and the Finding Nemo Submarine Voyage at Disneyland.
"Oren and Martin showed me what they had put together, and I thought it was really exciting. And the thought of being able to create characters and entertainment using speech recognition on an iPad device was enticing," says Podesta. "So, I joined them. At ToyTalk, we thought, Wouldn't it be great to make animated characters who could break through the fourth wall?"
As Podesta explains, with television shows such as Dora the Explorer and
Sesame Street, characters talk to the audience and then wait for the audience to respond, before continuing with the show. Typically, there is a few seconds of delay, and the conversation or communication is single stream - while the person may answer the character, the character cannot hear them. But with a microphone, camera, and iPad device connected to the cloud or Internet, there is now a two-way stream, "and you can have those characters reach through the fourth wall in a believable way and create a new type of entertainment," he says.
Moreover, the AI enables the characters to become more intelligent as they continue to converse, leading to even more engaging interaction.
Why would someone leave what many, including Podesta himself, consider a dream job at one of the industry's top studios - and one that many consider the best at storytelling - to join a start-up? "I joined Pixar in the '90s because I saw something in Toy Story that was amazing and intriguing that I wanted to explore," he says. "It's a similar situation now. I see a lot of potential in this and the tools [ToyTalk] has come up with to create 'believable' interactive experiences."
Podesta cites both his feature-film (especially the Toy Story movies) and ride-film experience as beneficial in this new endeavor. "I am always excited about emerging technology to create entertainment," he says. "This is super exciting, and the technology creates a big, open world for an artist to do crazy things."
Stage 2
As Podesta explains, the phrase "in the can" is bantered about quite a bit in the film and television worlds. "You are done with the project, and that is it. There are no further changes," he notes. "Because we are creating something that we can update at any time, we are able to add content and depth as we go for a constantly evolving experience."
One of the biggest differences between The Winston Show and film or television projects is how ToyTalk executes the animation. The team is starting out with six hours of content - which entails the characters' sides of the conversations - and that, in terms of time, could double depending on the interaction. "That's a lot of animation!" Podesta points out.
The content creation process is similar to that for any other animation. Ideas are brainstormed, storyboards are generated (Podesta hand-draws them in Adobe Photoshop), and they are pitched to the entire company, which comprises approximately 20 employees presently. Once the green light is given, the concept goes to the writers. Like in television, the production design and animation teams work with modelers to build the characters; next the voices are recorded. Then, it is all put together.
"If you would walk into ToyTalk today, you would think you were at a film or television studio," says Podesta.
The basic tool set is similar, too. The team uses Autodesk's Maya for the modeling and animation cycles. "We hire animators who know the software so they can animated as they normally would," says Podesta. "We create lots of pieces of animation, and they are edited together - bits of animation are assigned for one line. It's almost like editing the performance."
The driving force behind The Winston Show is the proprietary technology created at ToyTalk: It's called PullString, and it was created from scratch to make The Winston Show possible. "We wrote the software in-house because there is nothing really out there for creative people to use to write for an artificial-intelligence character. This type of software is usually put together for an engineer who knows computer code. I wanted to hire writers who knew how to write and didn't want them to worry about the technical side of things," Podesta explains.
So the writers use PullString to devise the scenes, and the software acts as the AI engine that drives the whole experience. PullString is sort of like a game engine in that it is the brains behind the project. "It is what pulls the artwork that is generated, where we put together the animation after it's created in Maya," says Podesta, "along with the background images, the characters, the sets, lines, audio, and dialog needed for a scene." PullString grabs it, compiles it all, and that is what drives the app on the device. The animation rendering is done live; it is not pre-rendered.
Stage 3
Devising PullString was indeed the biggest hurdle, but there were other technical challenges as well, mainly because of all the technology being used behind the scenes - from speech recognition that is ingested into the AI engine, to the animation being displayed on the screen and rendered out in real time, to many others that make the interaction seamless. "You don't necessarily notice or pay attention to [the technology]. What's important to us are the characters and the stories," explains Podesta.
The characters and imagery are bright and simplistic in style to be attractive for kids and to facilitate the real-time interactions, especially since the interaction and conversation interpretation consumes a good deal of the processing.
Of the five sketches, the most challenging to do was the Fireside Chat, since there is a good deal of back-and-forth conversation. "There are so many things to may be talked about, and we had to create a believable experience," says Podesta. Here, the voice recognition software sends the verbal responses to the cloud and converts the speech to text. Then, the information is processed and PullString devises a suitable answer or response. Another big challenge was the Writer's Room, wherein the user gets to decide how their story progresses, while a living, animated page comes to life on the other side.
The app, which launched September in the Apple Store, is free of charge. It contains the five main sketches, which are geared toward children five years and older, although, as Podesta points out, the team did not "write down" to the children.
From making toys and characters talk at people on a movie screen, to enabling them to converse with people, the team at ToyTalk is focused on continuing to break new ground in entertainment.
"The idea of having characters you can really go back and forth with through that fourth wall is something that just has not been realized before and opens up a wealth of possibilities in terms of entertainment, and my hope is that this is just the beginning of something bigger," says Podesta.
Karen Moltenbrey is the chief editor of CGW.