by Diana Phillips Mahoney
Breaking up is hard to do, but when a relationship has clearly run its course-when one party is dragging another down-there's nothing left to do but say goodbye. Over the next few years, countless designers, artists, engineers, and scientists for whom visual computing has become standard practice will be doing just that. They will be bidding farewell to the tools that for years have provided entry to their digital worlds: the keyboard and mouse. As happens with the demise of any long-term relationship, the initial separation might be difficult. But the trauma will soon be diminished by an unbridled appreciation for the freedom they've attained thanks to a new crop of technologies aimed at changing how users interact with 3D digital data.
 |
| Geared toward easing the creation of 3D organic forms such as this ballerina, a unique software interface developed at CalTech lets users model objects by moving a Cybergloved hand through space. |
This revolution will be a boon to nearly all of the professionals toiling in the computer graphics trenches for whom using a mouse to interact with high-end 3D visualizations has become the technological equivalent of using pedals to power a Porsche. Although it might "work," it is clearly limiting the power and potential of the medium.
"The speed of computers and the memory capability is increasing exponentially, while user interface technology has only been increasing linearly over time. As a result, there's this huge gap between the capability and the usability of the computer," says Thomas Massie, founder of the Cambridge, Mas sachusetts-based haptic-interface company SensAble Tech nologies. "You're not going to make a computer twice as usable by doubling the resolution on the screen. It's obvious that the interface is what needs to change."
The awareness of such a need is the driving force behind recent commercial and research initiatives aimed at improving user interfaces. This is particularly important to companies involved in high-end interactive simulations. While the graphics technology for such applications might be top-rate, the value of that technology is diminished without similarly state-of-the-art interactive capabilities.
"Our customers are running more complex, sophisticated applications, but they feel hampered by the traditional mouse/keyboard/joystick paradigm," says Jeff Brum, marketing manager at virtual-reality company Fakespace Technologies (Mountain View, CA). "With the keyboard and mouse, you're typing in data and you're having to move and click to get things done. You might not be able to perform functions quickly enough to gain the information you want, or you might not be able to navigate as easily as you want. It's clear that people can process visual information more effectively if they're using more natural, familiar interaction methods."
 |
| The Fakespace NeoWand lets users navigate through and manipulate virtual objects with the push of a button, as well as transfer control of the interaction to a user who may be nearby or halfway around the world. |
Toward this end, Fakespace has recently introduced a number of new interface tools to help its 3D display customers get the most out of their visual data. One of the company's the recent product releases is the NeoWand, a handheld input tool for use in the company's immersive display environments. The device, which looks like a flat iron and feels like a familiar remote control, has a series of buttons that operate such functions as navigation, manipulation of virtual objects, and menu viewing and selection. "[The NeoWand] lets the user highlight, grab, and take control of an object, and hand that control to someone who might be nearby or halfway across the world," says Brum.
The company has also developed a navigation device called the NavPod, which acts as a steering wheel through a virtual environment. The user grips each end of a 12-inch long horizontal bar and pushes or turns it to drive the viewing perspective. "When someone navigates with a mouse, they tend to use the cursor as their pointer and means for identifying their location in virtual space. But a 2D cursor interrupts the 3D effect," says Brum. "The NavPod lets users navigate 3D-scale models, such as a prototype of a car, airplane, physical plant, or cityscape, without interrupting the 3D experience."
Another new interface being marketed by Fakespace is the Cubic Mouse, initially developed by the National German Research Center for Information Technology (GMD) for the exploration of 3D seismic data. The handheld, cube-shaped device enables users to manipulate huge sets of 3D data. Three rods that run through the cube's axes are used to set orthogonal slicing planes that can be moved through a visualization to reveal the interior structures. Turning the cube in space changes the position of the volumetric data being viewed, and clicking a button freezes the position. "Once you find the precise coordinates that you're looking for, you can lock the object, then rotate and view it from different angles," says Brum. "This lets users locate any point in a huge volume of data as quickly and accurately as possible."
The device is cube-shaped to mimic the traditional cubic representation of seismic data. "It's familiar to people working in geophysical applications," says Brum. "The idea was to create a tool that they wouldn't have to spend time thinking about or learning how to operate, so they could focus on the visual information." Extending the CubicMouse beyond geophysical applications is a natural progression of the technology, Brum adds. "We've demonstrated it with automotive data, cutting planes through car designs, and also with 3D MRI data, finding precise coordinates through the brain."
With its eye on the future, Fakespace recently joined forces with SensAble Technologies, the Australian Government's Cooperative Research Centre for Advanced Computational Systems, and Teneo Computing for a technology demonstration of a haptics-based 3D display system that could revolutionize the analysis and interpretation of large datasets. The focus of the demonstration was a virtual-model display system designed to enable users to feel and manipulate stereoscopic images. The system combines Fakespace's projection-based portable Mini WorkBench display with perhaps the most significant interface development of the past decade-SensAble's Phantom haptic device. The Phantom comprises a stylus attached to a force-feedback arm, which, in conjunction with the WorkBench display, enables users to not only see their 3D data as if it were floating physically in front of them, but also to feel and interact with it in the same way. Among the potential application areas to benefit from such a system, says Brum, are CAD, assembly planning, biomolecular engineering, geophysical exploration, and education and training.
Before such applications can be realized, however, the system has to be refined for commercialization. "There's still a lot to iron out," says Brum. "The mechanical integration of the Bench and the Phantom is relatively straightforward, but the software requires further development. There are a few tricks to getting the perspective correct to feel like you are interacting with this object hovering in space."
 |
| The three rods of the Fakespace/GMD Cubic Mouse set orthogonal slicing planes that can be moved through a 3D visualization to explore interior structures from various perspectives. |
Although the Haptic MiniWorkBench does not yet exist as a commercial product, the Phantom certainly does, and the tool and its manufacturer have been making interface news since coming on to the computer graphics scene in 1993. "The Phantom was a distillation of my earlier interface research, which was focused on trying to build a device that encompassed the whole hand," says Massie. "I realized that simulating interactions with something at the end of a handle is more reasonable than trying to simulate something that's going to touch every point on your hand." Instead of trying to put the user's hand through "the looking glass" of the computer screen, he says, "the approach we've taken is to extend a tool handle from the computer side out of the glass toward the user, so the user grabs the handle and interacts with the computer that way."
This distinction ties directly into SensAble's business objective. "We are not trying to re-create reality. We are trying to make a better user interface, one that's more intuitive because it uses the same metaphors you use in the real world." Consequently, the company is not pursuing a "bigger, better" Phantom that lets a user whole-handedly reach in and manipulate a digital scene. Rather, says Massie, "our goal is to take the technology we have and basically improve upon it, get more applications to use it so it becomes available to a wider array of users."
SensAble's first giant step in this direction was the introduction late last summer of its Phantom-based FreeForm modeling system, with which users are able to sculpt and form digital clay and foam using the same tools and techniques that are employed in the physical world. The resulting objects can then be delivered downstream to other design or CAD packages for refinement or further development, or they can go directly to a rapid prototyping device to create a physical part from which molds can be made.
 |
| Sensable Technologies' haptic modeling tool FreeForm lets designers focus on their creations rather than their tools. At Siggraph 99, early adopter Io Walczak easily transferred her sculpting skills to the digital medium. |
Such a capability provides one of the "purest" means for maintaining design intent throughout a production process, says Massie. "It gives the creators more control over the output because they don't have to rely on a lofty process or an army of individuals to produce the design they conceptualized." Additionally, it enables a more efficient production process and a better end product. "If a change needs to be made to a digitally carved model, you don't have to carve another one from scratch out of clay, you can modify the one you've got." It also lets designers experiment with shapes that they couldn't easily or possibly obtain using traditional CAD tools. "A lot of the CAD-generated objects we see around us look the way they do because that's what CAD can make. [With FreeForm], we're not constrained artificially by these things."
While the eventual integration of the Phantom and FreeForm with 3D displays such as the Fakespace MiniWorkBench will certainly enhance the technology's appeal, the fact that 3D display media have not yet reached the masses makes the haptic device more critical, says Massie. "Touching something with the Phantom resolves all of the ambiguities on the screen. For example, if you see a shaded, rendered sphere on screen, your brain is trying to interpret this 2D projection and reconstruct a 3D image from that without the advantage of binocular vision. So you're subconsciously making decisions like whether the sphere is concave or convex. As soon as you touch it, you know immediately what it is."
Classically trained sculptor Diana Walczak of the Kleiser-Walczak Con struction Company is one of the early adopters of the tool, and has already come to appreciate the creative freedom it enables. "I was trained as a sculptor. So when I started computer animation, I found a strong disconnect between my self and the virtual sculpture, because of the missing sense of touch and because traditional [computer] modeling tools require considerable training and mastery of technical processes." FreeForm, she says, "speeds up the organic modeling process by orders of magnitude, because it provides a more direct link between the artistic thought process and the end product. It frees the sculptor to create."
 |
| Still a "concept" technology, the Haptic MiniWorkbench combines a Fakespace stereoscopic 3D display with the SensAble Phantom to enable users to feel and interact with objects appearing to float in space. |
In addition to commercial efforts by SensAble, Fakespace, and many other companies to bring user interfaces up to technological par with today's visual computing capabilities, academic and corporate research facilities worldwide are researching ways to improve the human-computer interface using innovative software, hardware, or both.
Among the many such works-in-progress is an interactive artistic medium developed at the California Institute of Technology. Called Surface Drawing, the system lets users create intricate, organic 3D shapes without the constraints im posed by traditional design software and the mouse/keyboard interface. The system is implemented with the Responsive Work bench (a 3D interactive tabletop display) and an 18-sensor CyberGlove to meas ure hand movements. Users create objects by moving a hand through space. The path of the hand forms surface pieces that merge when they touch. With a stylus in the non-gloved hand, users can add and erase information and manipulate the objects.
Caltech researcher Steven Schkolne developed the system as a way to compensate for the discrepancy between traditional artistic media and computer design software. "The biggest problem with today's 3D programs is that they have so many tools and features that you spend more time thinking about the tool and the interface than you do thinking about the art that you're making," he says. "Instead of sketching things out, you put in points and then push and pull them to specify what you want. It's all very mathematical, but not very artistically intuitive."
One of the limitations of Surface Drawing compared to traditional computer modeling tools is that it doesn't offer users absolute control of their designs. "If you're trying to build something with absolute sharpness or perfectly right angles, you could have some problems," says Schkolne. He notes, however, that the tool is really geared toward organic forms where such precision isn't necessary. In fact, he says, "the need for precision has kept people trapped in old-style computer modeling tools. If you look at the way artists work, there's no computer there enforcing their accuracy. Artists learn how to control their bodies to communicate what they want to communicate. A similar principle is at work with Surface Drawing."
As is the case with other new interface media, Surface Drawing is not meant to mimic reality, but rather to complement it. "One of the neatest things about Surface Drawing is that it's something that doesn't exist in the physical world, and can't exist in the physical world. Imagine having this substance that you make float in the air. It can't really happen. But [the tool] does allow people to work in a familiar manner-generating ideas and thinking about the model as they're working on it, rather than worrying about the precision or numbers behind it." Schkolne is in the process of refining Surface Drawing to make it easier to create smoother, sharper objects. Currently, there are no formal plans to commercialize the tool, although a number of companies have expressed interest in the technology.
Video and computer vision are the driving forces behind another unique interface called the Visual Conductor. Developed by researchers in Lucent Technologies' Bell Labs, the Visual Conductor uses two video cameras to capture a stereo view of the hand and baton expressions of a live conductor. Using computer vision, the system detects beat events and gestures related to rhythm patterns and dynamics and uses this information to control the tempo and volume of orchestra music delivered from the computer through a MIDI (Musical Instrument Digital Interface) system.
In addition to the musical simulation, the researchers developed an animation program through which the motion of an articulated, rendered ballet dancer could be synchronized in a "non-trivial" way to the music. "Trivial synchronization would be just scaling the time-speed ing up and slowing down the movie. That would n't work for dance, because if the conductor were to stop when the dancers are in mid-air, they would remain in mid-air," says system developer Jakub Segen. Consequently, the researchers developed a non-linear time scaling algorithm that incorporates an awareness of the physics of the dancer's body. It will not, for example, allow the body to remain in mid-air. Some part of the dance will be finished before the music slows down or speeds up. Both the dancing and the music are driven by the Visual Conductor interface.
 |
| Artistic freedom rather than geometric precision is the creative goal driving the CalTech Surface Drawing system with which these objects were created. "The need for precision has kept people trapped in old-style computer modeling tools," says CalTech' |
According to Segen, the Visual Conductor addresses two of the major pitfalls of existing computer interfaces: limited degrees of freedom and some physical, mechanical connection to the computer. "Current interfaces, such as the mouse and the joystick, have too few degrees of freedom to be used effectively with 3D interactive programs. Even though I can do high-end graphics and simulations on a Pentium, I cannot emulate anything using a mouse that would be even remotely similar to, say, reaching and grasping an object. I can invent tricks to compensate for this, but the tricks have to be learned and inevitably feel awkward," he says. In contrast, using computer vision, "one can get an infinite number of degrees of freedom, because eventually, every pixel of an image can be turned into a degree of freedom."
 |
|
Additionally, the Visual Conductor is a wireless interface. "You don't need to touch anything. It removes the user from the machinery," says Segen. "Right now, we are too connected to our machines-the terminal, keyboard, and so forth. With [the Visual Conductor], the captured input serves as the spatial interface." The fact that the interface is "deviceless" is critical to the acceptance of such tools, he says. "People don't like change, so the ultimate interface is one that doesn't involve any changes to the user's natural environment."
Among the developmental challenges still to be met before the technology behind the Visual Conductor can be commercialized is the creation of a generic tool that's going to be useful for broad application. "This is a very young field, so there are many open issues on the basic research level," says Segen. "For example, we don't have a method that would allow me to follow every joint, every part of a human body, rather than just the hands, using only the cameras without changing the environment." There's also the issue of economy. "Conceivably, we can build systems that will work right now, but at incredible cost. We need to figure out how to do it at reasonable cost."
The technology behind the Visual Conductor can potentially be applied to many areas. One example, says Segen, is robotics. "Controlling a robot by hand has been a longstanding problem. This approach would enable such control through the use of cameras mounted around the room." Other logical application areas include 3D computer games in which multiple degrees of freedom would ease navigation through scenes and business presentations, in which smooth and natural interaction is a key to success. "No one wants to have to fool around with awkward devices," says Segen.
There's no question that the most effective user interface is one that is comfortable and natural. And the most comfortable and natural interface is one that's nearly invisible. "The goal is to engineer an interface where there's no separation between the control and the presentation of information, and to hide the complication of that underlying engineering," says Hiroshi Ishii, an interface researcher at the Massachu setts Institute of Technology.
This was the objective driving Ishii's development of a tangible interface called musicBottles, which uses bottles as containers and controls for digital information. The system comprises a specially designed magnetic table and corked vessels that "contain" the sounds of various musical instruments. The bottles are embedded with electromagnetic tags for identification. When a cork is removed from a bottle, the corresponding instrument is heard. A pattern of colored light is rear-projected onto the table's translucent surface to reflect changes in pitch and volume. Users can structure a musical composition by physically manipulating the bottles.
 |
| Ordinary glass bottles that are fitted with electromagnetic tags and placed on a special magnetic table (below) contain a virtual symphony in MIT's musicBottles system. In addition to music, the bottles could be filled with virtually any kind of digital i |
Although it's called musicBottles, the interface is limited only by one's imagination. "The bottles are generic objects. You can put any kind of information you want to in them," says Ishii. "You could have bottles with stocks you've invested in, and by opening the bottles, you could access updated stock information. Or you can have stories or music or images."
Ishii employs a bottle metaphor because bottles are such familiar objects. "There's no uncertainty about how to use them," he says. In addition, such tangible, comfortable interfaces have the advantage of flexibility. "You can easily put them anywhere-in the kitchen, living room, bathroom, wherever they might make sense." In contrast, he says, "the PC box with its keyboard and mouse doesn't fit to the architectural space in which we are living. It's more like a special hole to cyberspace. It's very disconnected from the everyday environment. Our efforts are aimed at giving all information some physical form, so it becomes not only a representation, but also an interface or control."
The bottom line for all new interface development is that users want tools that will let them realize the full potential of their data. "People want to spend more time working with their data than having to figure out ways to get to the information they need, and they certainly don't want to be hampered by the tools that are supposed to help them do that," says Jeff Brum of Fakespace. "The more they can move beyond visualizing their data to 'realizing' it through enhanced interaction and navigation, the more appealing and useful the technology as a whole will become."
Diana Phillips Mahoney is chief technology editor of Computer Graphics World.