Chapter
in Emotions
in Humans and Artifacts, eds. R. Trappl, P. Petta, and S.Payr, MIT Press,
April 2003.
Originated from a symposium at OFAI in
Creating Emotional Relationships With Virtual Characters
Andrew
Stern
PF.
Magic
andrew@interactivestory.net, www.interactivestory.net
1 Introduction
During the workshop on “Emotions in Humans and Artifacts” participants presented research addressing such questions as, how do emotions work in the human mind? What are emotions? How would you build a computer program with emotions? Can we detect the emotional state of computer users? Yet as the discussions wore on, instead of providing some answers to these questions, they persistently raised new ones. When can we say a computer “has” emotions? What does it mean for a computer program to be “believable”? Do computers need emotions to be “intelligent”, or not?
These scientific, engineering and philosophical questions are fascinating and important research directions; however, as an artist with a background in computer science and filmmaking, I found my own perspective on emotions in humans and artifacts generally left out of the discussion. I find the burning question to be, what will humans actually do with artifacts that (at least seem to) have emotions? As a designer and engineer of some of the first fully-realized, “believable” interactive virtual characters to reach a worldwide mass audience, Virtual Petz and Babyz (PF Magic 1995, Mindscape 1999), I know this is a question no longer the domain of science fiction. Today millions of people are already encountering and having to assimilate into their lives new interactive emotional “artifacts.”
Emotions have been a salient feature of certain manmade artifacts, namely stories and art, long before the scientific study of emotion began. Looking to the past we see a tradition of humans creating objects in one form or another that display and communicate emotional content. From figurative painting and sculpture to hand puppets and animated characters in Disney films such as Snow White, manmade artifacts have induced emotional reactions in people as powerful and meaningful as those generated between people themselves.
Now we are at point in the history of story and art where humans can create interactive artifacts, using the computer as a new medium. We can now write software programs that can “listen” to a human, process what it just “heard”, and “speak” back with synthetically generated imagery and sound, on machines that are already in the households of millions of families around the world. The new urgent challenge for artists and storytellers is to create interactive artifacts that can induce the same emotional reactions and communicate emotional content as traditional non-interactive stories and art have done. In fact because the computer theoretically can be programmed to tailor the experience to an individual, it could become the most powerful medium of all for creating affective stories and art.
The question becomes, how do you do that? How do emotionally powerful stories and art “work” anyway? Alas, creating an artifact that produces a meaningful emotional reaction in a human is considered an “art” itself. Although techniques and advice on the artistic process have been published – works such as The Art of Dramatic Writing by playwright Lajos Egri, The Illusion of Life by Disney animators Thomas and Johnston, Letters to a Young Poet by poet Rainer Rilke – the act of creating emotionally powerful artifacts is by and large considered elusive, mysterious and unquantifiable. Even in art school the typical approach is to teach students to imitate (“master”) traditional styles and techniques, after which it is hoped the student will be ready to “find their own style,” which sometimes never happens.
Naturally it is difficult to discuss the art of creating emotionally powerful artifacts in the context of a scientific workshop, in the way one would approach a computer science or engineering problem – which helps explain the general reluctance to research the topic, and the not-so uncommon attitude among artificial intelligence researchers that the topic is “mushy”, ill-formed, or worst of all, unimportant. Art and entertainment is considered to be fun, not a serious pursuit. This view is short-sighted. On the contrary stories and art are among the most serious and meaningful pursuits we have. We communicate ideas and experiences to each other in this way. The fact that the problem is, to a degree, mushy and unquantifiable, makes it all the more challenging and difficult to undertake.
This paper puts forth virtual characters as an emerging form of manmade artifact with emotional content. Virtual characters go by a few other names in the AI research community, such as believable agents, synthetic actors and synthetic personalities. These are embodied autonomous agents that a user can interact with in some fashion, animated as real-time graphics or built as physical robots, which appear to have personality, emotion and motivation, designed to be used in art or entertainment. Over the past decade several media labs have been exploring the issues involved in building virtual characters (such as Bates et al 1992, Blumberg 1995, Perlin 1995 and Goldberg 1997, Hayes-Roth 1996, Elliot et al 1997). Some groups have designed architectures and implemented prototypes that have been demonstrated at academic conferences. But it should be made clear that in the business of creating emotional artifacts, in the final analysis, prototypes and demos are not enough. The point of creating emotionally powerful experiences, whether interactive or not, is to induce a reaction in an audience, in “users.” These creations must be experienced by the general public to serve the purpose for which they were created in the first place. Until this happens the work created in closed-door media labs is ultimately incomplete.
The public has been consuming interactive entertainment for two decades now, in the form of software products from the videogame and computer game industry. Unfortunately the experiences offered in these games are mostly juvenile, primarily focused on fighting, shooting, racing and puzzle-solving, and the virtual characters offered in them are most often shallow, one-dimensional cardboard-cutouts. Such games can be emotionally powerful experiences for those who play them, but they do not appeal to the majority of the population. Very few successful pieces of interactive entertainment or art have been made with emotional content for a mass audience – that is, the kind of “personal relationship” stories that books, theater, television and movies offer, or the kind of “high art” exhibited at museums and art shows (Stern 1999a, Mateas 1999).
This remainder of this paper suggests new ways to employ virtual characters to create emotionally powerful interactive experiences, using our Virtual Petz and Babyz projects as case studies. The techniques used to create these projects will be presented and discussed, with emphasis on the importance of design. Finally we will attempt to address the question of what it could mean for a human to have an emotional relationship with a virtual character.
2 Do you feel it? The case for emotional
relationships
Animation and artificial intelligence technologies for creating real-time interactive virtual characters are currently being researched and developed in academic labs and industry companies. We are told that soon we will have virtual humans that look photorealistic, with behavior driven by some degree of AI. Many are working with the intention that these characters will become functional agents, at our command to perform a variety of complicated and menial tasks. They will learn our likes and dislikes, able to autonomously communicate and negotiate with others. And they will become teachers in the virtual classroom, always ready and willing to answer our questions.
At first glance it seems natural that adding emotions to these virtual characters should greatly enhance them. After all, real people have emotions, so virtual human characters should have them too. Some researchers in neuroscience and psychology point to emotion as an important factor in problem solving capabilities and intelligence in general (Damasio 1994). As Marvin Minsky put it, “the question is not whether intelligent machines can have emotions, but whether machines can be intelligent without any emotions” (Minsky 1985). Virtual characters may very well need emotions to have the intelligence to be useful.
But when thinking in terms of a virtual character actually interacting with a user, is emotional behavior really appropriate for these types of applications? In real life it is arguable that interactions with “functional agents” (e.g., waiters, butlers, secretaries, travel agents, librarians, salespeople) are often best when emotions are not involved. Emotional reactions can often be irrational, illogical and time-consuming, which work against the efficient performance of tasks. Of course in any transaction, politeness and courtesy are always appreciated, but they hardly qualify as emotional. We expect teachers to be a bit more personable and enthusiastic about their material than a travel agent, but do we want them to get angry or depressed at us?
While emotions may be required for intelligence, I would argue that the most compelling interactions with virtual characters will not be in the area of functional agents. If a user encounters a virtual character that seems to be truly alive and have emotions, the user may instead want to befriend the character, not control them. Users and interactive virtual characters have the potential to form emotional relationships with each other – relationships that are more than a reader’s or moviegoer’s affinity for a fictional character in a traditional story, and perhaps as meaningful as a friendship between real people. By an emotional relationship we mean a set of long-term interactions where the two parties pay attention to the emotional state of the other, communicate their feelings, share a trust, feel empathetic, and establish a connection, a bond.
2.1 Virtual friends
The recent success of several “virtual pet” products, popular among both kids and adults, offers some support for this idea. The most sophisticated of these characters are animated on the computer screen, such as Dogz and Catz (PF.Magic 1995 - 1998), and Creatures (Grand et al 1997), but some are displayed on portable LCD keychain toys or even embodied as simple physical robots, such as Tamagotchi (Bandai 1996), Furby (Tiger Electronics 1998), and Aibo (Sony 1999). Users “nurture” and “play” with these pets, feeding them virtual food, petting them with a virtual hand, and generally giving them attention and care lest they runaway or die. While there can be some blurring into the domain of videogames, in their purest form virtual pets are not a game, because they are non-goal-oriented; it is the process of having a relationship with a virtual pet that is enjoyable to the user, with no end goal of winning to aim for.
As of this writing, there have been no completed formal studies of virtual pets; a study is currently underway by (Turkle 1999). In our experience with the Dogz and Catz products, our anecdotal evidence suggests that depending upon the sophistication of the virtual character, the emotional relationship that a user can form with it ranges anywhere from the attachment one has to a favorite plant to the bond between a master and their dog. Children are the most willing to suspend their disbelief and can become very attached to their virtual pets, playing with and feeding them everyday. It is precisely those irrational, illogical and time-consuming emotional interactions that may hamper a functional agent that are so engaging and entertaining here. (Please refer to the Appendix of this paper to read real customer letters we have received about Petz.) Only a few Petz users, mostly technology-oriented adult men, have requested that their Petz be able to perform functional tasks such as fetching e-mail.
What is most interesting about the phenomenon of virtual pets are not the toys and software themselves – some of which have minimal interactivity and little or no artificial intelligence driving them – but the fact that some people seem to want to form emotional relationships with them. Some appear quite eager to forget that these characters are artificial and are ready and willing to engage in emotional relationships, even when some of the virtual pets offer little or no reward or “warmth” in return. This offers some promise for the public’s acceptance of the concept of a more advanced virtual friend.
As commercially successful as these virtual pets are, it seems likely that emotional relationships at the level of favorite plants or pets will be far easier to accomplish than the level of friendship between two adults. An owner-to-pet relationship dynamic is much simpler than a person-to-person one, and much less communication is required between the two parties. Most importantly, the relationship is inherently unequal. An owner of a real pet chooses (even purchases!) their real-life cat or dog. Therefore the act of buying a virtual pet as a toy or piece of software does not violate the hierarchy of a real-world owner-to-pet relationship.
As people we do not get to choose which other people will be friends with us. Friends, by definition, choose to be friends with one another. Therefore, even if we create an interactive virtual character that can perform all the behaviors required for an emotional relationship between human adults, as a manmade artifact that can be bought and sold, could a “true” friendship could be formed at all? This is an open question that invites exploration.
2.2 Interactive stories
Stories have long been our primary way to observe and understand our emotional relationships. If stories could be made interactive – where users could immerse themselves in virtual worlds with characters they could talk to, form relationships with, touch and be touched by, and together alter the course of events, literally creating a new story in real-time – then we would have a new form of interactive entertainment that eclipses videogames. Like traditional stories from books, theater, television and movies, an interactive story would be affecting and meaningful, but made all the more personal because the user helped shape it and create it (Stern 1998).
Virtual characters programmed to simulate the dynamics of
emotional relationships could be used as starting points for creating
interactive stories. In her recent book, Hamlet
on the Holodeck: The Future Of Narrative in Cyberspace,
Janet Murray suggests that interactive virtual characters “may mark the
beginning of a new narrative format” (
2.3 Interactive art
Autonomous interactive virtual characters are only just beginning to make their way into installation and performance art. The goal of Simon Penny’s 1995 “Petit Mal” was, according to the artist, “to produce a robotic artwork which is truly autonomous; which was nimble and had ‘charm’”, to give “the impression of being sentient” (Penny 1997). Petit Mal was a tall, thin robot on bicycle wheels which could quietly and gently move about in a room, able to sense when it was approaching walls or people. Celebrated video artists Lynn Hershman and Bill Viola have begun experimenting with combining video imagery of people and some simple interactivity. Mark Boehlen and Michael Mateas recently exhibited “Office Plant #1”, a robot plant that will bloom, wither and make sounds in response the mood of your office environment (Boehlen and Mateas 1998). Other efforts include the large, destructive autonomous robots from the industrial performance art of Survival Research Labs, the RoboWoggles (Wurst and McCartney 1996), and the robot installations of Alan Rath. The potential for exploration and discovery in this area seems untapped and wide open.
3 I need your love: Virtual dogs,
cats and babies
Recognizing a dearth of consumer software with characters that displayed emotions or personality, the startup company PF.Magic was formed in 1992 with the mission to “bring life to entertainment.” We wanted to break out of the mold of traditional videogames (e.g., flight simulators, sports games, running-jumping-climbing games, shooters, puzzle games) to create playful new interactive experiences with emotional, personality-rich characters.
3.1 Dogz and Catz
By 1995 the personal computer was powerful enough to support the real-time animation we felt was required for a convincing virtual character. The first Dogz program, originally conceived by company co-founder Rob Fulop and created by Adam Frank and Ben Resner, was a simple idea: an animated virtual dog that you could feed and play with. As a product it was very risky; an emotional relationship as the sole basis of an interactive experience had never been done before. It was unknown at the time if anyone would pay money to interact with a virtual character in this way.
The program quickly generated interest from a wide range of customers male and female, kids and adults, which is typically unheard of in entertainment software. We followed up Dogz with a companion product, Catz, establishing the explicit design goal to create the strongest interactive illusion of life we could on a PC. To imbue the Petz characters with personality and emotion we began cherry-picking techniques from computer animation and artificial intelligence, to construct what eventually became a powerful real-time animation engine tightly integrated with a goal-based behavior architecture.
The Virtual Petz characters are socially intelligent autonomous agents with real-time 3-D animation and sound. By using a mouse the user moves a hand-shaped cursor to directly touch, pet, and pick up the characters, as well as use toys and objects. Petz grow up over time on the user’s computer desktop and strive to be the user’s friends and companions. The interaction experience is non-goal oriented; users are allowed to explore the characters and their toys in any order they like within an unstructured yet active play environment. This freedom allows users to socialize with the Petz in their own way and at their own pace. This also encourages users to come up with their own interpretation of their pet's feelings and thoughts. To date the Virtual Petz products have sold over two million copies worldwide.
Figure 1. Virtual Petz
The goal of the Petz characters is to build an emotional relationship with user. Their behaviors are centered around receiving attention and affection. They feed off of this interaction. Without it they become lethargic, depressed, and if ignored for long enough, they will run away.
The most direct way the user can show affection to the Petz is through petting. By holding down the left mouse button users can pet, scratch and stroke with a hand cursor; the Petz immediately react in a variety of ways depending on what spot on their body is being petted, how fast, and how they feel at the time. Users can also pick up the characters with the right mouse button and carry them around the environment. We found that being able to (virtually) touch and hold the characters to be a very effective way of building emotional relationships and creating the illusion of life.
The Petz have equal footing in their relationship with the user. The toys and objects in their environment have direct object-like interaction for both the user and the characters. Petz have full access to the toy shelf, and if they really want something, they have the freedom to get it themselves. This helps express the unpredictability and autonomous nature of the Petz. It also requires users to share control of the environment with them. For example, by picking up and using a toy the user can initiate play. Throwing a ball may initiate a game of fetch, or holding a tugtoy in front of a pet may begin a game of tug-of-war. Similarly, a pet can get its own toy and bring it to the user to initiate play. The act of sharing control of the environment and cooperative decision-making helps further strengthen the relationship.
We have created a variety of personalities - playful terriers,
grumpy bulldogs, hyper
3.2 Babyz
Our newest virtual characters, Babyz, released in October 1999, have the same non-goal-oriented play and direct interaction interface as the Petz. The user adopts one or more cute, playful babies that live in a virtual house on the computer. The Babyz have a similar cartoony style as the original Petz, but have more sophisticated, emotive facial expressions, and some simple natural language capability. Babyz vary in size, shape and personality, and in some ways appear to be smarter and cleverer than real one-year-old babies would be. They want to be fed, clothed, held and nurtured, but are also quite playful and mischievous. Users can think of themselves as their parent or babysitter, which ever they feel most comfortable with.
The user can nurture a Babyz character by holding and rocking it, tickling it, feeding it milk and baby food, putting on a fresh diaper, giving it a bubble bath, soothing it if it gets upset, giving it medicine if it gets sick, laying it down to sleep in a crib, and so on. Play activities include playing with a ball, blocks, baby toys, music, dancing and singing, and dress-up. Through voice recognition Babyz can understand and respond to some basic spoken words (such as “mommy”, “baby”, “yes”, “no”, “stop that”), and can be read simple picture books.
Figure 2. Babyz
The Babyz characters develop over time, appearing to learn how to use toys and objects, learning to walk and to speak a baby talk language. In this version of the product, they will always be babies – never progressing beyond a stumbly walk and simple baby talk. (All behaviors are pre-authored, with the user’s interaction unlocking them over time, to create the illusion that the Babyz are learning.) If the user has more than one baby adopted, they can interact and form relationships with one another. Babyz can be friends and play nicely together or engage in long-term sibling rivalries. The program becomes an especially entertaining and chaotic experience with three active Babyz all getting into mischief at the same time (Stern 1999b).
3.3 Behaviors to support emotional relationships
To allow for the formation of emotional relationships between the user and the Petz and Babyz characters, we built a broad base of interactive behaviors. These behaviors offer the characters and the user the means of communicating emotion (or the appearance of emotion) to each other. This section will detail these interactions and behaviors, specifying in each the emotions we intended to be perceived by the user. Our hope is that by having the characters express emotion in a convincing and life-like way, the user will instinctively feel empathetic; and at the same time, if given the means to express their own emotions in return, users will feel like they are connecting to the characters on an emotional level.
Affection. Users can express affection to the characters by touching them and holding them with their mouse-controlled hand cursor. For Petz the touching is petting; for Babyz it is tickling. Petz express affection to the user by sweetly barking, meowing or purring, licking and nuzzling the hand cursor, and bringing the user a toy. Babyz will smile, giggle and laugh, coo, act cute, and say “mama” or “dada” in a loving voice. When perceiving a lack of affection, Petz will howl and yowl, sounding lonely; Babyz will cry and say “mama” or “dada” in a sad tone of voice. The intent is for the user to perceive the feelings of love, warmth, happiness and loneliness in the characters.
Nurturing. Users
can feed, clothe and give medicine the characters. Petz express the need to be nurtured by
acting excited when food is brought out, begging, acting satisfied and grateful
after eating, or disgusted when they don’t like the food. Babyz may hold out
their arms and ask for food, whine and cry if hungry or need a diaper change,
and may throw and spit up food they don’t like.
Users are meant to perceive feelings of craving, satisfaction, pleasure, gratefulness, dislike and disgust.
Play. By picking up a toy users can initiate play
with one or more of the characters, such as a game of fetch or building blocks.
Petz or Babyz may join the user’s invitation to play, or get a toy of their own
and begin playing by themselves or each other, waiting for the user to join
them. A character may react if the user
is ignoring them and instead playing with another character. Emotions intended
to be perceived by the user include excitement,
boredom, aggressiveness, timidity, laziness, and jealousy.
Training. Users can give positive and negative reinforcement in the form of food treats, water squirts (for Petz), and verbal praise or discipline to teach the characters to do certain behaviors more or less often. Petz or Babyz are programmed to occasionally act naughty, to encourage users to train them. During these behaviors users are meant to perceive the emotions of feeling rewarded, punished, pride, shame, guilt, and anger.
3.4 Effective expression of emotion
None of the aforementioned behaviors would seem believable to the user unless the characters effectively expressed convincing emotions. We found all of the following techniques to be critical for successful real-time emotion expression in virtual characters.
Emotion expression in parallel with action. During any body action (such as walking, sitting, using objects, etc.) Petz and Babyz characters can display any facial expression or emotive body posture, purr or cry, make any vocalization or say any word in any of several emotional tones. This allows a baby character to look sad and say it wants while it crawls towards the user. Catz can lick their chops and narrow their eyes as they stalk a mouse. Characters can immediately sound joyful when the user tickles their toes. We found if a virtual character cannot immediately show an emotional reaction, it will not seem believable. Timing is very important.
Emotion expression at regular intervals. Programming the characters to regularly pause during the execution of a behavior to express their current mood was a very effective technique. For example, while upset and crawling for a toy that it wants, Babyz may stop in place and throw a short tantrum. Or just before running after a ball in a game of fetch, Dogz may leap in the air with joy, barking ecstatically. These serve no functional purpose (in fact they slow down the execution of a behavior), but they contribute enormously to the communication of the emotional state of the character. Additionally, related to Phoebe Sengers’ concept of behavior transitions (Sengers 1998), when Petz or Babyz finish one behavior and are about to begin a new one, they pause for a moment, appearing to “stop and think” about it, look around, and express their current mood with a happy bark or timid cower.
Emotion expression through customization of behavior. Some behaviors have alternate ways to execute, depending on the emotional state of the character. Mood may influence a character’s style of locomotion, such as trotting proudly, galloping madly in fear, or stalking menacingly. A hungry character may choose to beg for food if lazy, cry and whine for food if upset, whimper if afraid, explore and search for food if confident, or attack anything it sees if angry. The greater the number of alternate ways a character has to perform a particular behavior, the stronger and deeper the perceived illusion of life.
Prioritization of emotion expression, and avoidance of thrashing. It is possible for a character to have
multiple competing emotions, such as extreme fear of danger simultaneous with
extreme craving for food. We found it to
be most believable if “fear” has the highest priority of all emotion
expression, followed by “craving” for
food and then extreme “fatigue”. All other emotions such as “happiness” or
“sadness” are secondary to these three extreme emotional states. It is also
important that characters do not flop back and forth between conflicting
emotions, else their behaviors appear incoherent.
Theatrical techniques. Our
characters are programmed to obey several important theatrical techniques, such
as facing outwards as much as possible, looking directly outwards into the eyes
of the user, and carefully positioning themselves relative to each other
(“stage blocking”). If the user places a
character offscreen, behind an object or at an odd angle, the characters
quickly get to a visible position and turn to face the user. If two characters plan to interact with one
another, such as licking each other’s noses or giving each other an object,
they try to do this from a side view, so the user can see as much of the action
and emotion expression as possible in both characters. Similar techniques were identified in the
context of virtual characters in (Goldberg 1997).
3.5 Animation and behavior architecture
Animation and behavior are tightly integrated in the Petz and Babyz architecture. An attempt was made during software development to construct a clean modular code structure; however both time constraints and practical considerations forced us to at times adopt a more ad hoc, hackish approach to implementation.
The lowest level in the architecture is an animation script layer where frames of real-time rendered 3-D animation are sequenced and marked with timing for sound effects and action cues, such as when an object can be grabbed during a grasping motion. Above this is a finite state machine that sequences and plays the animation scripts to perform generic but complicated low-level behaviors such as locomotion, picking up objects, expressing emotions, and so on. The next level up is a goal-and-plan layer that controls the finite state machine, containing high level behaviors such as “eat”, “hide” or “play with object”. Goals are typically spawned as reactions to user interaction, to other events in the environment, or to the character's own internal metabolism. Goals can also be spawned deliberately as a need to regularly express the character's particular personality or current mood. At any decision point, each goal’s filter function is queried to compute how important it is for that goal to execute under the current circumstances. Filter functions are custom code in which the programmer can specify when a goal should execute. Part of the craft of authoring behaviors is balancing the output of these filter functions; it is easy to accidentally code a behavior to happen far too often or too seldom for believability.
Alongside instantiated goals are instantiated emotion code objects such as “happy”, “sad” and “angry”. Emotions have filter functions much like goals, but can also be spawned by the custom logic in states or goals. These emotion code objects themselves can in turn spawn new goals, or set values in the character’s metabolism. For example, the filter function of a pet’s “observe” goal may be activated in reaction to the user petting another pet. The “observe” goal is written to spawn a “jealousy” emotion if the other pet is a rival and not a friend. The “jealousy” emotion may in turn spawn a “wrestle” goal; any fighting that ensues could then spawn additional emotions, which may spawn additional goals, and so on. Goals are constantly monitoring what emotion objects are currently in existence to help decide which plans to choose and how they should be performed; states monitor emotions to determine which animations, facial expressions, and types of sound to use at any given moment.
Note that by no means did we implement a “complete” model of emotion. Instead, we coded only what was needed for these particular characters. For example, the Babyz have no “fear” emotion, because acting scared was not necessary (or considered entertaining) for the baby characters we were making. Fear was necessary however for the Petz personalities such as the scaredy cat. The emotion lists for Petz and Babyz varied slightly; it would have been inefficient to try to have both to use the same exact model.
At the highest level in the architecture is the “free will” and narrative intelligence layer. This is custom logic that can spontaneously (using constrained randomness) spawn new goals and emotions to convey the illusion that the character has intent of its own. This code is also keeping track of what goals have occurred over time, making sure that entertaining behaviors are happening regularly. It keeps track of long-term narratives such as learning to walk, sibling rivalries and mating courtship.
4 Feeling holistic: The importance of
design
In creating an interactive emotional artifact, even the best animation and artificial intelligence technology will be lost and ineffective without a solid design. In this section we discuss the importance of the overall design of an interactive experience to ensure that a virtual character's emotions are effective and powerful to the user.
Concept and context. The type of characters you choose and the context you present them in will have a great impact on how engaging and emotionally powerful the interactive experience is. Judging by the confusing and poorly thought-out concepts in many pieces of interactive entertainment today, we feel this is a design principle too often ignored. In our products were careful to choose characters that people immediately recognize – dogs, cats, and babies – which allow users to come to the experience already knowing what to do. They immediately understand that they need to nurture and play with the characters.
Even though Petz and Babyz are presented in a cartoony style, we were careful to keep their behavior in a careful balance between cartooniness and realism. This was important to maintain believability and the illusion of life; if the Petz stood up on their hind legs and began speaking English to one another, users would not have been able to project their own feelings and experiences with real pets onto the characters. One of our maxims was “if Lassie could do it, our Petz can do it.” That is, the Petz can do a bit more than a real dog or cat would normally do, but nothing that seems physically or mentally impossible.
A very important design principle in Petz and Babyz for supporting emotional relationships is that users play themselves. Users have no embodied avatar that is supposed to represent them to the characters; the hand cursor is meant to be an extension of their real hand. The characters seem to “know” they are in the computer, and they look out at the user as if they actually see them. There is no additional level of abstraction here; you are you, and the characters are the characters. This is akin to a first-person versus third-person perspective. If the user had an avatar that they viewed from a third-person perspective, the other characters would be required to look at that avatar, not at the user directly, thereby weakening the impact of their emotional expression.
Direct, simple user interface. Petz and Babyz are almost completely devoid of the typical user interface trappings of most interactive entertainment products. To interact with the characters, users operate the hand cursor in a “natural”, direct way to touch and pick up characters and objects. No keyboard commands are required. All of the objects in the virtual world are designed to be intuitively easy to use; you can throw a ball, press keys on a toy piano, open cabinets and so on.
Of course this simplicity limits the amount of expressivity offered to the user. We cannot make objects and behaviors that require more complicated operation, such as a holding a baby and a milk bottle at the same time. While we could program some obscure arbitrary keyboard command sequence to accomplish this, we have chosen not to in order to keep the interface as pure and simple as possible. To allow the user more expressivity we would be required to add more intuitive interface channels, such as a dataglove or voice recognition. In fact instead of typing words to your Petz and Babyz (which you would never do in real-life of course), the latest versions of the products allow you to speak some basic words to the characters.
In general we feel that user interface, not animation or artificial intelligence technology, is the largest impediment for creating more advanced virtual characters. With only a mouse and keyboard users are very constrained in their ability to naturally their express emotions to virtual characters. When interacting with characters that speak outloud, users should be able to speak back with their own voice, not with typing. Unfortunately voice recognition is still a bleeding-edge technology. In the future we look forward to new interface devices such as video cameras on computer monitors that will allow for facial and gesture recognition (Picard 1997).
Natural expression. When trying to achieve believability we found it effective for characters to express themselves in a natural way, through action and behavior, rather than through traditional computer interface methods such as sliders, number values, bar graphs or text. In Petz and Babyz the only way the user can understand what the characters seem to be feeling is to interpret their actions and physical cues, in the same way an audience interprets an actor's performance. We do not display bar graphs or text messages describing the characters’ internal variables, biorhythms or emotional state. By forcing a natural interpretation of their behavior, we don't break the illusion of a relationship with something alive.
Favor interactivity and generativity over a high resolution image. In Petz and Babyz we made a tradeoff to allow our characters to be immediately responsive, reactive and able to generate a variety of expressions, at the expense of a higher resolution image. Surprisingly most game developers don’t make this tradeoff! From a product marketing perspective, a beautiful still-frame is typically considered more important than the depth and quality of the interactive experience. Of course there is a minimum level of visual quality any professional project needs, but we feel most developers place far too much emphasis on flashy effects such as lighting, shading and visual detail (i.e., spectacle), and not enough emphasis on interactivity and generativity.
Purity versus “faking it”: take advantage of the Eliza effect. The “Eliza effect” – the tendency for people to treat programs that respond to them as if they had more intelligence than they really do (Weizenbaum 1966) – is one of the most powerful tools available to the creators of virtual characters. As much as it may aggravate the hard-core computer scientists, we should not be afraid to take advantage of this. “Truly alive” versus “the illusion of life” may ultimately be a meaningless distinction to the audience. 99% of users probably won’t care how virtual characters are cognitively modeled – they just want to be engaged by the experience, to be enriched and entertained.
5 Conclusion
This paper has put forth virtual characters as a new form of emotional artifact, and the arrival of emotional relationships between humans and virtual characters as a new social phenomenon and direction for story and art. The design and implementation techniques we found useful to support such emotional relationships in the Virtual Petz and Babyz projects have been presented. We will conclude with some final thoughts on what it could mean for a person to have an emotional relationship with a virtual character.
Are relationships between people and virtual characters somehow wrong, perverse, even dangerous – or just silly? Again we can look to the past to help us answer this. Audiences that read about or see emotional characters in traditional media – painting, sculpture, books, theater, television and movies – have been known to become very “attached” to the characters. Even though the characters are not real, they can feel real to the audience. People will often cry when the characters suffer and feel joy when they triumph. When the written novel first appeared it was considered dangerous by some; today we find that this is not the case. However television, a more seductive medium than the novel, has certainly captured the free time in the lives of many people. Some consider the effect of television and videogames on children’s development to be a serious problem; the media has even reported a few outrageous stories of people going to death-defying lengths to take care of their virtual pet Tamagotchis. Designers should be aware that manmade characters have the potential to have a powerful effect on people.
Why create artificial pets and humans? Isn’t it enough to interact with real animals and people? From our perspective on making Virtual Petz, this was not the point. Our intent was not to replace people’s relationships with real living things, but to create characters in the tradition of stuffed animals and cartoons. And while some people are forming emotional relationships with today’s virtual characters, by and large they are still thought of as sophisticated software toys that try to get you to suspend your disbelief and pretend they are alive. However as we move towards virtual human characters such as Babyz, the stakes get higher. As of this writing we have not yet gotten feedback from the general public on their feelings and concerns about Babyz.
Also, the characters made so far have been “wholesome” ones, such as dogs, cats and babies, but one could easily imagine someone using these techniques to create characters that could support other types of emotional relationships, from the manipulative to the pornographic. Inevitably this will happen. Of course the promise and danger of artificial characters has long been an area of exploration in literature and science fiction, ranging from friendly, sympathetic characters such as Pinocchio and R2D2 to more threatening ones such as Frankenstein and HAL9000.
As virtual characters continue to get more life-like, we hope users keep in mind that someone (human) created these virtual characters. Just as an audience can feel a connection with the writer, director or actor behind a compelling character on the written page or the movie screen, a user could potentially feel an even stronger connection to the designer, animator and programmer of an interactive virtual character. For the artist, the act of creating a virtual character requires a deep understanding of the processes at work in the character’s mind and body. This has always been true in traditional art forms, from painting and sculpting realistic people to novels to photography and cinema, but it is taken to a new level with interactive virtual characters. As the artist you are not just creating an “instantiation” of a character – a particular moment or story in the character’s life – you are creating the algorithms to generate potentially endless moments and stories in that character’s life.
People need emotional artifacts. When the public gets excited about buzzwords like “artificial intelligence” or “artificial life”, what they are really asking for are experiences where they can interact with something that seems alive, that has feelings, that they can connect with. Virtual characters are a promising and powerful new form of emotional artifact that we are only just beginning to discover.
Acknowledgments
Virtual Petz and Babyz were made possible by a passionate team of designers, programmers, animators, artists, producers and testers at PF.Magic / Mindscape that include Adam Frank, Rob Fulop, Ben Resner, John Scull, Andre Burgoyne, Alan Harrington, Peter Kemmer, Jeremy Cantor, Jonathan Shambroom, Brooke Boynton, David Feldman, Richard Lachman, Jared Sorenson, John Rines, Andrew Webster, Jan Sleeper, Mike Filippoff, Neeraj Murarka, Bruce Sherrod, Darren Atherton and many more. Thanks to Robert Trappl and Paola Petta for organizing such a fascinating and informative workshop.
Appendix: Real customer letters
I
had a dog that was a chawawa and his name was Ramboo. Well he got old and was very sick and suffering so
my parents put him to sleep. Ever since then I have begged my parents for a new
dog. I have wanted one soo bad. So I heard about this
dogz on the computer. I bought it and LOVE it!!! I have adopted 9 dogs. Sounds
a bit to much to you ehhh? Well I have alot of free time on my hands. So far everyday I take each
dog out one by one by them selves and play with them, feed them, and brush
them, and spray them with the flee stuff. I love them all. They are all so differnant with differant personalitys. After I take them out indaviually
then I take 2 out at a time and let them play with me with each other. Two of
the dogs my great Dane and chawawa
dont like to play with any of the other dogs but each
other. This is a
incrediable program. I had my parents thinking I was
crazy the other night. I was sitting here playing with my scottie Ren and mutt stimpy and they where playing so well together I dont know why but I said good dog out loud to my computer.
I think my parents wondered a little bit and then asked me what the heck I was
doing. But thankz PF.Magic. Even though I cant
have a real dog it is really nice to have some on my screen to play with. The
only problem now is no one can get me away from this computer, and I think my
on-line friendz are getting a little mad cause im not chatting just playing fetch and have a great time
with my new dogz. Thanks again PF. magic. I love this program and will recomend it to everyone I know!!!!!!!
I
am a teacher and use the catz program on my classroom PC to teach children both
computer skills and caring for an animal.
One of the more disturbed children in my class repeatedly squirted the
catz and she ran away. Now the other
children are angry at this child. I
promised to try and get the catz back.
It has been a wonderful lesson for the children. (And no live animal was involved.) But if there is any way to get poor Lucky to come homze to our clazz, we would very
much appreciate knowing how to do it. Thanks for your help, Ms. Shinnick's 4th grade,
Dear
PF. Magic, I am
an incredible fan of your latest release,Petz 3,I
have both programs and in Janurary 1999,my cherised Dogz Tupaw was
born. He is the most wonderful dogz and
I thank you from the bottom of my heart, because in Janurary
through to the end of April I had Anorexia and i was
very sick. I ate and recoverd because i cared so much about Tupaw and i wanted to see him grow up. I would have starved without
you bringing Petz 3 out. Please Reply to this,it would mean alot to
me. Oh,and please visit my webpage,the
url is http://www.homestead.com/wtk/pets.html. Thankyou for releasing
petz 3,Give your boss my best wishes, Sincerily, Your Number One Fan, Faynine
I
just reciently aquired all
your Petz programs and I think they are great! I really love the way the
animals react. I raised show dogs and
have had numerous pets of all kinds in my life and making something like this
is great. I am a school bus driver and have introduced unfortunate kids to your
program. Children who not only can they not afford a computer but they can't
afford to keep a pet either. This has taught them a tremendous amount of responsibilty. I am trying to get the school to incorporate
your programs so as to give all children a chance to see what it is like to
take care of a pet. It might help to put a little more compassion in the world.
Please keep me updated on your newest releases. Thanks for being such a great
company. Nancy M. Gingrich
Dear
PF.Magic, Hello! My name is Caitlin, and I'm 10 years old. I have Dogz 1 and
Catz 1, as well as Oddballz, and I enjoy them all very much. Just this morning
was I playing with my Jester breed Catz, Lilly. But I know how much better Petz
II is. For a while, I thought I had a solution to my Petz II problem. I thought
that if only I could get Soft Windows 95 for $200, that
would work. Well, I took $100 out of my bank account (by the way, that's about
half my bank account) and made the rest. I cat-sit, I sold my bike, and I got
some money from my parents. Anyway, I really, really love animals (I'm a member
of the ASPCA, Dog Lovers of America, and Cat Lovers of America) but I can't
have one! That's why I love Petz so much! It's like having a dog or cat (or
alien for that matter) only not. It's wonderful! I have a Scrappy named Scrappy
(Dogz), Chip named Chip (Dogz), Bootz named Boots
(Dogz), Cocker Spaniel named Oreo (Dogz), Jester named Lilly (Catz), and Jester
named Callie (Catz). And then every single Oddballz breed made. =) I don't mean
to bore you as I'm sure this letter is getting very boring. I would love SO
MUCH to have Petz II. I really would. (At this point in the letter I'm really
crying) I adopted 5 Catz II catz at my friend's house, but I go over to her
house so little I'm sure they'll run away. I'd hate for them to run away. Is
there anything I can do? I love my petz, and I'm sure they'd love Petz II.
Thank you for reading this. Please reply soon. ~*~ Caitlin and her many petz
~*~
My
husband went downtown (to
Dear
Sirs, Just wanted to thank-you for the pleasure my
petz have brought me. I am paralyzed from the neck down yet your program has
allowed me too again enjoy the pleasure of raising my own dogz. I have adopted
5 so far. I love them equally as if they were real. Thanks again
References
Bandai. 1996. Tamagotchi keychain
toy. Website URL: http://www.bandai.com
Bates, J. 1992. The Nature of Characters in
Interactive Worlds and The Oz Project. Tech Report
CMU-CS-92-200,
Bates, J. 1994. The Role of
Emotion in Believable Agents.
Technical Report CMU-CS-94-136,
Blumberg, B. 1997. Multi-level
Control for Animated Autonomous Agents: Do the Right Thing… Oh, Not That… In Creating Personalities for Synthetic Actors, R. Trappl and P. Petta (Eds.),
Boehlen, M. and Mateas, M. 1998.
Office Plant #1: Intimate space and contemplative entertainment. Leonardo,
Volume 31 Number 5: 345-348.
Damasio, A. 1994. Descartes'
Error: Emotion, Reason and the Human Brain.
Putnam Publishing Group,
Dautenhahn,
K. 1997. Ants Don’t Have Friends – Thoughts
on Socially Intelligent Agents. In
Proceedings of the 1997 AAAI Fall Symposium, Socially Intelligent Agents,
Elliot, C.; Brzezinski, J.; Sheth, S.; and Salvatoriello, R. 1997.
Story-morphing in the Affective Reasoning paradigm: Generating stories
automatically for use with ``emotionally intelligent'' multimedia agents. Institute for Applied Artificial
Intelligence,
Egri,
L. 1946.
The Art of Dramatic Writing.
Frank, A.; Stern, A.; and Resner, B. 1997.
Socially Intelligent Virtual Petz.
In Proceedings of the 1997 AAAI Fall Symposium,
Socially Intelligent Agents, FS-97-02, pp. 43-45.
Frank, A.; and Stern, A. 1998. Multiple Character Interaction Between Believable Characters. In Proceedings of the 1998
Computer Game Developers Conference, pp. 215-224. Miller Freeman,
Goldberg, A. 1997. IMPROV: A System for Real-Time Animation of
Behavior-Based Interactive Synthetic Actors.
In Creating Personalities for Synthetic Actors, R. Trappl and P. Petta (Eds.),
Grand, S.; Cliff, D.; and A. Malhotra. 1997. Creatures: Artificial
Life Autonomous Software Agents for Home Entertainment. In Proceedings of the First
Intl. Conference on Autonomous Agents,
Hayes-Roth,
B.; van
Kline,
C; and Blumberg, B. 1999. The Art and Science of Synthetic Character
Design. 1999 Convention of the Society for the Study of
Artificial Intelligence and the Simulation of Behavior (AISB),
Loyall, A. 1997. Believable Agents: Building Interactive
Personalities, Ph.D. Dissertation,
Mateas,
M. 1999.
Not Your Grandmother's Game: AI-Based Art and Entertainment. In Proceedings of the 1999 AAAI Spring
Symposium, Artificial Intelligence and Computer Games, SS-99-02, pp.
64-68.
Meehan,
J. 1976. The metanovel : writing stories
by computer. Ph.D.
Dissertation, Dept. of Computer Science,
Minsky, M. 1985. The Society
of Mind.
Murray,
J. 1997. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. The
Free Press,
Pemberton,
L. 1989. A Modular
Approach to Story Generation. In 4th Conference of the European Chapter of the Association for
Computational Linguistics, pp. 217-224.
Penny,
S. 1997. Embodied Cultural Agents: At
the Intersection of Robotics, Cognitive Science and Interactive Art. In Proceedings of the 1997 AAAI Fall Symposium, Socially
Intelligent Agents, FS-97-02, pp. 43-45.
Perlin,
K. 1995.
Real-time responsive animation with personality. IEEE Transactions on
Visualization and Computer Graphics.
PF Magic / Mindscape. 1995 - 1999. Dogz and Catz: your Virtual Petz, Babyz. Website URLs: http://www.petz.com and
http://www.babyz.net
Picard,
R. 1997. Affective
Computing. MIT
Press.
Reilly, W. 1996. Believable Social and Emotional Agents. Ph.D. Thesis. Technical Report CMU-CS-96-138,
Rilke,
R. 1934.
Letters to a Young Poet. W. W.
Norton and Company,
Sengers,
P. 1996. Socially Situated AI: What It Means and Why
it Matters. In
Proceedings of the 1996 AAAI Symposium, Entertainment and AI/A-Life.
Sengers,
P. 1998.
Do the Thing Right: An Architecture for
Action-Expression. In
Proceedings of the Second Intl. Conference on Autonomous Agents, pp. 24-31. ACM Press,
Sony Electronics. 1999. Aibo
robot dog. Website URL:
http://www.world.sony.com/robot/get/meet.html
Stern, A.; Frank, A.; and Resner, B. 1998.
Virtual Petz: A Hybrid Approach to Creating Autonomous, Lifelike Dogz
and Catz. In Proceedings of the Second Intl. Conference on
Autonomous Agents, pp. 334-5. ACM
Press,
Stern, A. 1998. Interactive Fiction: The Story is Just
Beginning. IEEE Intelligent Systems,
Volume 13, No. 5, pp. 16-18. Los
Alamitos: IEEE Computer Society.
Stern, A. 1999a. AI Beyond Computer Games. In Proceedings of the 1999 AAAI Spring
Symposium, Artificial Intelligence and Computer Games, SS-99-02, pp.
77-80.
Stern, A. 1999b. Virtual Babyz: Believable Agents with
Narrative Intelligence. In Proceedings of the 1999 AAAI Fall Symposium, Narrative
Intelligence.
Thomas,
F; and Johnston, O. 1981. Disney
Animation: The Illusion of Life.
Tiger Electronics. 1998. Furby
toy. Website URL: http://www.furby.com
Turner,
S. 1994. The Creative Process: A
Computer Model of Storytelling and Creativity.
Turkle,
S. 1999. Ongoing
virtual pet study. Website URL:
http://web.mit.edu/sturkle/www/vpet.html
Weizenbaum,
J. 1966. Eliza. Communications of the
ACM, 9:36-45.
Wurst, K. and McCartney, R. 1996.
Autonomous Robots as Performing Agents. In Proceedings of the 1996 AAAI Symposium, Entertainment and
AI/A-Life.