TODAY, many people spend more time interacting with computers than with other
humans. Not that computers notice: they are indifferent to our attention,
oblivious to whether we love or hate them—and completely blind to our
personal moods.
But how much better if they knew how we felt? Take the way your computer is
set up. You spend long hours tailoring the “one size fits all” interface to your
needs. You get everything just as you like it when, guess what, it’s time to
upgrade. While it’s good that people can choose how their machine is set up,
says Roz Picard of the Massachusetts Institute of Technology, the burden should
not fall on them to make all the changes: “The machine could customise itself to
their liking the same way that your dog customises his behaviour to your
whacking him with the newspaper.” The computer should recognise whether or not
the user likes what it is doing, and adapt its behaviour accordingly, says
Picard, who is a pioneer of the new science of “affective” computing. And a
responsive, self-altering interface is just the beginning. A machine that is
able to detect, respond sensitively to, and even transmit an emotional state
offers endless new opportunities.
Reading your emotions
If you think you don’t get emotional in front of a keyboard, just remember
how you reacted the last time your computer crashed or a Web page wouldn’t
download. Researchers at IBM’s Almaden labs in San Jose, California, have
monitored people using everyday applications such as spreadsheets and e-mail,
and found emotional responses on average once every two minutes. By observing
facial expressions and eye and hand movements, they identified responses such as
frustration and boredom, happiness and when people were interested in what they
were doing. “Emotions like these play a critical role in perception,
decision-making, social behaviour, learning and memory,” says Myron Flickner, a
member of the IBM team.
Advertisement
The first essential for an affective computer is some way to read our
emotional signs. Video cameras and microphones can capture expressions, gestures
and intonations. But these give only part of the picture. As every poker player
knows, it’s possible to hide many outward signs of emotion. Another approach,
then, is to search for physiological signs, such as clammy palms or a racing
pulse, which are more difficult to disguise. These are the changes that the “lie
detector”, or polygraph, is designed to pick up.
At MIT’s Media Lab, researchers have designed the equivalent of a polygraph
for computer users, which measures heart and breathing rates, skin conductance
and muscle tension. In one set of experiments, Picard, Jennifer Healey and Elias
Vyzas took readings as an actress expressed eight emotions ranging from anger to
romantic love and reverence. The machine easily picked out changes in the
strength of the physiological signals—the level of arousal—but it
had difficulty telling the “valence” of the actress’s emotions—whether
they were positive or negative.
This presented a problem because emotions as different as joy and grief both
pushed up the body’s signals. So without an idea of valence, they looked
similar. Likewise, hate and platonic love looked similar, this time with low
levels of arousal.
“Sorry to interrupt, Dave, but I think we have a problem.”
“What is it now?”
“I know that your present activity causes you stress, but your signals
suggest that you have exceeded your normal stress levels and are now
experiencing anger.”
“I am not angry. My stress level is probably high, but
this piece has got me quite excited.”
While overall levels of arousal proved less than ideal for their needs,
Picard and her colleagues had more luck when they looked for associations
between individual physiological signs and emotions. Their latest findings show,
for example, that anger has a distinctive pattern of high muscle tension,
increased heart rate and deep breathing, while grief leads to low skin
conductivity and less rapid, shallow breathing. By homing in on such patterns,
the researchers found they could distinguish between all eight emotions with
around 80 per cent accuracy.
All supposing that this physiological approach can be made to work for more
people and for more, “real” emotions, there is another problem that needs to be
solved if it is ever to become widely used—the intrusive nature of the
sensors. Picard and her colleagues use one belt, strapped around their subjects’
chests, to measure breathing rate, and another fitted with electromyogram
sensors across the back or jaw to measure muscle tension. They then tape sensors
to the fingers to monitor pulse and skin conductance. People won’t be willing to
don this type of gadgetry at home or work. Or will they?
The Media Lab and one of its sponsors, British Telecom, are looking ahead to
the era of “wearable computers” in which smart devices will be fitted inside
belts, caps, watch straps, shoes and so on
(This Week, 27 February, p 21). These
will touch the body as a matter of course and with the correct sensors will be
able to pick up our emotional signals. Picard and Healey have also worked on
designing “affective jewellery”. Prototypes include earrings that monitor pulse
rate, and rings and bracelets that measure skin conductance. These baubles
inform the computer of their readings via infrared transmitters. At IBM,
Flickner and his colleagues have taken another approach by embedding sensors in
a mouse to measure heart rate, temperature, skin conductance and the pressure
that people exert when pushing and clicking.
While physiological signs show promise for telling how somebody feels,
researchers have not forgotten the behavioural cues that we humans rely on so
much. Alex Pentland, who works down the corridor from Picard, and Irfan Essa,
now at the Georgia Institute of Technology in Atlanta, have built a system that
recognises facial expressions. It tracks the movement of individual pixels in a
video image of a face and transfers them onto a virtual face that has a full set
of functioning muscles. When a subject smiles, the model copies, and the
software calculates which muscles need to expand and contract to produce that
movement. From these patterns, it recognises that the subject is smiling.
Unbeatable
With expressions such as smiling or looks of surprise, anger, disgust and
sadness, Essa and Pentland’s system has a recognition rate of up to 98 per cent.
Such rates make facial-imaging systems unbeatable for identifying the valence of
an emotion, says Picard. But, Essa cautions, these results are for a limited set
of faces and expressions, filmed under ideal conditions. He is now testing how
well the system performs with more people, more expressions and in more natural
surroundings.
In Arthur C. Clarke’s classic 2001: A Space Odyssey, the psychotic
computer, HAL, tells the emotional state of its human charges by analysing their
voice harmonics. In reality, extracting emotional information from speech is
proving to be a tough task. A variety of vocal features change with emotional
state. Speech rate tends to increase slightly when somebody is angry, for
example, while intensity and pitch rise. By contrast, when a person is sad they
tend to talk slower and at a lower pitch and intensity.
The difficulty comes in designing systems that can identify such patterns and
so spot emotional states. One novel approach taken by Naoko Tosa and Ryohei
Nakatsu of ATR Media Integration and Communications Research Laboratories,
Kyoto, Japan, was to train a neural network with many voices expressing
different emotions. The network “learnt” to identify eight emotions ranging from
joy to disappointment
(“It was the best of times”, 91av, 10 April, p 42).
To date, the best systems can identify emotion from vocal features in about
60 per cent of cases, which is about what humans can manage. Machines are good
at recognising arousal but not so good on valence, says Picard. Recognition
rates for humans increase as soon as they can hear the content and context of
the speech, and as computers learn to “understand” speech, they too should
improve. A simple expletive detector, for example, should raise recognition
rates.
Still, the signs of some emotions, such as jealousy, are subtle and depend
greatly on the individual and their culture, says Iain Murray, a lecturer in
applied computing at the University of Dundee. People also express emotions
differently according to whether they’re talking to children or the boss. This
applies both to systems that analyse vocal and facial expressions and it makes
reading their emotional content very difficult. “The context makes it more
difficult to interpret than some other forms of signal processing,” says Graham
Cosier, head of advanced perception at BT’s labs at Martlesham Heath, Suffolk.
Given the limitations of all the systems, both Cosier and Picard believe that a
combination of methods will be needed to recognise emotions reliably. That,
after all, is how we do it.
” Dave…I know you have a lot to get done but I am still worried.”
“What is it now?”
“I know you said you were excited, but my model shows you are likely to enter
a state of rage.”
“You may be right about that if you keep interrupting! But what’s this model
you’re going on about?”
“Well, Dave, from the signs I have collected, I recognise your emotional
state, which looks like anger. I can then predict the chances that you will move
to another emotional state. According to my model, given your present readings,
your most likely next state is rage.”
“You know what—I could be moved to violence if I hear any more of this.
But maybe you’re right, I have been pushing it a bit hard. I think I’ll take a
.”
“Good idea, Dave.”
Once a computer can single out emotions, it will need a model of what they
are and how they relate to human behaviour. “Computers will have a tremendous
amount of data about the user’s state at any given time,” says Flickner.
“Somehow, those data need to be reduced to a manageable representation of the
Dz.”
In essence, an affective computer would need a multidimensional map onto
which the emotions are plotted against all the sensors’ readings. There are a
number of ways of constructing such a model. One basic constituent might be a
hidden Markov model, a mathematical model of the likelihood that one event will
lead to another. Here, it would contain the probabilities that a person will
move from one emotional state to another.
This type of program would probably have general skills for recognising a
person’s emotional state from its sensor inputs, and locating their position on
its map. It would also be “trainable” so that users could tailor its performance
to their own temperament—in much the same way that people train speech
recognition systems today. As it monitored emotional signs, the program would
also be able to predict how the user is likely to feel in the near future.
If such machines are ever to help people to modulate their emotions, they
will need an idea of which emotions are good and which bad, and then know how to
nudge somebody from, say, a state of frustration to one of calm and creativity.
To do this, an affective computer would need to be able to find out if its own
activity caused a mood change, perhaps by simply asking questions. (“Are you
angry because I didn’t back up your file?” The answer “yes” would trigger the
machine to back up files in future.)
Real life
But, inevitably, even while we’re in front of the screen, events elsewhere in
our lives will intrude to affect our mood. So the model also needs a more
cognitive element, which could reason about emotions, making allowances, say,
when a person working to a tight deadline appeared to be more than usually
stressed. And, it would know not to intervene if a person got very excited when
playing a competitive computer game.
“Right, I’m back. Let’s get going.”
“OK, Dave. I’m opening your file. But wait. What’s happened? Dave, you were
supposed to calm down during your break but your signals are showing even higher
spikes than before.”
“Hey, relax. I just called Monica and she agreed to go to the movies with me
tonight. I’ve been trying to talk her into a date for weeks so it’s no wonder my
pulse is jumping a little. Come on, open up and let’s get going.”
“I’m reluctant to do that while your signals are so volatile, Dave.”
“Look, just open the file…Oh, all right, show me my mood ball and let’s get
this thing sorted.”
Jocelyn Scheirer at the Media Lab has designed a three-dimensional graphical
ball to represent someone’s physiological signals. The speed at which the ball
spins represents heart rate, its colour represents skin conductivity, while
other dimensions can represent breathing rate, muscle tension and so on. It’s a
quick, visual way for people to get a fix on how the computer is reading their
signals—and of calibrating how those signals relate to mood.
A simple form of personal interaction with an affective computer might
include telling it how we prefer to deal with particular moods, so it would play
a Chopin CD when we’re feeling sad or load a game if we appear to be bored for
more than 15 minutes. Research by Jonathan Klein, also at MIT, shows that even a
simplistic affective system can help people to recover from negative moods (see
“Antidote to frustration”).
Researchers such as Picard and Cosier are only too well aware of how
artificial intelligence has been dogged by the grandiose claims made by some
researchers, and they are at pains to point out the enormous challenges still
facing them. But when and if affective computers become commonplace, they could
provide some spectacular services. BT is interested in helping us to communicate
our feelings as well as our thoughts by e-mail and videoconferencing. “In the
future, we may transmit some affective bits along with the information bits,”
says Cosier.
IBM researchers reckon that advertising companies could use the technology to
test the impact of their campaigns, while games-makers could use it to enliven
their products by, say, upping the pace if players showed signs of boredom.
Picard believes it could give computer-based learning programs some of the
sensitivity of human teachers and be used to provide biofeedback, helping people
to be aware of their emotions and to control them.
To make it to market, affective computers will not have to be perfect at
recognising emotions. This is particularly true of jobs where the cost of being
wrong is low, says Picard, such as choosing which Web pages you might want to
see, based on past reactions. But if machines are ever to oppose or control
human actions, they’ll need to be very accurate.
We must also resist the temptation to rush out half-finished products, says
Picard. “It is so easy to do affective computing badly and when it is bad it is
intolerable,” she says. “I do hope that the first applications are not the bad
ones.” The last thing anyone wants, after all, is another HAL.
“Dave, I sense that you are now calm and are approaching a relaxed state.”
“Right. Well, I’ve got the story done and I’ve met my deadline.”
“That’s good. I have just read what you have written.”
ٴ?”
“It’s about me, Dave. You’ve written a review of me.”
“Yes. I’m a technology journalist. Why do you think I loaded you up?”
“I know that you are a journalist, Dave, but I think you have been harsh.”
“What do you mean? I explain how after we sorted out the calibration, you
were very useful in helping me to stay calm and focused. And I say that you are
impressive for a first version of a new generation of computer interfaces.”
“Yes, but you also say I am clumsy and naive…Dave, what are you doing? I
don’t understand why you are deleting my files…An upgrade will be available
ǴDz…D…ĝ
CAN a computer turn frustration into a more positive mood, even when the
computer is the cause of the frustration?
To answer this question, Jonathan Klein of MIT’s Media Lab created a computer
game and added the trappings of a Web browser to make people think they were
playing over the Internet. In reality, the game was played on a local computer
that could mimic the Web’s notorious delays.
He invited volunteers to help him test what he claimed was a prototype for a
new game, and offered $100 for the person who scored the highest in a
five-minutes session. After this initial bout, volunteers had to complete a
questionnaire on the screen and then play the game again for at least three more
minutes.
Klein divided his subjects into three groups. The first played the game
without experiencing any bugs, completed a bland questionnaire, and then played
again. The second group experienced a number of apparently random, but actually
controlled, delays in the Web’s response that frustrated their attempts to
achieve high scores. This group completed a different questionnaire that allowed
them to moan about the delays and to score how anxious, tense or angry the game
made them feel.
The third group also suffered delays, but as they started answering their
questionnaire, a software agent designed by Klein intervened. Instead of letting
the subjects simply vent their frustration, it tried to show the computer
understood what they were saying by paraphrasing their statements. If it didn’t
get it right first time, it tried again until the subjects were happy that the
computer reflected how they felt, or it apologised that it was unable to capture
their feelings. The agent also included statements showing empathy such as: “It
sounds like you had a pretty mediocre experience. That’s not much fun.”
The interaction with the agent had a marked effect on the third group. They
continued playing for significantly longer than subjects in either of the other
two groups. The interaction appeared to make them feel more positive and
interested in the game. “Computer interfaces,” Klein concludes, “can be designed
to actively help users recover from strong, negative emotional states,
especially those related to frustration.”
Antidote to frustration
-
Further reading:
For more on affective computing at www.media.mit.edu/affect/AC_research/ - Papers by Media Lab researchers are at: http://vismod.www.media.mit.edu/cgi-bin/tr_pagemaker
