91av

Computer images that ‘merge’ two or more faces have recently become
a staple of satirists, offering distorted portraits of the famous that alternately
amuse and horrify us. But the technique used to create these composite images
could have applications far beyond that of entertainment, however thought-provoking:
soon forensic science, telecommunications and even medicine could well be
using the method to refine their approaches.

The basic technique is not new – nor is the idea that it can be used
for practical ends. In 1878 Francis Galton, an English explorer and scientist
and a cousin of Charles Darwin, devised a photographic technique called
composite photography, in which he superimposed images of two or more faces
by means of multiple exposures. A similar effect can be achieved with a
stereoscope and two photographs; the viewer sees a different face with each
eye, but perceives one composite face.

For the technique to work, the facial images used must be of the same
size and pose, and must have features of similar dimensions and be carefully
aligned so that the pupils of the eyes coincide. The composite, or average,
then resembles a real human face. But surprisingly, the composite looks
like neither of the faces used to create it.

Galton decided the composite technique might have a practical application
if used to define the facial characteristics of particular social classes
of people – a reflection of the Victorian mania for defining types and categories.
He said there were characteristics common to individuals of one group, such
as criminals, and these characteristics would be maintained in a composite
of faces from that group. The process, he thought, would ‘average out’ any
idiosyncratic variations in the faces of individuals and expose their latent
criminality, for instance.

In effect, Galton was creating archetypes. He superimposed photographs
of the faces of army personnel, for a definitive portrait of health; of
tuberculosis victims, for disease; and of convicted felons, for criminality.
But in trying to reveal the definitive physiognomies of various human ‘types’,
he displayed his own Victorian upper-class prejudices. Galton felt that
his sample of officers and privates from the select Royal Engineers, for
example, all possessed superior ‘bodily and mental qualifications’. Times
change, however, and the portrait he produced now appears rather stiff and
supercilious.

Galton enlisted the help of Edmund Du Cane, Her Majesty’s Director of
Prisons, in his attempt to define the facial attributes of people with criminal
tendencies. He superimposed photographs of people convicted of murder, manslaughter,
or violent robbery. Given his social prejudice, he was surprised to find
that the composite face tended to look more respectable than the individual
ones used to make it. He wrote: ‘The special villainous irregularities .
. . have disappeared, and the common humanity that underlies them has prevailed.’
But Galton persisted in thinking that the composite portrayed an individual
of ‘low type . . . who is likely to fall into crime’. John Stoddard, an
American contemporary of Galton’s who made composite portraits of college
students, backed up the observation that the composite or average face was
better-looking and freer from irregularities than the original faces.

Galton’s technique also had its limitations. In his composite photographs,
very few of the facial features appear sharply focused with well-defined
edges. The eyes are most clearly defined because in the original alignment
of the photographic images the distance between eyes was standardised across
the sample of faces by either blowing up or reducing the originals. The
noses in a sample of faces will vary fairly widely in length, width and
overall shape, so the edges of the nose in a composite face will be blurred
in accordance with these variations. The most extensive blurring appears
around the jaw and hairlines, where individual differences are greater.

Recent developments in computer graphics have made it possible to improve
Galton’s composite technique. In 1990, Judith Langlois and Lori Roggman
of the psychology department at the University of Texas began using graphics
to fine-tune the method. They assembled a sample of faces and manipulated
the dimensions of each, creating the composite image in two stages. Each
facial image was first adjusted in overall size and orientation so that
the pupils of the eyes were aligned. The images were then distorted vertically
so that the distance from a point between the eyes to the centre of the
top lip was equal across the faces.

Langlois and Roggman then generated the composite face by averaging
the intensity values of corresponding pixels, or picture elements, from
the stretched images. In computer graphics, images are displayed and stored
as a regular array of pixels – typically 512 rows and columns. Each pixel
has an associated intensity value and is equivalent to an individual grain
of film emulsion in a photographic image.

The results Langlois and Roggman achieved were better than those of
Galton. But the facial features of the composite were still blurred. Even
when the separation of the eyes and mouth is standardised, the shape and
position of other features inevitably vary. So, if the jaw area in a sample
of faces varies considerably in size and shape, the composite will appear
blurred around the jawline.

Even greater problems of focus arise with the hairline. Current hairstyles
are so diverse that a simple composite image tends to show an unnatural
‘ghosting’ around the hairline. A similar effect can be seen in Galton’s
portrait of the average criminal, whose unshaven appearance probably results
from blending bearded with clean-shaven faces. Facial stubble carried negative
connotations at the time, and so may have made the face seem more sinister
to Victorian eyes.

To avoid blurring, we used computer graphics to manipulate individual
images more extensively before we combined them to form a composite. The
shape of each face was defined as a series of about 200 coordinates, or
reference points, marking the outline of each facial feature. We then defined
the shape of the prototype face for the sample by calculating the average
position of each of the feature ‘markers’ across all of the original faces.
Finally, the original facial images were distorted so that they took on
the shape of the prototype or sample average.

Imagine a thin sheet of patterned rubber pinned out on a flat surface.
If you drag the pins to new, predetermined positions, the pattern on the
rubber will become systematically distorted. We modified the shape of individual
facial images in a similar way, by distorting the shape and position of
each facial feature to match the proportions of the corresponding feature
in the prototype face. The faces could then be averaged to produce a clearly
focused facial composite.

Merging digitally stored images is not a new idea; it was first applied
to videos. It is easy to blend one image frame with another by averaging
the intensity values at corresponding pixels in the two images. Computers
are ideal for this kind of work. The familiar ‘slow dissolves’ of television,
for example, where one scene fades out as another emerges, are produced
by video computer.

The same technique can also be applied to two images of faces. But simply
dissolving one facial image into another will result, again, in a composite
with blurred features. Where the two faces are very poorly aligned, the
composite will resemble a photographic double exposure, with two mouths,
two sets of eyebrows, and so on. This is something that Michael Jackson,
the American pop singer, could afford to avoid in the video accompanying
his latest single ‘Black or White’.

Our method of obtaining facial prototypes allows us to gradually transform
one face into another. To transform Margaret Thatcher into John Major for
instance, we first modified the shapes of the original pair of faces, changing
the configuration of one face in gradual steps until it took on the configuration
of the second, and vice versa. We then blended pairs of images of the two
faces, at stages in their transformations when their configurations matched,
with the composites first dominated by Thatcher’s face and then by Major’s.

The technique’s effect is startling. At the half-way point in the transformation
there is just as much of one person in the image as there is of the other.
When viewers examine this half-way image closely, they experience a sort
of perceptual flip-flop: at one moment it may look more like Thatcher, at
the next, more like Major.

Composite facial prototypes that are generated by computer are useful
in investigating social and memory-related aspects of face recognition:
for example, the differences between male and female faces. By Galton’s
reasoning, gender archetypes could be compared to reveal the consistent
differences between the sexes. If there are consistent differences in the
shapes of the average male and female faces (and if these differences are
important in controlling perception of gender) exaggerating these differences
should enhance the apparent masculinity or femininity.

We used a computer to blend together the faces of 16 female students
to create an average female student. Then we repeated the process with 16
male faces. Averaging all 32 faces produced a prototypical student with
a face that should have been androgynous.

To create the configuration of a ‘hyper-male’ face, we then doubled
the differences between each feature point in the average male and female
faces. If the tip of the male prototype’s nose was 10 units longer than
that of the female prototype, we exaggerated the nose length, making the
tip 20 units longer. To create the configuration of a ‘hyper-female’ face,
we took the female prototype and shortened the length of the nose so that
it was 20 units shorter than that of the average male nose. We repeated
this process for each feature – eyes, mouth, jaw and so on. The pixel images
of the average male and female faces were then distorted into their two
hyper-sex configurations. To determine the role of the facial features in
the perception of gender, we placed a mask around all the faces to remove
cues from the hair and hairline.

Unsurprisingly, the average male was rated more masculine-looking, and
the average female more feminine-looking, than the androgyne. The androgynous
face was perceived as neither masculine nor feminine. Furthermore, exaggeration
of gender differences works: the ‘hyper-male’ was rated more masculine than
the prototype male.

The experiment showed that, on average, the shapes of facial features
alone provide enough information for an accurate judgement of gender. For
the students studied, there are basic differences in facial structure between
the sexes. Male noses tend to be longer than those of females, and their
nostrils more protuberant; other differences are a larger overall facial
area, thicker eyebrows and squarer jaws. The shape of the area around the
cheekbones also varies in men and women, although the differences are subtler.

These differences between the sexes are average tendencies. We may be
unaware of some of them because of variations between individuals of the
same sex. Nevertheless, we probably perceive general tendencies in facial
appearance even when we cannot give an explicit verbal description of all
the differences between males and females.

Gender may be imprinted in the very structure of our faces; but what
makes a face attractive? This question has given rise to a number of socio-psychological
issues: the influence of ethnic background, sexual stereotype and historical
period on the perception of attractiveness. Our recent work with Sakiko
Yoshikawe of the psychology department at the Otemon-Gakuin University,
Osaka, shows that the effect of averaging on attractiveness is consistent
across Oriental and Caucasoid faces and for both Oriental and Caucasoid
viewers.

Since the last century, investigators have found that average composite
faces are more attractive than the faces from the original sample. Galton
noted that composites were ‘better-looking than their components’. Two factors
may contribute to this. The first is the smoothness of skin tones. The blending
process removes freckles and any skin irregularities present in the original
faces, creating an even, almost bland composite visage. The effect is similar
to that of soft-focus photography. The second is the shape of the composite
face and of its features. An average face, by definition, has an overall
average shape. Several authors have speculated that beauty is defined by
the extent to which facial shape approaches average proportions. In the
words of Langlois and Roggman, ‘attractive faces are only average’.

But such notable exceptions as Sophia Loren and Gerard Depardieu seem
to counter this claim: with anything but ‘average’ faces, they are nevertheless
considered very attractive. Perhaps in such cases personality rather than
physiognomy is the factor that signals attractiveness. Alternatively, there
could be a purely aesthetic reason involving some sort of balancing effect
of apparently disharmonious features. Unfortunately we readily confuse things
like a person’s ‘distinctiveness’ with our ‘familiarity’ of that person.
Familiarity itself is confused with ‘fame’. For these reasons, rating images
for their attractiveness is very difficult; social psychology suffers greatly
in this respect. Individuals who are aware of a distinguishing characteristic
counter it by dress, accoutrements, cosmetics or careful manners, for instance.
But again, this unusualness may simply be manipulated and confused with
other social and personal attributes.

IN SEARCH OF ATTRACTIVENESS

The techniques to produce composites to date have been limited, in that
efforts to create an average face shape have resulted in composites with
soft, blurred skin textures. We decided to assess the potential contributions
of face shape and skin texture on perceived attractiveness. With samples
of 16 male and 16 female faces, we first made composite male and composite
female faces. As before, both male and female subjects rated these composites,
with their blended skin textures and average shape, more attractive than
any individual face in the entire original set.

To determine the effect of skin texture alone, we worked backwards from
the male and female composites and reconstructed the general appearance
of each individual’s face, but also gave each the smooth skin textures characteristic
of the composite. These faces were rated as much more attractive than the
originals, confirming that smooth skin texture seems to play a critical
role in the perception of beauty.

To assess the role of face shape, we distorted the shape of each of
the original faces to that of the composite of the same sex. This transformation
left the skin pigmentation – including any freckles or blemishes – as it
was in the original image. The distorted faces were rated as slightly more
attractive than the originals. Also, the facial shape of a composite was
seen as more attractive than that of a reconstructed original, underlining
the importance of facial configuration.

So both the smoothness of skin texture and the difference of face shape
from the population average have important roles in the perception of facial
attractiveness. With the male and female faces studied, skin texture was
the single most important factor in perception of attractiveness. Manipulating
either skin texture or face shape alone produces a small but measurable
increase in the perception of attractiveness but manipulating both provokes
a marked increase.

Galton obtained images of historical figures by combining portraits
from different artists. He derived a composite medallion of Alexander the
Great by photographically blending six different ancient medals. Galton
argued that the composite would be a truer likeness because the irregularities
introduced by different artists would average out. He commented that a composite
portrait of Cleopatra, though more attractive than the original portraits,
still did not ‘give any indication of her reputed beauty; in fact, her features
are not only plain, but to an ordinary English taste are simply hideous’.
Far from disproving his own theory, Galton’s remarks simply illustrate the
overriding effect of personal, cultural and historical attitudes to beauty.

Until recently artists produced portraits of historical figures in standardised
poses – that is, full frontal or profile. This artistic convention makes
averaging these portraits a relatively easy task. Even today, with many
different shots of a celebrity, we can derive an average image. We blended
together several images of Thatcher from a variety of photographs, taken
full face and profile, and with and without an open mouth. By Galton’s logic,
we should thus derive a more representative image because the subject may
have adopted unusual expressions, for instance, in individual poses.

Computer modification of facial appearance has many applications. A
computer caricature can be made by exaggerating differences between a given
face and the prototype image. We found that caricatures are sometimes more
easily recognised than the real images from which they were manufactured.
The suggestion is that in memorising different faces, we selectively store
the way individual faces differ from the population average. As caricatures
draw attention to these distinctive elements, they make it easier to access
the underlying representations than realistic portraits. Forensic scientists
are already trying to devise automated caricaturing methods that will provide
computer graphic equivalents to the Photofit technique developed by Jacques
Penry in the early 1970s.

In telecommunications, face image distortion can form the basis of signal
transmission for videophones. This is because the shape of an individual’s
face can be reconstructed realistically from the positions of only 200 points.
The changing shape of mouth and face could be captured by the 200 points
and transmitted down the telephone line.

In reconstructive facial surgery, doctors at University College Hospital,
London are planning face changes with three-dimensional computer graphics.
In cosmetics, too, there is the search for what makes a face beautiful.
But it is important not to lose sight of one of the more rewarding applications
of computer graphics: entertainment.

Philip Benson and David Perrett work at the Psychological Laboratory
at the University of St Andrews. This article is based on a chapter from
the book Photovideo: Photography in the Age of the Computer, edited by Paul
Wombell and published by Rivers Oram Press, London.