COMPUTER-animated characters will soon be getting a lot more lippy. Software developed at MIT allows animated characters to mouth the appropriate lip movements for the words they’re saying, even if it hasn’t encountered them before.
The software will also make it possible for cellphones to realistically animate images of your friends’ faces, making them appear to read out text messages, for instance. It may even help Hollywood studios to reuse footage of long-dead actors by seamlessly adding new dialogue.
What makes the system unique, says Tony Ezzat at MIT’s Center for Biological and Computational Learning, is the way it extracts still images of lips from a video sequence of someone speaking, then uses them to construct animated images that can realistically mouth any words.
Advertisement
“The recorded subjects can be regular people, celebrities, ex-presidents or infamous terrorists,” says Ezzat, who developed the system with his colleagues Gadi Geiger and Tomaso Poggio. Unlike the cartoon-like animations of characters such as Ananova, Ezzat’s computer creations look indistinguishable from real people. In tests comparing unaltered video sequences of people talking with sequences animated using the new system, it proved almost impossible to tell which was which.
The lip-sync system is first trained using a sequence of video images of a person uttering a preset list of words and sentences. It breaks the images up into 46 individual still shots of different lip shapes—known as visemes—each with its associated sound unit, or phoneme.
When presented with speech, the system breaks it up into phonemes and pieces together an image sequence using the corresponding visemes. Then it “pastes” the lips back on the head, taking account of natural head and eye movements to add realism. The audio input might be be a straightforward voice recording, but if the software is given text it can use a speech synthesiser to supply the sound instead.
If the system encounters phonemes other than the 46 stored in its memory, it “invents” a new image of the lips based on the visemes either side in the sequence. It then morphs the series of images together so that the video sequence appears smooth.
Ezzat and his colleagues are now trying to make the system work in three dimensions so that it can handle faces from different angles. If successful, it could prove useful for movie directors who want to change dialogue in the editing suite without having to paste in new lip movements frame by frame. And it may even be possible to accurately reanimate dead actors, provided there is enough footage of them uttering different phonemes.
Ezzat will present the software at the computer graphics community’s annual SIGGRAPH meeting, to be held this July in San Antonio, Texas.