IT WAS the prodigy’s party piece. As a child, Wolfgang Amadeus Mozart toured
the courts and concert halls of Europe challenging audiences to test his ear for
music. In Frankfurt a newspaper advertised that Mozart would “instantly name all
the notes played at a distance, whether singly or in chords as on the clavier or
any other instrument”. Audiences everywhere were astounded by his talent. In
London he performed for King George III, and the Royal Society published a study
of his musical genius.
Mozart not only had a good ear. He also had a phenomenal memory. In 1769, at
the age of 12, he went to hear a performance of Gregorio Allegri’s
Miserere in the Sistine Chapel in Rome. In those days the piece was only
performed by the Vatican choir during Holy Week. The musical score was a closely
guarded secret and the Pope threatened to excommunicate anyone who copied it.
But after the performance Mozart went home and wrote out the music from
memory—returning a couple of days later to make some minor corrections.
The Pope was so impressed with the boy’s achievement that he didn’t
excommunicate him.
Transcribing music—listening to a piece then writing it down—came
naturally to Mozart, but for most musicians it’s a real slog. Even highly
skilled professionals can take days to transcribe a few minutes of music. Back
in the 1980s the Smithsonian Institution in Washington DC wanted to stage a
concert featuring music from Miles Davis’s classic Birth of the Cool
recordings. They roped in Lee Konitz, alto sax player on the sessions, to help.
Bill Crow tells the story in his book Jazz Anecdotes (Oxford University
Press, 1990). Konitz rang Davis, who could be downright unhelpful. Davis said he
didn’t know where the arrangements were. So Konitz hired four top arrangers to
help transcribe the music. It took weeks. Elated by his success, Konitz rang
Davis to tell him that they had managed to recreate the arrangements. “Man, you
should have asked me,” said Davis. “Those mothers are all in my basement.”
Advertisement
So why can’t a computer do the donkey work? A few years ago the idea of
putting a CD in your computer and printing out a sheet of music from it was
ridiculous. Now there are programs that can do an adequate job, as long as there
is only one instrument playing one note at a time. Polyphonic
transcription—emulating Mozart and naming notes in chords—is much
harder, but there are signs that it could be cracked.
What makes transcription so hard is that music is a hugely complex mixture of
sound. The pitch of middle C on a well-tuned piano is 261.6 hertz. But play this
note and you won’t just hear a pure frequency of 261.6 Hz. There are lots of
other frequencies—often called harmonics—in there too, one at double
the fundamental frequency and others at multiples of three, four, five and so
on. Play several notes together in a chord and you’ll hear an even more
complicated mixture of frequencies
(see Diagram). To transcribe the chord
you’ve got to distinguish the fundamental notes from the harmonics. But how do
you tell one from the other?
In computing terms, transcription means converting the frequencies into Midi.
Midi files aren’t actually music but they contain information about it, such as
which instrument is playing, which note it should play, when to play it and how
loud it should be. In other words a Midi file is an electronic musical score.
Store a piece in Midi format and you can print it as musical notation relatively
easily, or convert it into an audio file. But turning music into Midi is
difficult—a bit like writing down a recipe based on a chemical analysis of
the residue at the bottom of an old cooking pot.FIG-mg23225801.jpg
Accurate transcriptions would be a huge help for musicians. They’d be able to
learn music that hasn’t been scored—jazz, for example, is often not
written down or published, and the same goes for a great deal of non-Western
music. Even in classical music, automatic transcription would help musicians
analyse the nuances that great performers introduce into the original score.
Translating audio files into Midi will be very handy in other ways too. Midi
is a great way to compress music into a tiny space—Midi files are at least
a thousand times smaller than CD-quality audio files. “I think the real money is
going to be in compression, so that you can send music files over the Internet,”
says Mark Plumbley of Queen Mary, University of London, a member of a joint
research programme with the University of Indiana to transcribe music
automatically.
Many attempts at computerised transcription have concentrated on a single
instrument, the piano. Providing a piano is in tune, when you hit middle C
that’s the note you get. But with a violin, notes can be out of tune. The
piano’s predictability makes it the friendliest instrument to work with, says
Simon Dixon of the Austrian Research Institute for Artificial Intelligence in
Vienna.
Dixon has written a computer program to transcribe polyphonic piano music.
Like a number of other researchers he starts by separating the frequencies in
the music with a Fourier transform, a mathematical technique for picking out the
various frequencies from a complex signal. He then works out which notes have
been played by matching the pattern of peaks against a database of different
fundamentals and their harmonics. His system picks about 70 to 80 per cent of
the notes correctly—not bad, but you’d certainly notice something odd
about a familiar piece if a quarter of the notes were missing or wrong.
Matija Marolt of the University of Ljubljana in Slovenia also works with
piano music, but his software operates in a radically different way. It has
three stages. First, it increases the strength of weaker frequencies. Next it
filters out background noise. And finally, it uses neural networks trained on
more than 120 pieces of piano music to pick out the notes.
The first stage splits the music into 200 frequency bands and feeds them into
a model that mimics the way the ear works. This dampens louder sounds and
strengthens softer frequencies that could otherwise be drowned out—a step
which Marolt says helps to prevent his model missing notes.
Then he sends the frequency bands through a bank of software oscillators,
each tuned to a particular frequency. If an oscillator “hears” its own frequency
it starts oscillating along. The oscillators are organised into 88 overlapping
groups. Each group represents a note on the piano, containing its fundamental
and the main harmonics. The 261.6 Hz oscillator, for example, is in the middle-C
group but also in groups where 261.6 Hz is a harmonic.
If the music contains a middle C, all the oscillators in the middle-C group
will oscillate in phase, 261.6 times a second. But stray noises that are not
part of the music may also trigger the 261.6 Hz oscillator and a few others. If
only a few of the oscillators in a group have been induced, Marolt’s model cuts
their output. This weeds out noise—which normally consists of frequencies
that are not harmonically related, says Marolt. Finally, neural networks pick
out the notes from the outputs of the groups that are still producing a signal.
Marolt told a music technology meeting in Barcelona last month that on real-life
recordings his software picks about 85 per cent of the notes correctly.
Meanwhile, researchers at Queen Mary are tackling the transcription problem
with techniques borrowed from speech recognition. In many ways, identifying
notes in a piece of music is comparable to a classic speech recognition
problem—the cocktail party effect. At parties, the finely tuned human ear
can easily focus on one person even when the background noise is extremely loud.
It’s a trick that comes easily to us. But cracking the problem computationally
requires a degree of intelligence: the software not only has to recognise words
but fill in gaps and string them together so that they make sense.
Plumbley points out that speech recognition programs improved dramatically
when they were given the fundamentals of grammar. He says that music also has
its own grammar—some clusters of notes are more likely than others.
Software armed with this knowledge will produce better results.
The notes in a piece of music are not chosen at random. People don’t play all
88 notes on a piano simultaneously. Notes are often grouped together in chords,
and these have a limited number of permutations. Plumbley’s colleagues Mark
Sandler and Juan Bello at Queen Mary are applying this knowledge of chord
patterns to rule out the more unlikely possibilities.
Anssi Klapuri of Tampere University of Technology in Finland is working on a
more complicated problem—transcribing a whole orchestra. The problem is
much more complex because each instrument has a different balance of harmonics
when playing the same note. This “harmonic fingerprint” is one of the ways we
tell instruments apart, but it makes separating fundamentals from harmonics more
complicated. His software works with any combination of 26 different
instruments, ranging from the violin to the saxophone and even the human
voice.
Dividing the waves
Klapuri’s software separates the frequencies and then picks the loudest note.
This note, together with its harmonics, is subtracted from the signal and the
software then goes through the same process looking for the next loudest note,
then the next and so on. His software can pick the right first note 99 per cent
of the time even with the complex mixture of frequencies produced by a six-note
chord.
The quality of the audio signal deteriorates as sound is removed from it and
so the number of errors increases. On average, in a piece of music with a series
of four-note chords, the software will miss about 10 per cent of notes
altogether. For six-note chords this error rate is 20 per cent. Despite these
mistakes, the model is better at picking the notes than many musicians. Klapuri
compared his model’s results with the efforts of 10 music students. Most of them
were nowhere near as good as the software and only the two best were as accurate
at picking out the notes as the model.
But there’s more to converting audio files into Midi than just identifying
the right notes. You also need to know when the notes are played and how loud
they are. So most transcription software scans the frequencies and looks for a
sudden burst of sound energy, which identifies the start of a new note. The
models determine how loud a note is from its amplitude and assign a velocity to
it—the Midi equivalent of loudness. “Provided that we’ve found the correct
note we can determine the loudness rather accurately,” says Klapuri.
Despite the enormous strides made by researchers in the past few years there
are still technical hurdles. Marolt’s software doesn’t perform so well on Oscar
Peterson’s version of Bye Bye Blackbird, partly because Peterson’s
fingers are just too fast for the software. “Fast passages are one of the major
problems,” says Marolt. However, the software copes rather well with the more
sedate pace of Scott Joplin’s ragtime classic The Entertainer.
Bass notes are also problematic, mainly because some harmonics are far louder
than the fundamental. One of the commonest errors is picking a note that is an
octave higher than the note that was played.
The third source of error is repeating notes that are held for a long time.
Klapuri says that if a drummer hits a drum while a violinist is playing, the
drum drowns out the violin. But as the drum dies away the violin can be heard
once more. The software may be fooled into thinking that the violin has started
playing a new note. Marolt’s error statistics show that about 80 to 90 per cent
of wrongly transcribed notes are either octave errors or repeated notes.
In his 1770 report to the Royal Society, amateur scientist Daines Barrington
raved about Mozart’s talent (Philosophical Transactions, vol 60, p 54).
“I was witness of his most extraordinary abilities as a musician, both at some
publick concerts and likewise having been alone with him.” Will computers ever
be able to emulate Mozart’s ear? “We can never get all the notes,” says Klapuri.
“But if you redefine the question and say `will we be able to transcribe music
as well as humans?’ one day I think we will. But it will take a lot of time and
ڴڴǰ.”
-
Hear Matija Marolt’s software in action at
http://lgm.fri.uni-lj.si/~matic/research.html,
and Anssi Klapuri’s at
www.cs.tut.fi/~klap/iiro/
