Whistle a jaunty tune, and artificial intelligence can convert it into the style of a Mozart symphony played by an entire orchestra. Facebook has built an AI that can translate any style of music played on any instrument into a Mozart symphony, a Haydn string quartet, or even a Bach cantata with a full orchestra and chorus.
The system consists of two main parts: an encoder, which extracts the features of the input music, and a decoder, which makes those features match whatever style of music it’s being translated to.
Advertisement
The encoder doesn’t have a perfect representation of the input. Instead, it randomly changes some of the tones so that the decoder cannot simply regurgitate the exact notes back to you, only with different instruments.
“It makes it speak a more global, universal language of music instead of just recreating the notes,” says says at Facebook.
[video_player id=”LqsUwTiB” access_level=”subscriber”]
The decoder was trained on six different styles of classical music: Mozart’s symphonies, Haydn’s string quartets, Beethoven’s piano sonatas, Bach’s string cantatas for orchestra, chorus, and soloists, Bach’s organ works, and Bach’s keyboard works as played on a harpsichord.
The team tested their network by having it convert segments of music – some Bach, some Mozart, and some swing jazz, metal guitar riffs, and instrumental Chinese music – into piano sonatas. They then employed three human musicians to do the same task. The two conversions were then scored out of five by 40 people recruited online.
For now, humans are still better. The human conversions got average ratings above a 4, while the network’s scores were closer to 3. However, in tests where the researchers presented six clips of music – one original and five translations to other styles – to freelancers and musicians, almost everyone had a hard time identifying the original.
The network isn’t like other music synthesisers, says Google software engineer Adam Roberts. “What’s new here is being able to do the modification end-to-end directly from the audio” without converting the input into individual musical notes first, he says. You could sing directly into a microphone and the system will spit classical music right back out.
Journal Reference:
