[video_player id=”BtQ3b59j” access_level=”everyone”]
A neural network can distinguish between people and objects in a video, and speed up or slow down their movements separately while ensuring they smoothly interact with each other. This could be used to dramatise or de-emphasise certain movements or events in a video.
To achieve this, a team at Google and the University of Oxford split out each frame of video into separate layers and taught an AI to identify the people or objects in them.
Advertisement
This neural network homes in on the things in each layer by focusing on their movements. Then it further separates each object or person onto its own layer. The background is also isolated into a layer.
The neural network also tracks the way people or objects interact with the world around them in the video. “You also have to change the things in the scene that move with them – their shadows, reflections, or water splashes,” says Erika Lu at the University of Oxford, UK.
These details are also picked up by the neural network and sped up or slowed down. That’s done by deep learning, associating the elements around a person or object with the object itself. Previously, such elements had to be highlighted by hand – a time-consuming, costly process.
The AI can then stitch these back together after altering them. The result is the ability to speed up, say, one pair of ballroom dancers and slow down another in the same video, near-seamlessly masking the moment they cross over each other mid-spin.
“The paper will inspire further development of such techniques for advanced video editing in the future,” says Jia-Bin Huang at Virginia Tech University in the US. He says this work is “impressive”.
Huang points out that the method used requires training the AI on each individual video, making it time consuming. The authors also admitted some challenges, including the neural network struggling to pick up things like flashing lights as objects that needed to be discretely animated.
Reference:
