In the art of video editing, sound is really half the story. Through formative interviews with professional editors (N=10), we found that the manual task of adding sounds to video can be challenging. This paper presents Soundify, a system that helps editors match sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a wide range of audio categories. As a byproduct of our evaluation study, we create a dataset of 497 high-quality audio-video pairs covering 53 classes matched with Soundify and verified by humans. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Video:

 

Learn more: Soundify: Matching Sound Effects to Video 

In collaboration with: Anastasis Germanidis, Cristóbal Valenzuela and Yining Shi