The Visual Microphone: Passive Recovery of Sound from Video

摘要

When sound hits an object, it causes small vibrations of the object’srnsurface. We show how, using only high-speed video of thernobject, we can extract those minute vibrations and partially recoverrnthe sound that produced them, allowing us to turn everydayrnobjects—a glass of water, a potted plant, a box of tissues, or a bagrnof chips—into visual microphones. We recover sounds from highspeedrnfootage of a variety of objects with different properties, andrnuse both real and simulated data to examine some of the factors thatrnaffect our ability to visually recover sound. We evaluate the qualityrnof recovered sounds using intelligibility and SNR metrics and providerninput and recovered audio samples for direct comparison. Wernalso explore how to leverage the rolling shutter in regular consumerrncameras to recover audio from standard frame-rate videos, and usernthe spatial resolution of our method to visualize how sound-relatedrnvibrations vary over an object’s surface, which we can use to recoverrnthe vibration modes of an object.