SYSTEMS, METHODS, AND DEVICES FOR AUDIO-VISUAL SPEECH PURIFICATION USING RESIDUAL NEURAL NETWORKS

摘要

This application is directed to audio purification. An electronic device obtains image data corresponding to a sequence of image frames that focus on lip movement of a person. The electronic device also obtains audio data that is synchronous with the lip movement in the sequence of image frames and modifies the audio data using the image data, thereby reducing background noise in the audio data. In some embodiments, the audio data is separated to first audio magnitude data and first audio phase data corresponding to distinct audio frequencies. The first audio magnitude data are modified to second audio magnitude data based on the image data. The first audio phase data are updated to second audio phase data based on the second audio magnitude data. The audio data is modified when the audio data are recovered from the second audio magnitude data and the second audio phase data.