DUAL-STREAM 3D-CNN AND ATTENTION-MEDIATED FUSION FOR RESILIENT AUDIO-VISUAL SPEECH ENHANCEMENT

Abstract

The preservation of intelligible speech in acoustically hostile conditions such as industrial facilities, crowded urban street traffic, and emergency situations has been identified as one of the most critical obstacles to implementing speech recognition systems universally. Below 0dB signal-to-noise ratio (SNR), traditional acoustic denoising methods are incapable of eliminating interference, sometimes resulting in non-linear distortions which can destroy the phonetic boundaries needed for automatic machine processing. This paper aims to tackle this limitation through the creation of a cross-modal phonetic reconstruction approach that takes advantage of the innate noise tolerance in the visualization of articulation. The model leverages a dual-stream 3D-CNN encoder, where one stream captures spatiotemporal viseme dynamics based on video input from around the mouth region, whereas another stream extracts features of a degenerated spectrogram using the acoustic encoder. The fusion between the two streams is performed via the cross-attention module, with the former stream acting as a template for the alignment of auditory phonemes that need to be reconstructed in real-time. Training is accomplished with a hybrid loss function that optimizes the spectral similarity as well as phonetics in terms of a pre-trained ASR encoder. Evaluation results obtained by training on a dataset consisting of LRS3-TED and VoxCeleb2 under non-stationary noise at -5dB show superior performance compared to current solutions. The model attains an accuracy score of 94.6%, a precision of 0.92, and word error rate (WER) of 12.4%, whereas the baseline, an audio-only transformer network, shows scores above 38%. Additionally, speech perceptual quality and intelligibility improve to achieve a PESQ of 3.12 and STOI of 0.89. Inference speed is demonstrated to be up to 120 fps using multiple streams on a single GPU with latency below

18.4 ms.