What question did this study set out to answer?

To develop an acoustic eavesdropping system capable of identifying target speech in environments with multiple speakers without prior knowledge of conditions.

March 18, 2026

LAM-assisted Acoustic Eavesdropping in Multi-speaker Scenarios via Commercial mmWave Radar

Key Points

To develop an acoustic eavesdropping system capable of identifying target speech in environments with multiple speakers without prior knowledge of conditions.
Utilized commercial off-the-shelf millimeter-wave radar to collect faint signals from speech-induced vibrations.
Developed a denoising algorithm to enhance the captured signals.
Implemented a recognition model combined with a large audio model for keyword extraction from mixed speech signals.
Conducted experiments on a custom dataset with 120 hours of audio and 60 hours of radar data.
Achieved an average word error rate of 6.46% with 1-2 simultaneous speakers for identifying 36 target keywords.
Obtained an average word error rate of 14.97% in scenarios with 3-5 speakers.
Maintained average word error rates below 30% even with up to 7 simultaneous speakers.

Abstract

Eavesdropping on target speech in enclosed and private rooms, such as secret gatherings or private parties, has become a growing research trend and poses one of the most significant threats to personal privacy leakage. Previous studies often made various assumptions about the target and scenarios of eavesdropping, such as the presence of only a single speaker, a very slow speech rate, and clear separation of words, all of which greatly limited the applicability of eavesdropping systems. In this paper, we propose an acoustic eavesdropping system, mmMPS, based on commercial off-the-shelf millimeter-wave (COTS mmWave) radar, which is capable of eavesdropping on specific target speech in scenarios where multiple people speak simultaneously, without requiring prior knowledge of background voices, speech rate, or intensity. The system utilizes a single COTS mmWave radar to capture faint signals caused by the vibration of objects due to human speech, combines these with our proposed denoising algorithm to recover the speech signal, and then uses our recognition model along with large audio model (LAM) to extract target keyword speech from the mixed speech signals of multiple speakers. Extensive experiments were conducted on a nearly 120 hours audio dataset and 60 hours radar dataset that we created using LAM and collected in experimental scenarios. The system is capable of covering various target users, speech rates, and speech intensities. The results show that in a scenario with 1–2 simultaneous speakers, the system achieves the average WER/CER (Word/Character Error Rate) of 6.46% (the lower, the better) in identifying 36 target keywords. In a scenario of 3–5 speakers, the average WER/CER is 14.97%, and even in the scenario with up to 7 simultaneous speakers, the average WER/CER remains below 30%. To the best of our knowledge, mmMPS is the first system to achieve high-accuracy eavesdropping on target speech in multi-speaker scenarios, significantly expanding the application scope of acoustic eavesdropping.

Bookmark

Cite This Study

Liu et al. (Mon,) studied this question.

synapsesocial.com/papers/69ba432b4e9516ffd37a42f5 https://doi.org/https://doi.org/10.1145/3789687

Bookmark