a Analysis of Model Inversion Attack and Defense Strategies on Speaker Classification Models: An Architecture-Level Privacy Perspective

Abstract

Recent advances in Deepfake voice generation have intensified concerns over speech privacy, as constructing realistic synthetic voices requires access to representative speaker features. Model inversion attacks (MIAs) operationalize this threat by reconstructing such features directly from trained recognition models. However, prior studies rarely examined how model architecture governs inversion vulnerability or under what conditions differential privacy (DP) provides meaningful defense. This work bridges that gap by systematically evaluating the architectural determinants of inversion susceptibility across three speaker classification frameworks—Raw waveform convolutional, SincNet, and MFCC-based models—under white-box gradient-based attacks. Experiments were conducted on three public datasets (50_Speakers, UrbanSound8K, and TIMIT) using both traditional full-gradient and sliding window attack strategies, with performance assessed via attack success rate, attack time, and D-vector reconstruction distance. Key findings demonstrate that: (1) waveform-based architectures provide substantially stronger privacy protection than spectrogram-based models, with the Raw model exhibiting the highest resistance and SincNet offering a favorable trade-off between convergence stability and robustness; (2) a defensive temporal sweet spot exists at approximately 1.6–2.0 seconds of input duration, within which model resistance to inversion is maximized; (3) model depth correlates positively with attack resistance, while the sliding attack decays in effectiveness more rapidly than the full-gradient strategy as model complexity increases; and (4) differential privacy functions as an architecture-dependent defense amplifier—significantly enhancing robustness in Raw and SincNet models while providing negligible benefit to the fundamentally vulnerable MFCC architecture. These findings establish architecture-level design principles for privacy-preserving speech AI systems and demonstrate that effective security requires coherent architectural choices rather than isolated defensive mechanisms.