Preference-Based Asynchronous Speech Enhancement using Deep Learning with Feature Unlearning for Audio Privacy Preservation

Abstract

This research presents a privacy-preserving speech enhancement framework based on Transformer architectures, designed to improve audio quality while addressing user privacy and computational constraints. The proposed system leverages federated learning to train the enhancement model collaboratively across decentralized devices without sharing raw data, thereby minimizing privacy risks. To further strengthen privacy compliance and support user preferences, the framework integrates a feature unlearning mechanism that enables selective forgetting of specific user-contributed data, either pre- or post-training. The model is optimized for lightweight deployment, making it suitable for resource-constrained edge devices such as the Raspberry Pi, which functions as the client in our architecture. A Flask API infrastructure and Ngrok tunneling are used to facilitate secure communication between the Google Colab-based server and the client device. Experimental evaluations demonstrate the system’s effectiveness in enhancing speech quality while ensuring privacy and adaptability through unlearning, with minimal computational overhead on edge hardware.